Hi all, For people building gtk2hs, we've found that the large amount of heap space required by c2hs to be a problem. It means people with older machines with less than about 400Mb of RAM cannot build gtk2hs. For recent versions of Gtk, parsing and name analysis requires 350m of heap space. (ie, runing c2hs +RTS -M340m -RTS will run out of heap, but c2hs +RTS -M350m -RTS will be ok). So we've been pondering how to reduce the heap requirements. The key point is that we do not want to have to keep the whole of the AST + symbol maps in memory at once. For the parsing phase, this should not be a problem, the parsers works declaration by declaration, the only thing that is accumulated is the set of typedef names. There are two options here, we could write out each declaration one at a time to another file using the binary serialisation framework. Alternatively, if the list of declarations could be returned lazily by the parser then that should work ok. The harder bit is the name analysis. It reads the declaration list in a linear pattern (so it should work well with a lazy parser or a list of declarations deserialised one by one out of a file). The CTagNS namespace and CDefTable seem to be write only; which is good as they could be written out to file immediately. The CShadowNS is not generated during the name analysis phase. The CObjNS namespace is trickier since it is both written and used for lookups. We could live with keeping this one in memory or alternatively it should be possible to both write the map bit by bit and do random reads for the lookups. The lookups themselves do not retain any heap since they immediately write the value out into another map. The name analysis phase actually doesn't use many map lookup/insert operations. If each of these could be re-defined locally to work in the local NA monad and then the NA monad extended to know the files we are reading from/to then in the runNS we could switch between doing lookups/inserts from in heap FiniteMaps or to/from files. runNA :: NA a -> Either AttrC Files -> a -> CST s (Either AttrC Files) My point is that we wouldn't need to change any existing code paths. The use of intermediate files could even be controlled by a --conserve-memory flag or something (since it would probably slow down the cases where currently everything fits into memory). Just looking for feedback; particularly from Manuel as to whether he thinks this is a plan worth pursuing. Duncan
Duncan, To be honest, I am not too keen on using files to buffer the AST and symbol tables, as it adds another level of complexity. Moreover, an operation that is performed whenever C declarations are analysed is declaration chasing, where name analysis (using functions from CTrav.hs) follows declarations involving typedefs. Depending on where the types are defined, access can be non-local and may slow everything down quite a bit. Do you have any idea which data structures take most of the space? Or is it just that after expanding the header files of GTK widgets, the resulting C pre-processor output is so large that it just takes a lot of space to hold it? Manuel On Sun, 2005-01-30 at 16:34 +0000, Duncan Coutts wrote:
Hi all,
For people building gtk2hs, we've found that the large amount of heap space required by c2hs to be a problem. It means people with older machines with less than about 400Mb of RAM cannot build gtk2hs.
For recent versions of Gtk, parsing and name analysis requires 350m of heap space. (ie, runing c2hs +RTS -M340m -RTS will run out of heap, but c2hs +RTS -M350m -RTS will be ok).
So we've been pondering how to reduce the heap requirements. The key point is that we do not want to have to keep the whole of the AST + symbol maps in memory at once.
For the parsing phase, this should not be a problem, the parsers works declaration by declaration, the only thing that is accumulated is the set of typedef names. There are two options here, we could write out each declaration one at a time to another file using the binary serialisation framework. Alternatively, if the list of declarations could be returned lazily by the parser then that should work ok.
The harder bit is the name analysis. It reads the declaration list in a linear pattern (so it should work well with a lazy parser or a list of declarations deserialised one by one out of a file). The CTagNS namespace and CDefTable seem to be write only; which is good as they could be written out to file immediately. The CShadowNS is not generated during the name analysis phase. The CObjNS namespace is trickier since it is both written and used for lookups. We could live with keeping this one in memory or alternatively it should be possible to both write the map bit by bit and do random reads for the lookups. The lookups themselves do not retain any heap since they immediately write the value out into another map.
The name analysis phase actually doesn't use many map lookup/insert operations. If each of these could be re-defined locally to work in the local NA monad and then the NA monad extended to know the files we are reading from/to then in the runNS we could switch between doing lookups/inserts from in heap FiniteMaps or to/from files.
runNA :: NA a -> Either AttrC Files -> a -> CST s (Either AttrC Files)
My point is that we wouldn't need to change any existing code paths. The use of intermediate files could even be controlled by a --conserve-memory flag or something (since it would probably slow down the cases where currently everything fits into memory).
Just looking for feedback; particularly from Manuel as to whether he thinks this is a plan worth pursuing.
Duncan
_______________________________________________ C2hs mailing list C2hs@haskell.org http://www.haskell.org/mailman/listinfo/c2hs
On Mon, 2005-03-14 at 11:43 +1100, Manuel M T Chakravarty wrote:
Duncan,
To be honest, I am not too keen on using files to buffer the AST and symbol tables, as it adds another level of complexity.
Sadly this is quite true. I was commenting to some other Haskell hackers that there's probably a paper or so in designing a framwork / techniques for writing external algorithms in Haskell (ones where the dataset is not expected to fit in main memory).
Moreover, an operation that is performed whenever C declarations are analysed is declaration chasing, where name analysis (using functions from CTrav.hs) follows declarations involving typedefs. Depending on where the types are defined, access can be non-local and may slow everything down quite a bit.
Right, as I recall one of the maps was read-write but the others were write only.
Do you have any idea which data structures take most of the space? Or is it just that after expanding the header files of GTK widgets, the resulting C pre-processor output is so large that it just takes a lot of space to hold it?
I know the preprocessed header is large (765K for gtk 2.4) but I think c2hs's use is more than one would expect for that. I tried running c2hs on that 765K gtk.i file just now without any +RTS -M650m -RTS heap limit. I had to kill it after it had allocated 1.3Gb on my machine which has 1Gb of RAM and brought everything to a crawl. With a memory limit in place it can complete using 'only' 650Mb. So that's a 10x memory use compared to the original file. I have not been able to figure out exactly which bit of the data structure is taking so much space. I've found GHC's space profiling tools just don't tell me that (or I don't understand the profiling output enough). I suspect that there is a great deal of the AST that is kept but is never used. But I cannot pinpoint anything. I don't think it is the strings themselves. Using a sharing symbol table (like ghc uses) and packed strings made an insignificant difference in my tests. The space profiling does show that the finite maps take a very large proprotion of the space compared to the AST (but maybe I'm misreading the profile graphs since the maps are also retainers for bits of the AST) Sorry this isn't teribly helpful. Duncan
participants (2)
-
Duncan Coutts -
Manuel M T Chakravarty