Duncan, To be honest, I am not too keen on using files to buffer the AST and symbol tables, as it adds another level of complexity. Moreover, an operation that is performed whenever C declarations are analysed is declaration chasing, where name analysis (using functions from CTrav.hs) follows declarations involving typedefs. Depending on where the types are defined, access can be non-local and may slow everything down quite a bit. Do you have any idea which data structures take most of the space? Or is it just that after expanding the header files of GTK widgets, the resulting C pre-processor output is so large that it just takes a lot of space to hold it? Manuel On Sun, 2005-01-30 at 16:34 +0000, Duncan Coutts wrote:
Hi all,
For people building gtk2hs, we've found that the large amount of heap space required by c2hs to be a problem. It means people with older machines with less than about 400Mb of RAM cannot build gtk2hs.
For recent versions of Gtk, parsing and name analysis requires 350m of heap space. (ie, runing c2hs +RTS -M340m -RTS will run out of heap, but c2hs +RTS -M350m -RTS will be ok).
So we've been pondering how to reduce the heap requirements. The key point is that we do not want to have to keep the whole of the AST + symbol maps in memory at once.
For the parsing phase, this should not be a problem, the parsers works declaration by declaration, the only thing that is accumulated is the set of typedef names. There are two options here, we could write out each declaration one at a time to another file using the binary serialisation framework. Alternatively, if the list of declarations could be returned lazily by the parser then that should work ok.
The harder bit is the name analysis. It reads the declaration list in a linear pattern (so it should work well with a lazy parser or a list of declarations deserialised one by one out of a file). The CTagNS namespace and CDefTable seem to be write only; which is good as they could be written out to file immediately. The CShadowNS is not generated during the name analysis phase. The CObjNS namespace is trickier since it is both written and used for lookups. We could live with keeping this one in memory or alternatively it should be possible to both write the map bit by bit and do random reads for the lookups. The lookups themselves do not retain any heap since they immediately write the value out into another map.
The name analysis phase actually doesn't use many map lookup/insert operations. If each of these could be re-defined locally to work in the local NA monad and then the NA monad extended to know the files we are reading from/to then in the runNS we could switch between doing lookups/inserts from in heap FiniteMaps or to/from files.
runNA :: NA a -> Either AttrC Files -> a -> CST s (Either AttrC Files)
My point is that we wouldn't need to change any existing code paths. The use of intermediate files could even be controlled by a --conserve-memory flag or something (since it would probably slow down the cases where currently everything fits into memory).
Just looking for feedback; particularly from Manuel as to whether he thinks this is a plan worth pursuing.
Duncan
_______________________________________________ C2hs mailing list C2hs@haskell.org http://www.haskell.org/mailman/listinfo/c2hs