On Tue, 2009-03-03 at 02:12 +0100, Manlio Perillo wrote:
Hi.
In the "help optimizing memory usage for a program" I discovered some interesting things:
1) Using lazy ByteStrings to read files it not a good choice, since the garbage collector is not able to proper garbage cleanup.
Running with -F1 RTS flag, however, keeps memory usage down.
It is certainly possible to have proper garbage cleanup. I can write programs using lazy ByteStrings that process many megabytes of data and yet run in 1Mb of heap space. At first guess it sounds like you're holding onto too much, if not the whole stream perhaps bits within each chunk. Each chunk read from the file is 64k big and keeping any substring will force the whole chunk to be retained. If you're only keeping a fraction of each chunk you can use the ByteString.Lazy.copy function to make a deep copy and let the original 64k chunk get collected. Duncan