Gracjan Polak writes:

The problem is that my prog allocates a lot just to free it immediatelly after. But what?

Use +RTS -hd instead, which will tell you the constructor. I bet you'll find it's (:), and that you are retaining a load of Chars from your input file, pending evaluation of the elements from your map. -k -- If I haven't seen further, it is by standing in the footprints of giants

Ketil Malde writes:

Gracjan Polak writes:

...

let entries = IntMap.fromList (map (\(a,b,c) -> (a,c)) (concat p))

Gut reaction: Map is lazy in its values (but probably not the key, which are checked for order), so you should force the 'c' before inserting it in the map. (There's probably a strict fromList or IntMap somewhere?)

I tried both Map and IntMap and there was no difference in memory total usage or usage pattern. Seems I'm already strict enough. Values are left lazy till the point where they are forced, and that is at write-out in my current excersise. I'd want to leave them lazy as in more involved transformation not all of them will be needed. -- Gracjan

Ketil Malde writes:

Then you get the memory behavior you ask for. Unevaluated strings are extremely expensive, something like 12 bytes per char on 32 bit, twice that on 64 bits, and then you need GC overhead, etc. ByteStrings are much better, but you then probably need to implement your own XML parsing.

My lazy chunks have type ByteString -> Object. Only internally they use ByteString.unpack to get the list of Word8s to parse them. My parser is totally my own so I can do anything I wish. Except it is hard for me to image a parser working on something else than [Word8]. How do I do this? So how do I get rid of those (:) and W8# that are allocated everywhere on my heap? Thanks for the suggestion for -hd, really useful option! -- Gracjan

Don Stewart writes:

ByteStrings have all the same operations as lists though, so you can index, compare and take substrings, with the benefit that he underlying string will be shared, not copied. And only use 1 byte per element.

Is there any parser built directly over ByteString that I could look at? Or maybe somebody implemented something like Text.ParserCombinators.ReadP for ByteString?

From the first sight it seems doable, so there is light at the end of the tunnel :)

-- Gracjan

gracjanpolak:

Gracjan Polak writes:

...

Don Stewart writes:

...

ByteStrings have all the same operations as lists though, so you can index, compare and take substrings, with the benefit that he underlying string will be shared, not copied. And only use 1 byte per element.

Is there any parser built directly over ByteString that I could look at?

Or maybe somebody implemented something like Text.ParserCombinators.ReadP for ByteString?

...

From the first sight it seems doable, so there is light at the end of the tunnel :)

Just a success report, after 58 min of coding I got kind of ReadP parser over ByteString working and my memory usage went down from 1500MB to... 1.2MB! Over 1000 times better! Incredible!

Thanks for the suggestion to do it with ByteStrings!

I hope to publish it when I clean it up enough!

That is really awesome! Sharing input strings around with bytestrings really should lead to excellent memory savings in parsing. I'm glad we see this confirmed. Will you be releasing the code soon? -- Don