Re: [Haskell-cafe] Data.List / Map: simple serialization?

On Thu, Jun 9, 2011 at 7:23 PM, Max Bolingbroke wrote:

Hi Dmitri,

On 9 June 2011 09:13, Dmitri O.Kondratiev wrote:

...

I wonder how Haskell will distribute memory between the buffer for sequential element access (list elements, map tree nodes) and memory for computation while reading in list, Data.Map from file?

Your list only has 30,000 elements. From the description of the problem, you traverse the list several times, so GHC will create an in-memory link list that persists for the duration of all the traversals. This is OK, because the number of elements in the list is small.

For the construction of the Map, it sounds like in the worst case you will have 30,000*30,000 = 900,000,000 elements in the Map, which you may not want to keep in memory. Assuming "show", "read" and list creation are lazy enough, and as long as you use the Map linearly GHC should be able to GC parts of it to keep the working set small. You should experiment and see what happens.

My advice is just write the program the simple way (with show and read, and not worrying about memory) and see what happens. If it turns out that it uses too much memory you can come back to the list with your problematic program and ask for advice.

Max

Yes, that's what I will try first - simple serialization with show and read, not worrying about memory. Thanks!