daniel.is.fischer:
Am Dienstag, 12. September 2006 22:26 schrieben Sie:
Daniel Fischer wrote:
The programme consumed more and more memory (according to top), kswapd started to have a higher CPU-percentage than my programme, programme died, system yelling 'Speicherzugriffsfehler', top displays 'kswapd<defunct>'. I believe that means my programme demanded more memory than I have available (only 256MB RAM + 800MB swap). Is that a segfault or what is the correct term?
That is probably due to (apart from the stupidity of my IO-code) the large overhead of Haskell lists.
Most certainly not. I'm pretty sure this is to a bug in your code. Something retains a data structure which is actually unneeded. Probably
Apparently. And my money is on a load of lines from the file (of which I need only the first and last Char).
a case of "foldl" where "foldl'" should be used or a "try" in Parsec code where it should be left out or a lot of "updateWiths" to a Map, etc. Or it could be a bad choice of data structure. I bet, it's the map you're using to represent the graph (which you don't even need to represent at all, btw).
No foldl nor parsec around. I represent the graph as a
UArray (Char,Char) Int
(I've switched to Int for the index type, too, when tuning the code), so that shouldn't use much memory (array size is 676). The array is built via accumArray, I hope that's sufficiently efficient (though now I use unsafeAccumArrayUArray, that's faster).
How could I solve the problem without representing the graph in some way? Possibly that could be done more efficiently than I do it, but I can't imagine how to do it without representing the graph in some data structure.
So the chunk of the file which easily fits into my RAM in ByteString form is too large as a list of ordinary Strings.
The chunk of file should never need to fit into RAM. If that's a problem, you also forgot to prime a crucial "foldl".
Forgive the stupid question, but where if not RAM would the chunk currently processed reside?
I agree. Some problems simply require you to hold large strings in memory. And for those, [Char] conks out around 5-10M (try reversing a 10M [Char]). -- Don