I had the same problem (stack overflow). The solution was to change
the >>= operator in the Get monad. Currently it is:
m >>= k = Get (\s -> let (a, s') = unGet m s
in unGet (k a) s')
but I changed it to:
m >>= k = Get (\s -> case unGet m s of
(a, s') -> unGet (k a) s')
It seems that the bind operator is lazy and this caused the stack overflow.
I have also another problem. Every Int and Word is stored as 64-bit
value and this expands the output file a lot. I have a lot of integers
and most of them are < 128 but not all of them. I changed the
serialization so that the Int and Word are serialized in a variable
number of bytes. Without this change the binary serialization was even
worse than the textual serialization that we had before. The file was
almost full with zeros.
I just haven't time to prepare a patch and to send it for review but
if other people have the same problem I will do it.
Best Regars,
Krasimir
On Wed, Aug 13, 2008 at 1:13 AM, Tim Newsham
I have a program that read in and populated a large data structure and then saved it out with Data.Binary and Data.ByteString.Lazy.Char8:
saveState db = B.writeFile stateFile =<< encode <$> atomically (readTVar db)
when I go to read this in later I get a stack overflow:
loadState db = do d <- decode <$> B.readFile stateFile atomically $ writeTVar db d
Stack space overflow: current size 8388608 bytes. Use `+RTS -Ksize' to increase it.
or from ghci:
d <- liftM decode (Data.ByteString.Lazy.Char8.readFile "savedState.bin") :: IO InstrsDb
fromList *** Exception: stack overflow
The data type I'm storing is a Map (of maps):
type DailyDb = M.Map Date Daily type InstrsDb = M.Map String DailyDb
What's going on here? Why is the system capable of building and saving the data but not in reading and umarhsalling it? What is the proper way to track down where the exception is happening? Any debugging tips?
I also noticed another issue while testing. If my program loads the data at startup by calling loadState then all later calls to saveState give an error:
Log: savedState.bin: openFile: resource busy (file is locked)
this does not occur if the program wasnt loaded. My best guess here is that B.readFile isnt completing and closing the file for some reason. Is there a good way to force this?
Tim Newsham http://www.thenewsh.com/~newsham/ _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe