On Tue, Aug 12, 2008 at 5:13 PM, Tim Newsham
The data type I'm storing is a Map (of maps):
type DailyDb = M.Map Date Daily type InstrsDb = M.Map String DailyDb
What's going on here?
The default marshalling scheme that Binary uses for lists and maps (which are flattened to lists before writing out) is not streamable. Instead of writing out data in chunks, it computes the length of the list and writes that, followed by the elements. Presumably on the read side, a huge thunk is being built up before any actual Map creation starts.
I also noticed another issue while testing. If my program loads the data at startup by calling loadState then all later calls to saveState give an error:
Log: savedState.bin: openFile: resource busy (file is locked)
this does not occur if the program wasnt loaded.
Your loading of state isn't being forced to complete, so the file handle is still open when you try to save to the same file. The H98 standard requires that file handles be locked for exclusive access during writes. To force the read to finish, use rnf. You can find a description of how to use it, and the typeclasses involved, here: http://book.realworldhaskell.org/beta/concurrent.html