Hi,
So I have a data instance which contains a few big matrices, and I want to save my instance to a file so I can `read` it back later
to avoid the long computation every time (I'm training a recurrent neural network).
First attempt, the simplest method:
writeFile "rnn" (show dataInstance)
It starts to take all of the memory and then bails out with `out-of-memory` error.
So I wrote a function to write the string chunk by chunk, without buffering, here is the code:
https://github.com/mdibaiee/sibe/blob/728df02fbdd6f134af107c098f5477094c61ea76/examples/recurrent.hs#L52-L64
Copy/pasted from the link:
saveRecurrent :: FilePath -> String -> Int -> IO ()
saveRecurrent path str chunkSize = do
handle <- openFile path AppendMode
hSetBuffering handle NoBuffering
loop handle str
hClose handle
where
loop _ [] = return ()
loop handle s = do
hPutStr handle $ take chunkSize s
hFlush handle
loop handle $ drop chunkSize s
But it doesn't work either, I still get `out-of-memory` errors. From what I understand, this should work, but it isn't.
I asked on IRC and someone said "Show is not lazy enough", if that's the case, I would appreciate an explanation of that.
Thanks,
Mahdi
_______________________________________________