Dear haskellers, I'd like assistance figuring out a strange space leak using zlib package. To make it easier to reproduce I've come up with the following snippet that pretty much resumes up the problem I'm trying to solve:
-- omitting imports and function signatures encode = compress . L.fromChunks
main = do { hSetBinaryMode stdout True ; hSetBinaryMode stdin True ; hSetBuffering stdin NoBuffering ; hSetBuffering stdout NoBuffering ; loop [] } where loop buff | length buff == 64 = L.hPut stdout (encode buff) >> loop [] | otherwise = do { eof <- hIsEOF stdin ; when (not eof) (fmap (: buff) (B.hGetSome stdin 512) >>= loop) }
N.B.: Removing the `compress' function from the above code also removes the space leak. Now, feeding the above program with ~8GB worth of binary data:
$ uname -a Linux mephisto.localhost.localdomain 3.8.7-1-ARCH #1 SMP PREEMPT Sat Apr 13 09:01:47 CEST 2013 x86_64 GNU/Linux $ ghc --version The Glorious Glasgow Haskell Compilation System, version 7.6.3 $ ghc -W -Wall -rtsopts --make -O2 test.hs $ sudo dd if=/dev/sda bs=4K count=2048K | ./test +RTS -M1M -s >/dev/null ... 8589934592 bytes (8.6 GB) copied, 277.263 s, 31.0 MB/s ... 2 MB total memory in use (0 MB lost due to fragmentation) ...
Which is fine. However, monitoring the RSS size is a different story:
$ while pidof test; do ps -o rss= -p $(pidof test); done | tail 16967 25620 16967 25628 16967 25628 16967 25628 16967 # ~16M 0
I know the RSS usually overestimates the memory consumption but the problem is that it is forever growing. The following I found very intriguing: * `+RTS -hc` gives me no hint about whats wrong [at least I couldn't see one]; * `+RTS -M1M` doesn't produce an error; * removing the `compress' functions makes the problem disappear; I couldn't figure these out and I don't think this is matter of strictness, though. Has anyone seen this before? Thanks in advance, ~dsouza