On Wed, Jul 16, 2014 at 5:28 PM, Ben Bangert
Yes, with the changes I don't see leaking behavior. As I mentioned in a separate email though, this isn't very useful because the various network libraries in the wild (Warp, Yesod, Websockets, etc) all use ByteStrings for reading frames, etc. on the way to/from the socket. These seem to cause memory fragmentation and issues reclaiming memory, which is the real issue I'm seeing.
OK good, that is a data point. BTW the master branch of my version of this test is still using bytestrings -- the main things that I replaced were the use of Handle from the high-level "Network" module and the use of async to run the threads --- not because I thought there was a problem with async but simply to rule out potentially confounding factors. Your issue may not be with bytestring fragmentation but that is one of the working hypothesis and now we are closer to discovering the issue. Personally I am suspicious that network's Handle API (which, by the way, there is almost no reason to use --- libraries like io-streams, pipes, and conduit provide a better user experience) might be the thing that is leaking RAM. (+johan) If the issue is GHC pinned heap fragmentation then you can try generating bytestrings with system malloc instead (which is what we are currently doing for network reads in io-streams and also IIRC what warp is doing). If doing that and linking your binary with e.g. tcmalloc fixes the issue, then I think that would probably be conclusive evidence. Your test has removed the usage of several components that were most likely
part of the problem I've had, but which I can't really remove from the fully functioning application as it would require rewriting libraries all the way down the stack.
Of course, but we are trying to find the leak here. Now we can re-add things to the test, starting with going back to Handle.
• The biggest thing you're spending RAM on here is stacks for the threads you create. By default the stack chunk size is 32k, you can lower this with +RTS -kcXX --- using 2kB stacks both programs use <40MB heap
BTW I lied about this, that parameter controls how much stacks grow if they overflow, the default stack size for threads is 1kB. Besides for not closing the sockets, your master branch had excellent
memory usage. I unfortunately wasn't able to try your io-streams branch as I got this compile error: https://gist.github.com/bbangert/d26f28b410faaad4e8d2
This thread is just full of fun, isn't it.
G
--
Gregory Collins