Re: [Haskell-cafe] Memory consumption issues under heavy network throughput/concurrency loads

On Jul 15, 2014, at 1:49 PM, Gregory Collins wrote:

+kazu, +Simon M

On Tue, Jul 15, 2014 at 7:18 PM, Ben Bangert wrote: I have created a very simple TCP based echo client/server here: https://github.com/bbangert/echo

Ben, could you please tell us what sort of machine you are running this on? Is it Mac, Linux, or Windows?

It occurs on my Mac, and it occurs on the linux AWS AMI instances I've compiled/run it on.

I took your test case and hacked it down to eliminate some possible sources of error (I was suspicious of Handle for a while, also of an old space leak bug in "forever" which I think is fixed now):

https://github.com/gregorycollins/ghc-echo-leak-bug

I should note the use of ByteString's was intentional (re: dropping hGetLine in the server), this is because many of the networking libs use ByteString's and I'd prefer not to rewrite all the libs from the TCP layer up to the Websocket layer. This stack of layers.... TCP - HTTP - Websocket all introduce various objects as they slice/dice the bytes into websocket frames using ByteStrings. It's quite possible the issue I'm seeing is related to how GHC handles the introduction of huge amounts of ByteString's that then must all be GC'd (esp regarding how the profile shows memory being lost to fragmentation). Someone on the #haskell channel suggested that due to how ByteString's are implemented, GHC is unable to move them around in memory which can lead to the fragmentation.

The revised echoserver is fine on my machine (stable at 22MB resident) but the echo client leaks. Happens with/without -O2 on GHC 7.8.3 for OSX.

Yes, I noticed the client leak, though since I'm mainly interested in running the server that didn't concern me as much.

Kazu, I think there's a good chance this is a bug in the multicore IO manager, the test code is doing little more than write + read + threadDelay.

Thanks a lot for looking at this, hopefully the ByteString issue is known or possible to fix short of looking at the larger program I have that exhibits it (which is all open-source so I can post that as well, the 'simple' websocket server and websocket tester looks like this: https://gist.github.com/bbangert/6b7ef979963d7cb1838e) using the yesod-websocket code from the https://github.com/yesodweb/yesod/commit/66437453f57e6a2747ff7c2199aa7ad25db... changeset). I'll see about better packaging the more involved websocket one that leaks/fragments memory substantially faster. Cheers, Ben