Releasing head of lazy ByteString
I have a streaming network protocol where each message in the stream is prefixed by a 64-bit message length marker. Lazy ByteStrings seem to be an elegant way to wrap network communications, so I'm using them. I have one concern though: how can I prevent my program from hanging on to the beginning of the stream? My code looks roughly like this: lazy <- getContents clientSock let (lenBS, rest1) = splitAt 8 lazy let length = runGet getWord64be lenBS let (msg, rest2) = splitAt (fromIntegral length) rest1 -- process msg The program never uses that initial "lazy" again, but it's there, and it's been assigned, so I assume that reference to the head of the stream will always be around, and thus always consuming memory. Is there a way to indicate to haskell that I want to "forget" about that assignment, so the runtime doesn't need to keep the stream around? In this case, I think I could do the stream processing in a tail-recursive loop, and thus lose the previous references to the stream, but is there a more specific way to tell haskell that I'm done with some data?
On Sun, Feb 20, 2011 at 9:38 AM, tsuraan
I have a streaming network protocol where each message in the stream is prefixed by a 64-bit message length marker. Lazy ByteStrings seem to be an elegant way to wrap network communications, so I'm using them. I have one concern though: how can I prevent my program from hanging on to the beginning of the stream? My code looks roughly like this:
lazy <- getContents clientSock let (lenBS, rest1) = splitAt 8 lazy let length = runGet getWord64be lenBS let (msg, rest2) = splitAt (fromIntegral length) rest1 -- process msg
The program never uses that initial "lazy" again, but it's there, and it's been assigned, so I assume that reference to the head of the stream will always be around, and thus always consuming memory. Is there a way to indicate to haskell that I want to "forget" about that assignment, so the runtime doesn't need to keep the stream around? In this case, I think I could do the stream processing in a tail-recursive loop, and thus lose the previous references to the stream, but is there a more specific way to tell haskell that I'm done with some data?
One easy trick is to put the rest of the function body in a separate top-level function, so that the 'lazy' variable is no longer in scope. Antoine
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
On Sunday 20 February 2011 16:38:17, tsuraan wrote:
I have a streaming network protocol where each message in the stream is prefixed by a 64-bit message length marker. Lazy ByteStrings seem to be an elegant way to wrap network communications, so I'm using them. I have one concern though: how can I prevent my program from hanging on to the beginning of the stream? My code looks roughly like this:
lazy <- getContents clientSock let (lenBS, rest1) = splitAt 8 lazy let length = runGet getWord64be lenBS let (msg, rest2) = splitAt (fromIntegral length) rest1 -- process msg
The program never uses that initial "lazy" again, but it's there, and it's been assigned, so I assume that reference to the head of the stream will always be around, and thus always consuming memory.
No, that need not be. If the compiler sees that it's never referenced again, it can be garbage collected (assuming the consumption pattern of msg allows that to be garbage collected incrementally). Try out whether your programme has a space leak by giving it a long input. If it has, ask again.
Is there a way to indicate to haskell that I want to "forget" about that assignment, so the runtime doesn't need to keep the stream around? In this case, I think I could do the stream processing in a tail-recursive loop, and thus lose the previous references to the stream, but is there a more specific way to tell haskell that I'm done with some data?
Normally, that shouldn't be necessary. If you're done with some data, it should be garbage collected pretty soon. There are some situations where an unintended and unneeded reference to some data is kept around and prevents it from being collected, but in general, the compilers are rather good at managing memory.
No, that need not be. If the compiler sees that it's never referenced again, it can be garbage collected (assuming the consumption pattern of msg allows that to be garbage collected incrementally).
Try out whether your programme has a space leak by giving it a long input. If it has, ask again.
Ok, I'll see if I have any problems. What I'm doing seemed like it might be the wrong way to go, so I figured I'd ask about it to see if I got any "don't do that" type responses. Sounds like what I'm doing is pretty much sane, so I'll see if I run into any trouble.
Normally, that shouldn't be necessary. If you're done with some data, it should be garbage collected pretty soon. There are some situations where an unintended and unneeded reference to some data is kept around and prevents it from being collected, but in general, the compilers are rather good at managing memory.
Thanks for the info. I'll be back if I have any trouble :)
On Sun, Feb 20, 2011 at 8:16 AM, tsuraan
Ok, I'll see if I have any problems. What I'm doing seemed like it might be the wrong way to go, so I figured I'd ask about it to see if I got any "don't do that" type responses. Sounds like what I'm doing is pretty much sane, so I'll see if I run into any trouble.
It's sane but it's fiddly and error-prone, which is why a lot of
people tend to be using iteratees (using either the iteratee or
enumerator packages) for this kind of thing these days.
G
--
Gregory Collins
On Sun, Feb 20, 2011 at 03:38:17PM +0000, tsuraan wrote:
lazy <- getContents clientSock let (lenBS, rest1) = splitAt 8 lazy let length = runGet getWord64be lenBS let (msg, rest2) = splitAt (fromIntegral length) rest1 -- process msg
The program never uses that initial "lazy" again, but it's there, and it's been assigned, so I assume that reference to the head of the stream will always be around, and thus always consuming memory.
The compiler should be able to figure it out, but if not you could
rewrite the first two lines as:
(lenBS, rest1) <- splitAt 8 `fmap` getContents clientSock
--
Helgi Kristvin Sigurbjarnarson
participants (5)
-
Antoine Latter -
Daniel Fischer -
Gregory Collins -
Helgi Kristvin Sigurbjarnarson -
tsuraan