On Wed, May 6, 2009 at 3:54 PM, Anton van Straaten
FWIW, I have an internal HAppS application that's been running continuously since November last year, used daily, with stable memory usage.
Do you have advice about the way you wrote you app? Things you knowingly did to avoid space leaks? Maybe a blog about your HAppS app?
I have also experienced it with writing a forever loop in Haskell that did polling from channels. I would leave my app running for, say, 4 hours and it would be using tons of memory.
If you posted an example of that, there are probably people who'd be interested in debugging it.
I'm not sure if I still have the source code or not. I did ask people at the time and the only thing I recall now was that I was told to make sure that I fully evaluate the thunks I create on each iteration of my forever loop. This was at least 2 years ago now. I don't know if I ever resolved it as it was just a toy benchmark anyway and I wasn't really happy with the results.
I think it's fair to say that keeping the memory usage low of a long running Haskell app is hard, but that is a general issue not just a Haskell issue. It's hard in most languages.
I don't agree with this. This should not be hard in language implementations with good garbage collectors, that don't have known limitations (this excludes e.g. conservative GC and reference-counting systems). In my experience, it's not hard to write stable long-running code in good implementations of languages like Haskell, Scheme, Common Lisp, or Java.
There are certainly cases where no automatic garbage collector could know when it is safe to collect certain things. A quick google search for java space leaks turned up this article: http://www.ibm.com/developerworks/java/library/j-leaks/ I think wikipedia uses the term "logical leak" for the type of space leak I'm thinking of. The garbage collector thinks you care about an object but in fact, you want it to be freed. Yes, it's because of a bug, but these are bugs that tend to be subtle and tedious.
I think what we need to address this is more information about preventative measures. What programming styles cause the problem and which ones solve it. I would say that I lack confidence recommending anyone to use Haskell for long running processes because I don't understand well the problem of keeping the usage low. If it is a well documented problem with documented solutions (more than just profiling), then I would regain my confidence because I know the problem can be worked around reliably. Does this make sense? Maybe it's already well documented?
This doesn't completely make sense to me, in that either a program has a space leak, or it doesn't. If it does, you debug it and resolve it. If a leak really is due to a problem with the language implementation or standard libraries, then that should be identified and reported. There shouldn't be any deep mystery here. At the very least, it should be possible to point to the relevant trac tickets if there really is a problem.
The ambiguity is me thinking of relative cost of finding/fixing these bugs. Testing for correctness is something we tend to automate very well. See unit testing for example. But, testing for space leaks is not something we have automated in my experience. Take Don's comment that you let it run for a while and then look at the profiling graph. How do you automate this and make it part of your automatic test suite? It seems like something that requires manual testing. So maybe you do some QA prior to releases? That sounds reasonable to me. The darcs team collects misbehaving repositories and has tools to automate the collection of statistics so that at regular intervals they can check the performance of darcs to make sure it's getting better or staying the same with each release (in practice I think these stats are collected by buildbots on each applied patch bundle). So then, at some point we must have a bag of tricks for dealing with these space leaks. I want to talk about those tricks. I'm not talking about bugs in a specific program, but instead about techniques and styles that are known to work well in practice. I hope that clarifies things a bit. Thanks, Jason