Daniel Carrera writes:

1) You say that grammar doesn't matter. Well, for some languages it matters more than others. German, for example, has a very particular word order that takes some effort to learn, and if you get it wrong people really won't understand you. In German it's ok if you conjugate wrong, but it's not ok if you put words in the wrong place. Second, some people actually enjoy grammar better and find that grammar helps them understand the language. I am one of those people. Different people learn differently. I learn rules more easily than disconnected words. When I learn vocabulary I do better by learning word families, and so on. The Germanic languages rely heavily in word derivation (not so much English) so that can be important for learners like me.

I haven't taken German into consideration. Perhaps I need to re-evaluate or restate my conviction. Maybe you can help me find a better way of putting it. The idea is that I spent years studying different languages, generally with a textbook. The textbooks tend to focus on teaching rules and grammar, with a little bit of vocabulary and dialog each chapter. I think the focus should be reversed. Obviously grammar is very important. But is reading about it effective for everyone? I know that some people enjoy studying formal grammars, myself included. I consider myself to be a highly logically-oriented (audio-digital?) learning type, as I expect many programmers are. However, I still don't remember most grammar lessons. The only way I successfully became fluent in a language (Esperanto) was through immersion, and that wouldn't have been possible without a decent vocabulary to start with. Fortunately Esperanto has a lot of English cognates and you can build a large vocabulary with it pretty quickly. And of course it has very forgiving sentence and a rather simple grammar, but I'm finding the experience to be very similar with Japanese so far. That being said, Esperanto, and even Japanese sentence structure perhaps is not as different as an agglutinative language like German. I'll need to study it more to find out.

2) Your analysis of word count is flawed. Sure, most of the words you read come from a very small vocabulary set, but most of the *meaning* in a sentence comes from the more obscure words. Imagine that you read this sentence: "In the newspaper I read that the __________ said that the problem is that the river has too much ________ ". In this sentence you can understand 90% of the words, but you have almost no idea of what's happening. What your word count test really shows is that human languages have a lot of redundancy. You could omit the word "the" from the above sentence and you would understand it almost as well. The word "the" is common and contains very little information.

Absolutely. I'm not trying to claim that you only need 1,000 words to become fluent, like some courses claim. I do think though that if you focus on particles, common verbs, etc. up front you'll get to immersive learning much faster. Again, this has been my personal experience. The idea is that once you can begin to read with a dictionary by your side you'll begin learning much faster because you can focus on reading what *you* are interested in rather than some contrived dialog from a textbook.

That said, do you have any stories in German? I can't figure out where to get the stories.

So far I've been focusing on Japanese. I only have 15 or so stories currently. They take a bit of time to create ;) For now, the navigation is basically to click the "Latest Links" link in the header bar or in the "Latest Links" box. Thank you very much for the feedback. I appreciate it, and I'll take what you've said into consideration when I rewrite the front page.

Chris Forno (jekor) wrote:

The idea is that I spent years studying different languages, generally with a textbook. The textbooks tend to focus on teaching rules and grammar, with a little bit of vocabulary and dialog each chapter. I think the focus should be reversed.

This varies wildly by textbook, with some bias for the language being taught. Personally I've found too many vocabulary textbooks and far too few grammar textbooks (that is, actual *grammar* textbooks not sentence-sized-vocabulary textbooks).

Obviously grammar is very important. But is reading about it effective for everyone?

In my experience learning and teaching languages, this too varies wildly by learner. Some people do better with an "examples first" or "vocabulary based" style where they must come to an intuition of the grammar rules; other people (such as myself) do better with a "rules first" or "grammar based" style where they must come to learn vocabulary on their own. Neither variety of person is superior nor, as far as I can tell, more common at large; so any good teacher or textbook should balance these "bottom up" and "top down" approaches. IMO vocabulary is easy to learn, it just takes time, whereas grammar is harder to figure out on one's own, and so is the better thing for a teacher to focus on. However, this says little about reference material (as opposed to learning material), and study guides walk a line between reference and teaching. JGram http://jgram.org/pages/viewList.php is an interesting study guide that takes a middle path, treating syntactic patterns the same as it does lexemes. This is particularly appropriate for a language like Japanese where it's not always immediately apparent whether something belongs to the "grammar" vs the "lexicon".

The only way I successfully became fluent in a language (Esperanto) was through immersion,

This is, hands down, the best way to learn any language. For it to work, as you say, some vocabulary is necessary; however, I think the amount of vocabulary needed at first is not so large as some think. Daily small-talk for getting/giving directions, ordering food, and the like comprise a large portion of beginner's language and requires remarkably little breadth of vocabulary (a couple hundred words or so). Small-talk also includes some of the most obscure and difficult-to-master grammatical patterns like greetings, getting the right tone of politeness/familiarity, and knowing what sorts of sentence fragments and other "ungrammatical" patterns are perfectly acceptable.

And of course it has very forgiving sentence and a rather simple grammar, but I'm finding the experience to be very similar with Japanese so far.

That being said, Esperanto, and even Japanese sentence structure perhaps is not as different as an agglutinative language like German. I'll need to study it more to find out.

Actually, Japanese is agglutinative too (moreso than German is). The basic structures of Japanese are quite simple, however the details needed for fluency are quite intricate. Phrase order is rather free, though it is not entirely free and it is easy to reorder things so that they no longer make sense to native speakers. Aside from a few of the common mistakes beginners make, if you mess up the cases/particles you'll end up with gibberish. -- Live well, ~wren

fft1976:

I've heard it's hard to contain a long-running Haskell application in a finite amount of memory, but this is probably not a problem if your

Hmm. Gossip driven development?

web site sleeps 0.001% of the time (like XMonad), or you can restart it every once in a while without anyone noticing.

Keeping footprints small and stable isn't so hard. After all, we have wonderful heap profiling tools. I recommend the type-based heap profiler, in particular. Run your app for a week, and look at the heap graph. You'll know if things are ok. (Try this in C++ !) -- Don

Jason Dagit wrote:

I don't mean to undermine your marketing efforts, but I don't think this is gossip driven.

I know from experience that lambdabot tends to be leaky. Otherwise, lambdabot wouldn't be running on my server to begin with. And, even so, Cale monitors lambdabot to make sure it is not using too many resources (and I complain when/if I notice it). I have heard similar stories related to hpaste and happs. I have also experienced it with writing a forever loop in Haskell that did polling from channels. I would leave my app running for, say, 4 hours and it would be using tons of memory.

I think it's fair to say that keeping the memory usage low of a long running Haskell app is hard, but that is a general issue not just a Haskell issue. It's hard in most languages. I think what we need to address this is more information about preventative measures. What programming styles cause the problem and which ones solve it. I would say that I lack confidence recommending anyone to use Haskell for long running processes because I don't understand well the problem of keeping the usage low. If it is a well documented problem with documented solutions (more than just profiling), then I would regain my confidence because I know the problem can be worked around reliably. Does this make sense? Maybe it's already well documented?

In particular, we need expert Haskell programmers, such as Don, to write more about how they avoid space leaks in long running apps. Again, profiling is nice, but that's more of a tuning effort.

Mmm, interesting. Clearly I don't run anything for long enough to notice this effect. (I just had a look at a Haskell program I wrote myself which I happen to have running in the background right now. Process Explorer tells me it's used 4 hours of CPU time, and it's memory graph is still flat. It was reading 106 MB shortly after I started it, and it still says 106 MB now. OTOH, it's a fairly trivial program, so...) But if you're going to advocate more expert knowledge being diseminated, I certainly won't argue against that! :-D Export knowledge is something it seems difficult to have too much of...

On Wed, May 6, 2009 at 2:28 PM, Don Stewart wrote:

dagit:

...

In particular, we need expert Haskell programmers, such as Don, to write more about how they avoid space leaks in long running apps. Again, profiling is nice, but that's more of a tuning effort.

I talk a bit about that in my LondonHUG talk:

   http://www.galois.com/blog/2009/04/27/engineering-large-projects-in-haskell-...

As I said earlier: stress test with heap profiling on (is one way to absolutely ensure you know what's going on).

I wish I could have gone to the talk. I have a feeling the slides only tell part of the story. I did see these bullet points which I think are related: This one on Laziness: • Makes time and space reasoning harder! –Mostly harmless in practice – Stress testing tends to reveal retainers – Graphical profiling knocks it dead • Must be able to precisely enable/disable • Be careful with exceptions and mutation • whnf/rnf/! are your friends This one on performance: • Really precise performance requires expertise • Libraries are helping reify “oral traditions” about optimization • Still a lack of clarity about performance techniques in the broader Haskell community though The last bullet point makes me think you do agree with me :) I thought about it more since my last email, and I think what I want is something like the Effective C++ and Effective Java books, but for Haskell. A series of expert advice "items" with advice about when to use them. Real-World Haskell is a very good title, but I feel like it's at a different audience and level. Looking over Real-World haskell I see that some of these topics are discussed, which is really good. In particular, Chapter 25 would be valuable to anyone trying to find space leaks. There you discuss reduction to normal form, for example, and some strictness issues and how to control evaluation. While I'm thinking out loud, it would be very cool if someone wrote some articles, say for the monad reader, that follow the formula of the Effective C++ books. Write up the oral traditions of how to effectively use Haskell along with the dos/don'ts of each idiom. I think it could make a nice companion to the wisdom of Real-World Haskell. Have we made any tools yet that analyze haskell code and give warnings about styles and idioms that are prone to causing space leaks? For example, leaky folds are a well known problem. I know Neil Mitchel was working on some utilities to offer advice about Haskell source. Anyone know if his programs can detect potential space leaks? Thanks, Jason

Bryan O'Sullivan wrote:

Jason Dagit wrote:

...

While I'm thinking out loud, it would be very cool if someone wrote some articles, say for the monad reader, that follow the formula of the Effective C++ books.

The last couple of times I've wanted a book like that, I wrote the book myself. It's a very effective way to get the book you want, compared to wishing.

There is of course the dilemma that writing such a book requires a thorough understanding of the subject matter, which one intends to acquire by reading the book in the first place. I see a _|_ lurking there. :) Regards, apfelmus -- http://apfelmus.nfshost.com

Jason Dagit wrote:

I know from experience that lambdabot tends to be leaky. Otherwise, lambdabot wouldn't be running on my server to begin with. And, even so, Cale monitors lambdabot to make sure it is not using too many resources (and I complain when/if I notice it). I have heard similar stories related to hpaste and happs.

FWIW, I have an internal HAppS application that's been running continuously since November last year, used daily, with stable memory usage.

I have also experienced it with writing a forever loop in Haskell that did polling from channels. I would leave my app running for, say, 4 hours and it would be using tons of memory.

If you posted an example of that, there are probably people who'd be interested in debugging it.

I think it's fair to say that keeping the memory usage low of a long running Haskell app is hard, but that is a general issue not just a Haskell issue. It's hard in most languages.

I don't agree with this. This should not be hard in language implementations with good garbage collectors, that don't have known limitations (this excludes e.g. conservative GC and reference-counting systems). In my experience, it's not hard to write stable long-running code in good implementations of languages like Haskell, Scheme, Common Lisp, or Java.

I think what we need to address this is more information about preventative measures. What programming styles cause the problem and which ones solve it. I would say that I lack confidence recommending anyone to use Haskell for long running processes because I don't understand well the problem of keeping the usage low. If it is a well documented problem with documented solutions (more than just profiling), then I would regain my confidence because I know the problem can be worked around reliably. Does this make sense? Maybe it's already well documented?

This doesn't completely make sense to me, in that either a program has a space leak, or it doesn't. If it does, you debug it and resolve it. If a leak really is due to a problem with the language implementation or standard libraries, then that should be identified and reported. There shouldn't be any deep mystery here. At the very least, it should be possible to point to the relevant trac tickets if there really is a problem. Anton

Jason Dagit wrote:

On Wed, May 6, 2009 at 3:54 PM, Anton van Straaten wrote:

...

FWIW, I have an internal HAppS application that's been running continuously since November last year, used daily, with stable memory usage.

Do you have advice about the way you wrote you app? Things you knowingly did to avoid space leaks? Maybe a blog about your HAppS app?

The app is written for a client under NDA, so a blog about it would have to be annoyingly vague. But I don't think there's much mystery about why it doesn't leak: The app does simulations. Each simulation uses at least about 10MB of memory, more depending on parameters. Typically a few thousand simulations are run successively, and the results are aggregated and analyzed. The computation itself is purely functional - it takes some input parameters and produces results. The results are written to a file. Since each run of a set of simulations is essentially independent, there's not much risk of space leaks persisting across runs. No doubt the potential for encountering space leaks goes up as one writes less pure code, persist more things in memory, and depend on more libraries. My main point in mentioning my app is that "long-running" isn't really the issue - that's just a way of saying that an app has space leaks that are small enough not to be noticed until it's stressed.

...

In my experience, it's not hard to write stable long-running code in good implementations of languages like Haskell, Scheme, Common Lisp, or Java.

There are certainly cases where no automatic garbage collector could know when it is safe to collect certain things.

If there are bugs in the user's program, sure - but that still doesn't make it "hard" to write applications that don't leak, given a decent GC. On the contrary, I'd say it's very easy, in the great majority of cases.

A quick google search for java space leaks turned up this article: http://www.ibm.com/developerworks/java/library/j-leaks/

I think wikipedia uses the term "logical leak" for the type of space leak I'm thinking of. The garbage collector thinks you care about an object but in fact, you want it to be freed. Yes, it's because of a bug, but these are bugs that tend to be subtle and tedious.

The example given in the IBM article is quite typical, but isn't subtle at all - it was simply an object being added to a table and never being removed. You can often find such bugs quite easily by searching the source tree, without touching a debugging tool. It's also possible to prevent them quite easily, with good coding practices (e.g. centralize uses of long-lived tables) and some simple code auditing practices. If you're dealing with code that's complex enough to involve the kinds of non-trivial mutually dependent references that you need in order to encounter truly subtle instances of these bugs, the increased difficulty of memory management comes with the territory, i.e. it's harder because the application is harder.

The ambiguity is me thinking of relative cost of finding/fixing these bugs.

To put this back into context, I was objecting to your having extended the space leak worrying to all GC'd languages. I'm saying that it isn't hard, using most decent language implementations, to avoid space leaks. For trivial cases such as the IBM example, it should be no harder in Haskell, either - possibly easier, since use of things like mutable tables is more controlled, and may be rarer. However, Haskell does theoretically introduce a new class of dangers for space leaks, I'm not denying that. Being pure and lazy introduces its own set of space leak risks. But on that front, I was disturbed by the vagueness of the claims about long-running apps. I haven't seen any solid justification for scaring people off about writing long-running apps in Haskell. If there is such a justification, it needs to be more clearly identified.

Testing for correctness is something we tend to automate very well.

How do you automate testing for performance under load? Space usage is a similar kind of dynamic issue, in general.

So then, at some point we must have a bag of tricks for dealing with these space leaks. I want to talk about those tricks. I'm not talking about bugs in a specific program, but instead about techniques and styles that are known to work well in practice.

OK. That's a bit different from FFT's original contention, "hard to contain a long-running Haskell application in a finite amount of memory." For my own part, I'm at least as non-strict as Haskell, and that bag of tricks, for me, is a thunk that hasn't yet been forced. Anton

FFT wrote:

Anton van Straaten wrote:

...

The app is written for a client under NDA, so a blog about it would have to be annoyingly vague.

...

No doubt the potential for encountering space leaks goes up as one writes less pure code, persist more things in memory, and depend on more libraries.

Exactly. I'm worried about, e.g. needing to use something as simple as a stream of prime numbers (see the recent thread about leaks there)

The issues here are going to be the same in Haskell as in every other language. There's always a tradeoff between the memory of caching old results vs the time of recalculating them. At present no language's RTS/GC is smart enough to make that tradeoff for you, and so memoization must be done manually. There are some tricks to help make this easier (e.g. weak pointers), but the problem will always remain. The only thing that makes this perhaps trickier than in other languages is that, AFAIK/R, the reflection API to ask the RTS how it's feeling about memory at any given point isn't terribly portable (between Haskell compilers) or polished/pretty. Then again, the other GC languages I've dealt with aren't much better and are often worse. -- Live well, ~wren

Actually, I'm unsure how to fix this. For an expression like this: Data.Map.delete key map how can I use seq (or something else) to sufficiently evaluate the above to ensure that the value associated with key can be garbage collected? My knowledge of Data.Map is limited to it's haddock documentation. -----Original Message----- From: haskell-cafe-bounces@haskell.org [mailto:haskell-cafe-bounces@haskell.org] On Behalf Of Tim Docker Sent: Thursday, 7 May 2009 6:04 PM To: haskell-cafe@haskell.org Subject: RE: [Haskell-cafe] Is Haskell a Good Choice for WebApplications?(ANN: Vocabulink) I think that multi-threading in combination with laziness makes space usage harder to manage. In fact, just today I have discovered a problem with a long running server process with a subtle space leak. With a regular process that communicates with the outside world via IO, I know that the act of communicating a value causes it to be fully evaluated. However, with a multi threaded process, communication occurs via writes to TVars/IOVars and nothing gets evaluated. This gives lots of opportunities for space leaks. In this particularly case cleanup code was removing a large entry from a map stored in a Tvar. Because that map is only read infrequently, however, the memory release is delayed. This is the second such problem I've found. The profiling tools do help in discovering them, but it still needs a bit of thought and analysis. I wonder if, for my application, I should work out some means of deepseqing every value written to a Tvar. Tim -----Original Message----- From: haskell-cafe-bounces@haskell.org [mailto:haskell-cafe-bounces@haskell.org] On Behalf Of wren ng thornton Sent: Thursday, 7 May 2009 2:06 PM To: haskell-cafe@haskell.org Subject: Re: [Haskell-cafe] Is Haskell a Good Choice for Web Applications?(ANN: Vocabulink) FFT wrote:

Anton van Straaten wrote:

...

The app is written for a client under NDA, so a blog about it would have to be annoyingly vague.

...

No doubt the potential for encountering space leaks goes up as one writes less pure code, persist more things in memory, and depend on more libraries.

Exactly. I'm worried about, e.g. needing to use something as simple as

a stream of prime numbers (see the recent thread about leaks there)

The issues here are going to be the same in Haskell as in every other language. There's always a tradeoff between the memory of caching old results vs the time of recalculating them. At present no language's RTS/GC is smart enough to make that tradeoff for you, and so memoization must be done manually. There are some tricks to help make this easier (e.g. weak pointers), but the problem will always remain. The only thing that makes this perhaps trickier than in other languages is that, AFAIK/R, the reflection API to ask the RTS how it's feeling about memory at any given point isn't terribly portable (between Haskell compilers) or polished/pretty. Then again, the other GC languages I've dealt with aren't much better and are often worse. -- Live well, ~wren _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

seq something like size map that will force a traversal of the entire tree, and ensure that the result is actually demanded, .. (Not tested)

and not recommended, either, I'm afraid!-) |> Actually, I'm unsure how to fix this. For an expression like this: |> |> Data.Map.delete key map |> |> how can I use seq (or something else) to sufficiently evaluate the above |> to ensure that the value associated with key can be garbage collected? You can ensure that the Map update no longer holds on to the old key by keeping a copy of the old Map in an unevaluated thunk, but for garbage collection, you'd also have to make sure that there are no other unused references to the old Map, and no other references to the value in that old key. I assume you knew that, but the discussion suggested that we should keep making such things explicit. |> My knowledge of Data.Map is limited to it's haddock documentation. That won't be sufficient in practice, eg., most of the documentation is silent about strictness properties. If you are willing to look a little bit under the hood, GHCi's :info is your friend: Prelude> :m +Data.Map Prelude Data.Map> :info Map data Map k a = Data.Map.Tip | Data.Map.Bin !Data.Map.Size !k a !(Map k a) !(Map k a) -- Defined in Data.Map ... You can see that the constructors are not exported (GHCi reports them qualified, even though we've brought Data.Map into scope), so any use you make of their strictness properties is version dependent. They are not likely to change soon, and should probably be documented in the haddock API specification (at least for the abstract constructors that are actually exported), so that we can rely on them with clear conscience, but that isn't quite the case at the moment. Anyway;-) You can see that size is actually pre-computed, so there's no guarantee that asking for it will traverse the internal representation, the element type is not stored strictly (which isn't all that unexpected, but sometimes surprises because it isn't documented), and everything else is strict. So, with the current representation, if you force the Map, its whole structure will be forced, though none of its elements will be. Hth, Claus PS. GHood is useful for observing dynamic strictness properties (what is inspected when, with what dependencies and results), though it won't help you directly with garbage collection issues (it can't observe when a value gets collected).

FFT wrote:

On Wed, May 6, 2009 at 8:20 PM, Anton van Straaten wrote:

...

The app is written for a client under NDA, so a blog about it would have to be annoyingly vague.

...

No doubt the potential for encountering space leaks goes up as one writes less pure code, persist more things in memory, and depend on more libraries.

Exactly. I'm worried about, e.g. needing to use something as simple as a stream of prime numbers (see the recent thread about leaks there)

Haskell lets you easily create "infinite" lists, which is a powerful and useful feature. But if you abuse that feature, you'll have problems, because you don't have infinite memory. The further you traverse an infinite list, while maintaining a reference to its head, the more memory you use. It'd be a stretch to characterize this as "hard". I don't see much connection between this and the space efficiency of long-running programs. I feel a bit like we're discussing a monster under the bed: "i bet it's huge! its teeth must be so sharp!" Maybe we should just look under the bed, i.e. implement what we need and see what happens? Anton

I think that multi-threading in combination with laziness makes space usage harder to manage. In fact, just today I have discovered a problem with a long running server process with a subtle space leak. With a regular process that communicates with the outside world via IO, I know that the act of communicating a value causes it to be fully evaluated. However, with a multi threaded process, communication occurs via writes to TVars/IOVars and leaves thunks unevaluated. This gives lots of opportunities for space leaks. In this particularly case cleanup code was removing a large entry from a map stored in a Tvar. Because that map is only read infrequently, however, the memory release is delayed. This is the second such problem I've found. The profiling tools do help in discovering them, but it still needs a bit of thought and analysis. I wonder if, for my application, I should work out some means of deepseqing every value written to a Tvar. Tim -----Original Message----- From: haskell-cafe-bounces@haskell.org [mailto:haskell-cafe-bounces@haskell.org] On Behalf Of Anton van Straaten Sent: Thursday, 7 May 2009 4:28 PM To: haskell-cafe@haskell.org Subject: Re: [Haskell-cafe] Is Haskell a Good Choice for Web Applications?(ANN: Vocabulink) FFT wrote:

On Wed, May 6, 2009 at 8:20 PM, Anton van Straaten wrote:

...

The app is written for a client under NDA, so a blog about it would have to be annoyingly vague.

...

No doubt the potential for encountering space leaks goes up as one writes less pure code, persist more things in memory, and depend on more libraries.

Exactly. I'm worried about, e.g. needing to use something as simple as

a stream of prime numbers (see the recent thread about leaks there)

Haskell lets you easily create "infinite" lists, which is a powerful and useful feature. But if you abuse that feature, you'll have problems, because you don't have infinite memory. The further you traverse an infinite list, while maintaining a reference to its head, the more memory you use. It'd be a stretch to characterize this as "hard". I don't see much connection between this and the space efficiency of long-running programs. I feel a bit like we're discussing a monster under the bed: "i bet it's huge! its teeth must be so sharp!" Maybe we should just look under the bed, i.e. implement what we need and see what happens? Anton _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Well this is interesting. So what you are saying is that if your haskell application requires a peek memory utilisation of (for example) 1GB, after the memory intesive computation has completed and the GC has run (assuming all references have been dropped) the GHC RTS will retain the 1GB allocated to the process. Does this occur on both windows and posix platforms, does anyone know ? If so then this is a real issue. It would be reasonable to expect that the RTS release resources to the OS when not explicitly required. jvl ----- Original Message ----- From: "Bulat Ziganshin" To: "FFT" Cc: "Chris Forno" ; ; Sent: Thursday, May 07, 2009 6:18 AM Subject: [Haskell] Re[2]: [Haskell-cafe] Is Haskell a Good Choice for WebApplications? (ANN: Vocabulink)

Hello FFT,

Wednesday, May 6, 2009, 11:59:53 PM, you wrote:

...

I've heard it's hard to contain a long-running Haskell application in a finite amount of memory

not exactly. you may alloc fixed pool of memory to application (say, 1gb) if you know that it never need more memory. but as far as you don't do it, memory usage grows with each major GC. ghc just don't provide any way to return memory to OS (there is a ticket on it, you can add yourself to CC list to vote for its resolution)

-- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

_______________________________________________ Haskell mailing list Haskell@haskell.org http://www.haskell.org/mailman/listinfo/haskell

wren ng thornton writes:

FWIW, the JVM also fails to release memory resources back to the OS. Given all the problems I've seen that one cause for long-running processes, I'm definitely in support of correcting any behavior like this in the GHC RTS.

I'm curious what real problems arise from this? As far as I can tell, virtual memory and overcommit ought to make this (i.e. not releasing memory back to the OS) mostly harmless. Is this on particular systems with different properties than your average run-of-the-mill Unix? -k -- If I haven't seen further, it is by standing in the footprints of giants

Ketil Malde wrote:

wren ng thornton writes:

...

FWIW, the JVM also fails to release memory resources back to the OS. Given all the problems I've seen that one cause for long-running processes, I'm definitely in support of correcting any behavior like this in the GHC RTS.

I'm curious what real problems arise from this? As far as I can tell, virtual memory and overcommit ought to make this (i.e. not releasing memory back to the OS) mostly harmless. Is this on particular systems with different properties than your average run-of-the-mill Unix?

Consider a process, A, which goes through a cycle of spawning off another process, B, and reaping its output. More specifically, A consumes a lot of memory when B is not running, but then flushes most of it before spawning B and then idles until B is done. Also B consumes a lot of memory (which is always returned to the system each time B is terminated). If A cannot return memory to the system prior to spawning B, this will cause the OS to over-commit memory, which will cause various OSes to either kill the process, start thrashing, or other Bad Things(tm). I'd have to ask my coworker which particular systems caused the most problems, but we target all of Linux, OSX, and Windows for the toolkit that includes both A and B. -- Live well, ~wren

Hello Simon, Thursday, May 7, 2009, 2:04:05 PM, you wrote:

...

...

I've heard it's hard to contain a long-running Haskell application in a finite amount of memory

not exactly. you may alloc fixed pool of memory to application (say, 1gb) if you know that it never need more memory. but as far as you don't do it, memory usage grows with each major GC. ghc just don't provide any way to return memory to OS (there is a ticket on it, you can add yourself to CC list to vote for its resolution)

http://hackage.haskell.org/trac/ghc/ticket/698

But let's be clear: this is not a memory leak, the issue is only that GHC's runtime retains as much memory as was required by the program's high-water-mark memory usage. Fixing this will never reduce the high-water-mark. So I'm not sure what you meant by "memory usage grows with each major GC".

1. none said that it's memory leak, at least in haskell code 2. it seems that one of us doesn't know intimate details of GHC GC :) the following is my understanding, learned from +RTS -s/-S listings with ghc 6.4 or so: copying GC: let's imagine that we perform major GC when 100 mb is allocated, of those 10 mb are live. doing GC, ghc will alloc 10+ mb from OS, and promote 100 mb freed to allocation area. so from this point program will occupy 110mb of memory (although only 10 mb is really used ATM) next major GC will occur when all these 100 mb will be allocated, i.e. overall memory allocated will be 110 mb. again, GHC will allocate as much memory as required for live data, increasing program footprint compacting GC: if major GC occurs when 100 mb is allocated, GHC increase memory footprint 2x (controlled by +RTS -F), and then perform GC. or it will perform GC and only then alloc more memory? i don't remember Simon, it will be VERY USEFUL if GC description that says about these behaviors will be added to manual. i'm definitely not the authoritative source on this topic and it seems that even you forget some details as time goes. this topic is very important for large programs and this question regularly arises here. we need authoritative description that pays attention to all GC-related RTS switches -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Hello Simon, Thursday, May 7, 2009, 5:45:53 PM, you wrote:

out of date and say 256k, I've just fixed that). The old generation is allowed to grow to 2x its previous size by default before being collected.

you are right. i just checked old logs - seems that previously i just misinterpreted them

...

we need authoritative description that pays attention to all GC-related RTS switches

I'm happy to add more text here, what else would you like to see?

how about text that describes GC scenario like you was wrote in this letter? plus, description of RTS +S report format - it may be added to this scenario, i.e. "after this major GC, the following line will be printed to +S report: ..." for example, i still doesn't understand why old_live + alloc - collect != new_live in the report below (i extracted two sequential lines): Alloc Collect Live GC GC TOT TOT Page Flts bytes bytes bytes user elap user elap 265016 303104 840008 0.00 0.00 0.11 0.17 0 0 (Gen: 0) 265612 266240 904196 0.00 0.00 0.11 0.17 0 0 (Gen: 0) and completely separate topic - +RTS -s report description also doesn't exist -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Hello Simon, Thursday, May 7, 2009, 6:58:02 PM, you wrote:

...

and completely separate topic - +RTS -s report description also doesn't exist

Scroll down in that section I linked to before:

http://www.haskell.org/ghc/docs/latest/html/users_guide/runtime-control.html...

you are right again. so, that remains: you shouldn't suppose that user have read 90's GC paper. give a short excerpt of it: how generational GC works and how memory usage converts to memory footprint. then descriptions of RTS options will serve as "modifications" to this "baseline" one minor update: manual said "For each garbage collection, we print: * How many bytes we allocated this garbage collection." i think it should be "How many bytes we allocated *SINCE* last garbage collection." -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Hello Jason, Thursday, May 7, 2009, 9:06:33 PM, you wrote:

...

you are right again. so, that remains: you shouldn't suppose that user have read 90's GC paper. give a short excerpt of it: how generational GC works and how memory usage converts to memory footprint. then descriptions of RTS options will serve as "modifications" to this "baseline"

Perhaps you could write a description here to the list or the wiki and after a few iterations it could be included in the GHC manual?

ok, once i will find spare time -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Am Donnerstag 07 Mai 2009 22:01:11 schrieb Andrew Coppin:

Simon Marlow wrote:

...

http://hackage.haskell.org/trac/ghc/ticket/698

I presume that the reason for this is to avoid handing memory back only to immediately need it again? (I.e., we don't want to be constantly asking the OS to allocate and deallocate memory. Allocate it once and then let the RTS handle it.)

How hard would it be to add a function to ask the RTS to shrink the allocated memory? E.g., you do something that you know consumes lots of RAM, you finish doing it, you know that your live set has probably gone way down now, so you ask the RTS to release some RAM if possible. Would that be difficult?

Another idea, I have no idea how hard or sensible it is: What if the GC detects how much memory is currently used (it does already do that, doesn't it?) and how much is allocated from the OS, and if less than a third or a quarter of the allocated memory is used, return the memory that exceeds twice the used memory to the OS? Behaviour might be configurable by an RTS flag.

on the other hand a function to release pool memory to the OS down to the current active level should (I hope) be easily implementable, and quickly incorporated into application where required, whereas arriving at one or more automatic deallocation policies would most likely require some analysis/trial and error. ----- Original Message ----- From: "wren ng thornton" To: Sent: Friday, May 08, 2009 9:33 AM Subject: Re: [Haskell-cafe] GC

Daniel Fischer wrote:

...

Am Donnerstag 07 Mai 2009 22:01:11 schrieb Andrew Coppin:

...

...

http://hackage.haskell.org/trac/ghc/ticket/698 I presume that the reason for this is to avoid handing memory back only to immediately need it again? (I.e., we don't want to be constantly asking the OS to allocate and deallocate memory. Allocate it once and

Simon Marlow wrote: then let the RTS handle it.)

How hard would it be to add a function to ask the RTS to shrink the allocated memory? E.g., you do something that you know consumes lots of RAM, you finish doing it, you know that your live set has probably gone way down now, so you ask the RTS to release some RAM if possible. Would that be difficult?

Another idea, I have no idea how hard or sensible it is: What if the GC detects how much memory is currently used (it does already do that, doesn't it?) and how much is allocated from the OS, and if less than a third or a quarter of the allocated memory is used, return the memory that exceeds twice the used memory to the OS? Behaviour might be configurable by an RTS flag.

+1 (assuming the GC does detect current usage).

I think this behavior (with flags to tune the multiples, or to disable it) is nicely high-level and easy for users to reason about.

For users with intimate knowledge of the GC and their program's behavior it could be helpful to have the alternative proposal of an impure function to give hints to the RTS, but in general this seems like far too low-level of an approach to solve the general problem.

-- Live well, ~wren _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

On Fri, May 8, 2009 at 2:37 PM, Andrew Coppin wrote:

John Lask wrote:

...

on the other hand a function to release pool memory to the OS down to the current active level should (I hope) be easily implementable, and quickly incorporated into application where required, whereas arriving at one or more automatic deallocation policies would most likely require some analysis/trial and error.

I would suggest that trying to do this "automatically" in a way that is optimal for all applications requires some fairly serious heuristics.

OTOH, for many Haskell programs it isn't a problem in the first place. (E.g., if you write something like Darcs which only ever runs for a few seconds, you barely need GC in the first place.)

That's not entirely true. Darcs has a fair number of operations that stream data and hence work well because the RTS can recycle the memory and keep the usage at a constant. I think I see your point and agree in principle though. A similar argument can be applied to laziness. Often times you end up evaluating most or all of the thunks, so what difference did it make? But, I guess it's a bit of an oversimplification in both cases.

I'm just thinking, the application writer probably knows more about their specific program than the compiler designers do, so it would be nice to be able to provide hints to the RTS.

Definitely. Jason