
Imagine some system with hard memory bound (e.g. 64M of physical memory, no swap). I don't want some accidental laziness (maybe even not in my code, but in some used package) to crash my program. So, let's think what we can do at runtime. Suppose RTS takes the parameter -- upper limit of consumed memory. When it sees that memory consumption is close to upper bound, it can: 1. force garbage collection 2. apply some heuristics to find and reduce some chunks which will benefit from reduction in terms of size 3. if nothing helps, throw an exeption. It can be caught in IO and memory-aware program can make apropriate decision -- e.g. abort opening a large file and gracefully warn the user. (And there still is a problem of foreign code whose memory consumption we know nothing about...) What do you think? -- Roman I. Cheplyaka :: http://ro-che.info/ "Don't let school get in the way of your education." - Mark Twain

This is a problem with partitioned operating systems used in avionics. The
airplane computers require certain partitions to exist between programs in
both time and space. The space guarantees are most easily enforced by
eliminating any dynamic memory allocation once the operating system enters a
'normal mode'. This means that all memory needs to be allocated on the stack
and ahead of time.
This reduces the calculation of the upper memory bound to checking what the
largest possible stack a process could grow to is. Of course, calculating
this maximum stack is difficult in its own right.
Unfortunately, nothing I've just discussed seems to port well to Haskell (I
could be wrong!). Given this, I think you're looking at a very hard problem
(but one I'd be incredibly interested in discovering a solution for!).
/jve
On Thu, Dec 24, 2009 at 5:38 PM, Roman Cheplyaka
Imagine some system with hard memory bound (e.g. 64M of physical memory, no swap). I don't want some accidental laziness (maybe even not in my code, but in some used package) to crash my program.
So, let's think what we can do at runtime. Suppose RTS takes the parameter -- upper limit of consumed memory. When it sees that memory consumption is close to upper bound, it can:
1. force garbage collection
2. apply some heuristics to find and reduce some chunks which will benefit from reduction in terms of size
3. if nothing helps, throw an exeption. It can be caught in IO and memory-aware program can make apropriate decision -- e.g. abort opening a large file and gracefully warn the user.
(And there still is a problem of foreign code whose memory consumption we know nothing about...)
What do you think?
-- Roman I. Cheplyaka :: http://ro-che.info/ "Don't let school get in the way of your education." - Mark Twain _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

On Thu, Dec 24, 2009 at 11:38 PM, Roman Cheplyaka
So, let's think what we can do at runtime. Suppose RTS takes the parameter -- upper limit of consumed memory. When it sees that memory consumption is close to upper bound, it can:
1. force garbage collection
This is already implemented. See the -M RTS option.
2. apply some heuristics to find and reduce some chunks which will benefit from reduction in terms of size
For example, by un-evaluating them. It would often be possible to reduce size like that, just so long as they aren't evaluated again the next second. Well, this one would be hard.
3. if nothing helps, throw an exeption. It can be caught in IO and memory-aware program can make apropriate decision -- e.g. abort opening a large file and gracefully warn the user.
That should be relative simple. Get implementing, will you?:D
(And there still is a problem of foreign code whose memory consumption we know nothing about...)
In theory, you could deal with that by hooking malloc. -- Svein Ove Aas

On Fri, Dec 25, 2009 at 5:14 AM, Svein Ove Aas
On Thu, Dec 24, 2009 at 11:38 PM, Roman Cheplyaka
wrote: So, let's think what we can do at runtime. Suppose RTS takes the parameter -- upper limit of consumed memory. When it sees that memory consumption is close to upper bound, it can:
1. force garbage collection
This is already implemented. See the -M RTS option.
Correct me if I'm wrong - it's been a while since I tried to use the -M option to deal with Gitit memory usage on darcs.net - but doesn't -M just kill the program after it reaches a set RAM, and doesn't trigger any GCs? -- gwern

Am Freitag 25 Dezember 2009 15:45:29 schrieb Gwern Branwen:
On Fri, Dec 25, 2009 at 5:14 AM, Svein Ove Aas
wrote: On Thu, Dec 24, 2009 at 11:38 PM, Roman Cheplyaka
wrote: So, let's think what we can do at runtime. Suppose RTS takes the parameter -- upper limit of consumed memory. When it sees that memory consumption is close to upper bound, it can:
1. force garbage collection
This is already implemented. See the -M RTS option.
Correct me if I'm wrong - it's been a while since I tried to use the -M option to deal with Gitit memory usage on darcs.net - but doesn't -M just kill the program after it reaches a set RAM, and doesn't trigger any GCs?
It does affect GC behaviour. According to the user's guide: " -Msize [Default: unlimited] Set the maximum heap size to size bytes. The heap normally grows and shrinks according to the memory requirements of the program. The only reason for having this option is to stop the heap growing without bound and filling up all the available swap space, which at the least will result in the program being summarily killed by the operating system. The maximum heap size also affects other garbage collection parameters: when the amount of live data in the heap exceeds a certain fraction of the maximum heap size, compacting collection will be automatically enabled for the oldest generation, and the -F parameter will be reduced in order to avoid exceeding the maximum heap size." Reducing the -F parameter (amount of memory reserved for the older generations as a factor of the amount of live data) triggers more GCs. So it tries to not exceed the limit you've given. You can indeed reduce the memory used by your programme via the -M option, but only to some extent. As far as I know it will not collect data held on to by live data to be reconstructed later if needed again.
participants (5)
-
Daniel Fischer
-
Gwern Branwen
-
John Van Enk
-
Roman Cheplyaka
-
Svein Ove Aas