Hello John, Thursday, March 16, 2006, 4:00:34 AM, you wrote:

...

i suggest checking of AVAILABLE physical ram, that is perfectly possible in windows

JM> the problem is that available physical ram is wasted ram. any good os JM> will never let there be any available ram because it will fill it up JM> with disk buffers if nothing else. yes, but nevertheless really good operating systems report the avail ram what includes cache buffers. for not-so-good OSes disk cache can be a good measure ;) JM> the only time actual available ram JM> appears is when a big application quits, but then it quicky fills up due JM> to normal disk activity. determining how much of those buffers/cached JM> info you can push out without adversly affecting system performance or JM> thrashing is not very easy so at best we get a sort of approximation. John, not all algorithms are well determined. what we need is just to get "better behavior" in the most of cases JM> the best solution would be for ghc's garbage collector to periodically JM> call getrusage(2) and look at the number of page faults and swaps it is JM> causing and switch to compaction when it sees a jump in that. I don't JM> know what the equivalent routine on windows would be. not all operating JM> systems fill in all the values of getrusage, BSD is the best, linux is JM> the worst at this. but on linux you can get similar info from JM> /proc/self. (i think it does fill in the values we need in getrusage JM> though, I forget which ones it doesn't have) i agree tha this should be even better. windows have such mechanism. but it will be interesting if all our optimization tricks will work only with superuser privileges :) -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

I think I complained earlier about '+RTS -MxxxM' not being respected, but was unable to reproduce the issue. I just saw this again, my process was, I thought, limited to 800M heap, but, just before I gave up and killed the process, 'top' told me: 18580 ketil 18 0 1762m 945m 256 D 3.0 93.5 33:52.81 rbr So it used more than double the amount of memory. However, running the same executable with the same -M option on a different computer, it stayed firmly at 32139 ketil 25 0 779M 779M 872 R 99.8 9.9 16:17 0 rbr Apparently, there is some subtle difference between the two systems that causes the RTS's heap limitation to fail. The uname -a on the two systems say: -M works: Linux ... 2.4.21-32.0.1.ELsmp #1 SMP Wed May 25 13:51:11 EDT 2005 x86_64 x86_64 x86_64 GNU/Linux -M fails: Linux ... 2.6.9-22.EL #1 Sat Oct 8 17:48:27 CDT 2005 i686 i686 i386 GNU/Linux Let me know if there's any other information I can supply. -k -- If I haven't seen further, it is by standing in the footprints of giants

Ketil Malde wrote:

Ketil Malde writes:

...

18580 ketil 18 0 1762m 945m 256 D 3.0 93.5 33:52.81 rbr

So it used more than double the amount of memory.

I can provide the source, but perhaps I should mention that the program basically just builds a large "Map Int Int". No tricky FFI, arrays or anything, at least not explicitly in my code.

Is it reproducible? I expect that the -M value might be exceeeded by a small amount sometimes, but double is surprising. Please supply the source, it saves time trying to reproduce. Cheers, Simon

Hello John, Thursday, March 16, 2006, 4:00:34 AM, you wrote: JM> the best solution would be for ghc's garbage collector to periodically JM> call getrusage(2) and look at the number of page faults and swaps it is JM> causing and switch to compaction when it sees a jump in that. just some more thoughts about this: 1. this can be implemented at first time just as user-level routines that controls ghc's internal variables which controls GC. Simon one time said how these vars can be used. after thorough testing and may be even comparison of different strategies this can go into the official RTS. one my idea was even allowing user-defined GC strategies by evaluation user-specified routines before each minor gc. i think that user-defined gc strategies may be a big win both for further developments in this area and for implementing custom strategies for large applications. making some simple official that allows to control these variables, perform minor gc and call some routine before each gc should be helpful in further development of new algorithms of optimal GCs usage 2. we can develop even better strategies by combining the 4 types of major GCs: 1) no major GCs at all 2) copying 3) copying with with returning memory to OS 4) compacting and switching between these 4 strategies according to external conditions. when there are enough memory, we can just don't perform major GCs (i know that this is debatable, but typical way to make programs faster is to use large -A or -H values). when heap size is close to size of available physical ram, we should perform at least copying GC. and when page faults grows, we should use only compacting algorithm 3. JM> the best solution would be for ghc's garbage collector to periodically JM> call getrusage(2) and look at the number of page faults and swaps it is JM> causing and switch to compaction when it sees a jump in that. and moreover, we should perform compaction immediately when swapping grows. imagine for example algorithm that had 80 mb residency and runs on a machine with 100 mb free. performing compacting GC each time when memory usage grows to 100 mb should be better strategy than waiting for 160 mb usage. of course, concrete calculation whether to start GC or not, is a delicate task, but at least it should be performed on each minor GC 4. i still don't understand - wgen GHC performs copying GC, whether it copies data from old pages sequential or not? if it releases old pages sequentially, we can run copying GC using the same amount of physical ram as required for compaction gc -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Simon Marlow writes:

Bulat Ziganshin wrote:

...

and moreover, we should perform compaction immediately when swapping grows. imagine for example algorithm that had 80 mb residency and runs on a machine with 100 mb free. performing compacting GC each time when memory usage grows to 100 mb should be better strategy than waiting for 160 mb usage. of course, concrete calculation whether to start GC or not, is a delicate task, but at least it should be performed on each minor GC

I'd like to urge you to have a go at implementing some of these strategies.

I must admit that I am slightly skeptical to complicated heuristics here, since it seems actual performance would depend a lot on memory manangement policies of the underlying OS. However, I find that setting GHC to use the equivalent of +RTS -Mxxx where xxx is about 80% of physical RAM generally works well. Currently, I do this through a C file reading values from sysconf(), which is a bit cumbersome. A way to set these defaults though the ghc command line would be welcome. -k -- If I haven't seen further, it is by standing in the footprints of giants