Hi, since my application takes up way to much memory, i decided to profile it (compile d it with -prof and -auto-all). After running it with +RTS -hc, it shows a graph that reaches a peak at around 85mb. However, when i check the memory usage with 'top' i see the application reach over 300mb. What am i missing here? PS the application is a large attribute grammar that performs program analysis on an AST that represents an application. regards, Niels.
Niels
since my application takes up way to much memory, i decided to profile it (compile d it with -prof and -auto-all). After running it with +RTS -hc, it shows a graph that reaches a peak at around 85mb. However, when i check the memory usage with 'top' i see the application reach over 300mb.
Are you checking 'top' on the profiled version? It will show more heap usage than the profile itself, because some heap space is taken up by the profiling data itself. But of course the profile is only supposed to report the heap space that is used in normal usage, excluding the overhead used to build the report. The profiling overhead can certainly double space usage, and with cost-centre stacks, it could be even higher. Don't forget to subtract the static size of the executable too. If, however, you are comparing /unprofiled/ 'top' with the profile, it suggests a fault in ghc's profiling code. Regards, Malcolm
Malcolm Wallace
After running it with +RTS -hc, it shows a graph that reaches a peak at around 85mb. However, when i check the memory usage with 'top' i see the application reach over 300mb.
I may be wrong, but I think profiling only tells you the size of live data for your program. The program itself, libraries, etc will not be counted. The program is also likely to allocate more memory from the OS than it is using, either because it is keeping dead data around between collections or because OS allocation has a certain granularity. (To check the former, you can perhaps try limiting the heap size.) I'm not sure allocated memory is ever returned to the OS - I tried strace'ing for brk(2) without getting much wiser. My experience is that top will report quite a bit more than profiling. 300Mb seems like a bit much for 85Mb heap, though. -k -- If I haven't seen further, it is by standing in the footprints of giants
Hello Malcolm,
Friday, September 09, 2005, 2:21:51 AM, you wrote:
MW> Niels
since my application takes up way to much memory, i decided to profile it (compile d it with -prof and -auto-all). After running it with +RTS -hc, it shows a graph that reaches a peak at around 85mb. However, when i check the memory usage with 'top' i see the application reach over 300mb.
MW> If, however, you are comparing /unprofiled/ 'top' with the profile, it MW> suggests a fault in ghc's profiling code. i don't know what value shows the -xc oprion, but memory usage in ghc-compiled program are increased because by default used "copying" garbage collection algorithm. using option "+RTS -c" will significantly reduce memory usage, which will make program faster if it currently needs swapping. you can also force, for example, using of compacting algorithm when memory usage grows more than 200 mb with options "+RTS -M1G -c20". see "4.14.2. RTS options to control the garbage collector" in ghc docs for details btw, i suggested to GHC team auto-switching to compacting algorithm when program begins swapping. you can vote for this suggestion if it what you need ;) -- Best regards, Bulat mailto:bulatz@HotPOP.com
On Fri, 2005-09-09 at 12:42 +0400, Bulat Ziganshin wrote:
btw, i suggested to GHC team auto-switching to compacting algorithm when program begins swapping. you can vote for this suggestion if it what you need ;)
That's much easier said than done. Automatically picking an optimal heap size for the current machine VM pressure is not so easy. Most OSs do not give any advice to apps on how much memory they can sensibly make use of or even tell you in any sensible manner when they are starting to thrash. Duncan
Hello Duncan, Friday, September 09, 2005, 5:17:06 PM, you wrote:
btw, i suggested to GHC team auto-switching to compacting algorithm when program begins swapping. you can vote for this suggestion if it what you need ;)
DC> That's much easier said than done. Automatically picking an optimal heap DC> size for the current machine VM pressure is not so easy. Most OSs do not DC> give any advice to apps on how much memory they can sensibly make use of DC> or even tell you in any sensible manner when they are starting to DC> thrash. in windows it can be made, by example, simply by comparing value of available physical memory at moment of starting GC with size of heap. if availbale physical memory is smaller, then it have meaning to start compacting algorithm instead of copying. compacting algorithm runs about 2 times slower if there is enough memory, but when swapping occurs it will be many times faster, of course you can see that dicussion with SimonM in this maillist under subject "garbage collection" -- Best regards, Bulat mailto:bulatz@HotPOP.com
On Fri, 2005-09-09 at 19:54 +0400, Bulat Ziganshin wrote:
Hello Duncan,
Friday, September 09, 2005, 5:17:06 PM, you wrote:
btw, i suggested to GHC team auto-switching to compacting algorithm when program begins swapping. you can vote for this suggestion if it what you need ;)
DC> That's much easier said than done. Automatically picking an optimal heap DC> size for the current machine VM pressure is not so easy. Most OSs do not DC> give any advice to apps on how much memory they can sensibly make use of DC> or even tell you in any sensible manner when they are starting to DC> thrash.
in windows it can be made, by example, simply by comparing value of available physical memory at moment of starting GC with size of heap. if availbale physical memory is smaller, then it have meaning to start compacting algorithm instead of copying. compacting algorithm runs about 2 times slower if there is enough memory, but when swapping occurs it will be many times faster, of course
It's a reasonable aproximation but it doesn't take into account any other programs running on the machine at the same time. And there's also the fact that VM pressure is rather dynamic in nature. Duncan
Hello Duncan, Friday, September 09, 2005, 8:38:14 PM, you wrote: DC> It's a reasonable aproximation but it doesn't take into account any DC> other programs running on the machine at the same time. And there's also DC> the fact that VM pressure is rather dynamic in nature. you are absolutely right, but at some time i was rescued from fear against unexact algorithms :) nevertheless, this unexact behaviour will be not very good for some purposes - memory usage of program will be dependent from environment/moment of time when program is runned -- Best regards, Bulat mailto:bulatz@HotPOP.com
Bulat Ziganshin wrote:
i don't know what value shows the -xc oprion, but memory usage in ghc-compiled program are increased because by default used "copying" garbage collection algorithm. using option "+RTS -c" will significantly reduce memory usage, which will make program faster if it currently needs swapping. you can also force, for example, using of compacting algorithm when memory usage grows more than 200 mb with options "+RTS -M1G -c20". see "4.14.2. RTS options to control the garbage collector" in ghc docs for details running the profilingbinary (that was compiled with -prof -auto-all) with the -c option resulted in;
top: showing a max of 161mb memory usage profiled-graph: showing a peak at a little more than 80mb i guess these results are more or less normal? It does however mean that my datastructures are pretty harsh on the garbage collector, doesnt it? Niels.
Hello Niels, Friday, September 09, 2005, 6:58:13 PM, you wrote: N> running the profilingbinary (that was compiled with -prof -auto-all) N> with the -c option resulted in; N> top: showing a max of 161mb memory usage N> profiled-graph: showing a peak at a little more than 80mb N> i guess these results are more or less normal? compile the program with profiling disabled and check it with `top` N> It does however mean that N> my datastructures are pretty harsh on the garbage collector, doesnt it? sorry, i don't understand your question, my english is poor enough. i can recommend you to check memory usage of program without profiling code with and without "+RTS -c" and compare it to memory typically available when you run your program. if larger value (memory requirements when using "copying" GC algorithm, used by default) are larger than memory available then you can go much faster by adding "+RTS -c" to cmdline (this can be also incorporated to executable itself). you can father speed up your program by incorporating conditional selection of GC algorithm (see my previous message) btw, you can see memory usage of program with "-s", "-S" or "-t" RTS options (see details in ghc docs) - it may be more convenient than using `top` -- Best regards, Bulat mailto:bulatz@HotPOP.com
participants (5)
-
Bulat Ziganshin -
Duncan Coutts -
Ketil Malde -
Malcolm Wallace -
Niels