On Thu, 6 Dec 2018 at 11:15, Ömer Sinan Ağacan wrote:

Hi,

...

I think what we want is a way to trigger GC at very regular intervals, after (say) each 10kbytes or 100kbytes or 1Mbyte of allocation. That might be expensive, but we’d get reproducible results.

If we could fix the nursery size to 10kb that'd trigger a GC in every 10kb of allocation (you could still allocate large objects as those are not allocated in the nursery, but perhaps that's not a problem in your benchmarks). Then by setting -G1 you could turn all GCs to major GCs (because first generation is always collected). Note that because each capability has its own nursery you may want to set the nursery size to alloc_per_gc / num_of_caps if you need more than one capability.

...

I don’t think that is possible right now – see the ticket – but it would be easy enough to do wouldn’t it? Just give only 10k or 100k or 1M to the allocator when setting it running again.

Right. A parameter for fixing the nursery size would be easy to implement, I think. Just a new flag, then in GC.c:resize_nursery() use the flag as the nursery size.

It's possible that +RTS -A100k -F1 would do what you want. It would keep the 2-generation setup, with a fixed nursery size, and -F1 would ensure that every collection is a major one. I haven't tested this, but if it doesn't work, it should! Cheers Simon

"Max. residency" is really hard to measure (need to do very frequent GCs), perhaps a better question to ask is "residency when the program is in state S". This is also hard to measure if your program is threaded or have other non-determinism, but this lets you decide when to measure residency. Currently we can't tell the GC to print residency stats, but perhaps we could implement a variant of `performGC` that prints residency after the GC. So in your program you could add `performGCPrintStats` after every iteration or step etc. Not sure how useful this would be, but just an idea..

...

Simon, Ben, Omer

As you’ll see in comments 55-72 of https://ghc.haskell.org/trac/ghc/ticket/9476, Sebastian has been a bit flummoxed by the task of measure residency

On 6.12.2018 13:09, Simon Peyton Jones wrote: profiles;

...

that is, how much data is truly live during execution.

A major GC measures that, but we are vulnerable to exactly when it happens (even with -G1) and that can lead to irreproducible results.

I think what we want is a way to trigger GC at very regular intervals, after (say) each 10kbytes or 100kbytes or 1Mbyte of allocation. That might be expensive, but we’d get reproducible results.

I don’t think that is possible right now – see the ticket – but it would be easy enough to do wouldn’t it? Just give only 10k or 100k or 1M to the allocator when setting it running again.

Would you consider this? Or are we just missing something obvious?

Needless to say, we want to do all this with full optimisation on, no cost-centre profiling.

Thanks

Simon

-- Ömer Sinan Ağacan, Haskell Consultant Well-Typed LLP, http://www.well-typed.com

Registered in England & Wales, OC335890 118 Wymering Mansions, Wymering Road, London W9 2NF, England

Hey, thanks, all! Measuring with `-A1M -F1` delivers much more reliable residency numbers. `-F` doesn't seem to be documented. From reading `rts/RtsFlags.c` and `rts/sm/GC.c` I gather that it's the factor by which to multiply the number of live bytes by to get the new old gen size? So effectively, the old gen will 'overflow' on every minor GC, neat! Greetings Sebastian Am Do., 6. Dez. 2018 um 12:52 Uhr schrieb Simon Peyton Jones via ghc-devs < ghc-devs@haskell.org>:

| Right. A parameter for fixing the nursery size would be easy to implement, | I think. Just a new flag, then in GC.c:resize_nursery() use the flag as the | nursery size.

Super! That would be v useful.

| "Max. residency" is really hard to measure (need to do very frequent GCs), | perhaps a better question to ask is "residency when the program is in state | S".

Actually, Sebastian simply wants to see an accurate, reproducible residency profile, and doing frequent GCs might well be an acceptable cost.

Simon _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Ah, I was only looking at `+RTS --help`, not the users guide. Silly me. Am Do., 6. Dez. 2018 um 20:53 Uhr schrieb Simon Marlow :

It is documented! https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/runtime_cont...

On Thu, 6 Dec 2018 at 16:21, Sebastian Graf wrote:

...

Hey,

thanks, all! Measuring with `-A1M -F1` delivers much more reliable residency numbers. `-F` doesn't seem to be documented. From reading `rts/RtsFlags.c` and `rts/sm/GC.c` I gather that it's the factor by which to multiply the number of live bytes by to get the new old gen size? So effectively, the old gen will 'overflow' on every minor GC, neat!

Greetings Sebastian

Am Do., 6. Dez. 2018 um 12:52 Uhr schrieb Simon Peyton Jones via ghc-devs :

...

| Right. A parameter for fixing the nursery size would be easy to implement, | I think. Just a new flag, then in GC.c:resize_nursery() use the flag as the | nursery size.

Super! That would be v useful.

| "Max. residency" is really hard to measure (need to do very frequent GCs), | perhaps a better question to ask is "residency when the program is in state | S".

Actually, Sebastian simply wants to see an accurate, reproducible residency profile, and doing frequent GCs might well be an acceptable cost.

Simon _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

This recently came up again. It seems that `+RTS -h -i0` will just turn every minor collection into a major one: https://gitlab.haskell.org/ghc/ghc/issues/17387#note_248705 `-i0` seems significantly different from `-i0.001`, say, in that it just turns minor GCs into major ones and doesn't introduce non-determinism otherwise. Sampling rate can be controlled with `-A`, much like `-F1` (but it's still faster for some reason). Am Mo., 10. Dez. 2018 um 09:11 Uhr schrieb Simon Marlow

:

https://phabricator.haskell.org/D5428

On Sun, 9 Dec 2018 at 10:12, Sebastian Graf wrote:

...

Ah, I was only looking at `+RTS --help`, not the users guide. Silly me.

Am Do., 6. Dez. 2018 um 20:53 Uhr schrieb Simon Marlow < marlowsd@gmail.com>:

...

It is documented! https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/runtime_cont...

On Thu, 6 Dec 2018 at 16:21, Sebastian Graf wrote:

...

Hey,

thanks, all! Measuring with `-A1M -F1` delivers much more reliable residency numbers. `-F` doesn't seem to be documented. From reading `rts/RtsFlags.c` and `rts/sm/GC.c` I gather that it's the factor by which to multiply the number of live bytes by to get the new old gen size? So effectively, the old gen will 'overflow' on every minor GC, neat!

Greetings Sebastian

Am Do., 6. Dez. 2018 um 12:52 Uhr schrieb Simon Peyton Jones via ghc-devs :

...

| Right. A parameter for fixing the nursery size would be easy to implement, | I think. Just a new flag, then in GC.c:resize_nursery() use the flag as the | nursery size.

Super! That would be v useful.

| "Max. residency" is really hard to measure (need to do very frequent GCs), | perhaps a better question to ask is "residency when the program is in state | S".

Actually, Sebastian simply wants to see an accurate, reproducible residency profile, and doing frequent GCs might well be an acceptable cost.

Simon _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs