Saurabh Nanda
Did you ever make any progress on this, Saurabh?
We made progress in some sense, by introducing a separate `stack build -j1` step in our CI pipeline for compiling packages that are known to use a lot of memory.
* -j just tells GHC to parallelise compilation across modules. This can increase the maximum heap size needed by the compiler.
From the docs, it wasn't very clear to me how -j interacts with -M when both the options are passed to the GHC process. Is it the max heap size across all build, or per build?
The short answer is that they don't interact: -j is a GHC flag whereas -M is an RTS flag. -M controls the amount of heap that the RTS allows the mutator (GHC in this case) to allocate. This includes all threads. GHC when run with -j is just like any other threaded Haskell program.
* -M is a bit tricky to define. For one, it defines the maximum heap size beyond which we will terminate. However, we also use it in garbage collector to make various decisions about GC scheduling. I'll admit that I'm not terribly familiar with the details here.
Note that -M does not guarantee that GHC will find a way to keep your program under the limit that you provide. It merely ensures that the program doesn't exceed the given size, aborting if necessary.
Quoting from https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/runtime_cont... :
*> The maximum heap size also affects other garbage collection parameters: when the amount of live data in the heap exceeds a certain fraction of the maximum heap size, compacting collection will be automatically enabled for the oldest generation, and the -F parameter will be reduced in order to avoid exceeding the maximum heap size.*
It just makes it sound that the RTS is going to tweak the GC algo, and the number of time GC is run, to avoid crossing the heap limit. However, I've found the GHC process easily consuming more memory than what is specified in the -M flag (as reported by top).
Yes, as I mentioned we do tweak some things in the GC; however, these tweaks are really a best-effort attempt to avoid going over the limit. It's entirely possible that your mutator will be terminated if it wants to use significantly more than the limit set with -M. There is relatively little else the RTS can do in this case that wouldn't require explicit cooperation from the mutator to keep working sets down. Cheers, - Ben