Re: Memory usage exploding for complex pattern matching
Hello Ben,
We came up with a "minimal" example that at least is self contained and triggers the problem; `runghc Minimal.hs` should finish with about 5GB of RAM.
Hmm, with 8.2 I'm seeing a peak memory usage of 3.5 GB, with that peak occurring during renaming, not desugaring (when exhaustiveness checking happens). Are you certain that the exhaustiveness checker is the responsible party?
Not at all! In fact, I ran some comparisons with "-ddump-phases" and plotted those. I'm attaching the graphs here. From base to "added E" we added 16 more pattern matches, then each "added" phase is an additional 16 pattern matches. The simplifier stands out there. I couldn't run a bigger case as I had hit the ceiling of my machine. Cheers, Victor
"Victor Miraldo (UU)"
Hello Ben,
We came up with a "minimal" example that at least is self contained and triggers the problem; `runghc Minimal.hs` should finish with about 5GB of RAM.
Hmm, with 8.2 I'm seeing a peak memory usage of 3.5 GB, with that peak occurring during renaming, not desugaring (when exhaustiveness checking happens). Are you certain that the exhaustiveness checker is the responsible party?
Not at all! In fact, I ran some comparisons with "-ddump-phases" and plotted those. I'm attaching the graphs here. From base to "added E" we added 16 more pattern matches, then each "added" phase is an additional 16 pattern matches. The simplifier stands out there.
I couldn't run a bigger case as I had hit the ceiling of my machine.
I tried 8.4 and indeed the renamer/typechecker takes dramatically less memory (yay!), leaving desugar as the largest allocator (allocating around 3GB) while compiling the minimal example. Furthermore, disabling the pattern match checker (via -fmax-pmcheck-iterations=0) dramatically reduces desugarer allocations (to around 60MByte). It looks like you are right. Unfortunately, it's not clear what can be done to work around this short of disabling the match checker for the entire module, as suggested by mpickering. Ideally we would have some notion of local warning flags (e.g. #602), allowing us to disable this (rather expensive) warning for generated code. Sadly, we aren't there yet. Perhaps you would like to propose such a mechanism? Cheers, - Ben
Let's make a Trac ticket for this (since there is a nice small reproducer). Or, if it seems a dup, attach the reproducer to some existing ticket.
Maybe we should revert the (recent) patch that makes TH code behave like source code, until this pattern-match check business is solved.
Simon
| -----Original Message-----
| From: ghc-devs
participants (3)
-
Ben Gamari -
Simon Peyton Jones -
Victor Miraldo (UU)