Pattern matching desugaring regression? Re: [Haskell-cafe] Why does my module take so long to compile?

Ccing ghc devs since that’s a better forum perhaps
Crazy theory:
this is a regression due the the partial changes to pattern matching
coverage checking in 8.10 that finished / landed in ghc 9
Why:
Desugaring is when pattern/ case statement translation happens I think?
And the only obvious “big thing” is that you have some huge , albeit sane
for a compiler, pattern matching
I’d first check if the new ghc 9 release doesn’t have that regression in
build time that you experienced. And if it does file a ticket.
I may be totally wrong, but that seems like a decent likelihood !
On Mon, Feb 15, 2021 at 12:20 PM Troels Henriksen
Carter Schonwald
writes: How big are your data types in the file In question? Do you ghc generics or deriving or template Haskell? Could you share a link to the file in question ?
The file does not define any large data types itself, but it operates on some fairly large data types (an AST, it's a compiler). But so do many other modules that work just fine. It uses no generics, nontrivial deriving, or Template Haskell. It's this file:
https://github.com/diku-dk/futhark/blob/master/src/Futhark/Pass/ExtractKerne...
I also found a handful of other modules in my project that are significantly slower to compile in GHC 8.10, and seemingly also because of the desugarer, but none of them have any obvious smoking guns like generics or TH.
The only commonality I can find is that the affected modules contain functions with a relatively large typeclass context. I use ConstraintKinds to make them more concise, and I guess unfolded there may be 30-40 equality/class constraints in total. Like this:
type DistLore lore = ( Bindable lore, HasSegOp lore, BinderOps lore, LetDec lore ~ Type, ExpDec lore ~ (), BodyDec lore ~ () )
where the 'Bindable' constraint in particular then implies a further handful of "~" constraints:
class ( ASTLore lore, FParamInfo lore ~ DeclType, LParamInfo lore ~ Type, RetType lore ~ DeclExtType, BranchType lore ~ ExtType, SetType (LetDec lore) ) => Bindable lore where ...
FParamInfo/LParamInfo/etc are type families. Are such constraints particularly slow?
-- \ Troels /\ Henriksen

Carter Schonwald
Ccing ghc devs since that’s a better forum perhaps Crazy theory:
this is a regression due the the partial changes to pattern matching coverage checking in 8.10 that finished / landed in ghc 9
Why: Desugaring is when pattern/ case statement translation happens I think? And the only obvious “big thing” is that you have some huge , albeit sane for a compiler, pattern matching
I’d first check if the new ghc 9 release doesn’t have that regression in build time that you experienced. And if it does file a ticket.
I may be totally wrong, but that seems like a decent likelihood !
You may be right! Another module that regressed is also mainly characterised by large-but-not-insane case expressions: https://github.com/diku-dk/futhark/blob/d0839412bdd11884d75a1494dd5de5191833... I'll try to split these modules up a little bit (I should have done so a while ago anyway) and maybe that will make the picture even clearer. -- \ Troels /\ Henriksen

Hi, I'm not sure I see all the context of the conversation, but it is entirely possible that code with many local constraints regresses the pattern-match checker (which is accounted to Desugaring in the profile emitted by -v2), I'm afraid. That simply has to do with the fact that we now actually care about them, previously they were mostly discarded. I'd be glad if you submitted a relatively isolated reproducer of what is fast with 8.8 and slow with 8.10 (even better 9.0). I hope that things have improved since we fixed https://gitlab.haskell.org/ghc/ghc/-/issues/17836, which is part of 9.0 but not of 8.10. Cheers, Sebastian Am Mo., 15. Feb. 2021 um 19:04 Uhr schrieb Troels Henriksen < athas@sigkill.dk>:
Carter Schonwald
writes: Ccing ghc devs since that’s a better forum perhaps Crazy theory:
this is a regression due the the partial changes to pattern matching coverage checking in 8.10 that finished / landed in ghc 9
Why: Desugaring is when pattern/ case statement translation happens I think? And the only obvious “big thing” is that you have some huge , albeit sane for a compiler, pattern matching
I’d first check if the new ghc 9 release doesn’t have that regression in build time that you experienced. And if it does file a ticket.
I may be totally wrong, but that seems like a decent likelihood !
You may be right! Another module that regressed is also mainly characterised by large-but-not-insane case expressions:
https://github.com/diku-dk/futhark/blob/d0839412bdd11884d75a1494dd5de5191833...
I'll try to split these modules up a little bit (I should have done so a while ago anyway) and maybe that will make the picture even clearer.
-- \ Troels /\ Henriksen _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

It is very likely that issue 17386 is the issue. With
{-# OPTIONS_GHC -Wno-overlapping-patterns -Wno-incomplete-patterns
-Wno-incomplete-uni-patterns -Wno-incomplete-record-updates #-}
my module(s) compile very quickly. I'll wait and see if GHC 9 does
better before I try to create a smaller case (and now I at least have a
workaround).
Sebastian Graf
Hi,
I'm not sure I see all the context of the conversation, but it is entirely possible that code with many local constraints regresses the pattern-match checker (which is accounted to Desugaring in the profile emitted by -v2), I'm afraid. That simply has to do with the fact that we now actually care about them, previously they were mostly discarded.
I'd be glad if you submitted a relatively isolated reproducer of what is fast with 8.8 and slow with 8.10 (even better 9.0). I hope that things have improved since we fixed https://gitlab.haskell.org/ghc/ghc/-/issues/17836, which is part of 9.0 but not of 8.10.
Cheers, Sebastian
Am Mo., 15. Feb. 2021 um 19:04 Uhr schrieb Troels Henriksen < athas@sigkill.dk>:
Carter Schonwald
writes: Ccing ghc devs since that’s a better forum perhaps Crazy theory:
this is a regression due the the partial changes to pattern matching coverage checking in 8.10 that finished / landed in ghc 9
Why: Desugaring is when pattern/ case statement translation happens I think? And the only obvious “big thing” is that you have some huge , albeit sane for a compiler, pattern matching
I’d first check if the new ghc 9 release doesn’t have that regression in build time that you experienced. And if it does file a ticket.
I may be totally wrong, but that seems like a decent likelihood !
You may be right! Another module that regressed is also mainly characterised by large-but-not-insane case expressions:
https://github.com/diku-dk/futhark/blob/d0839412bdd11884d75a1494dd5de5191833...
I'll try to split these modules up a little bit (I should have done so a while ago anyway) and maybe that will make the picture even clearer.
-- \ Troels /\ Henriksen _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
-- \ Troels /\ Henriksen

Don’t forget ghc 9 is already out! :)
On Mon, Feb 15, 2021 at 2:10 PM Troels Henriksen
It is very likely that issue 17386 is the issue. With
{-# OPTIONS_GHC -Wno-overlapping-patterns -Wno-incomplete-patterns -Wno-incomplete-uni-patterns -Wno-incomplete-record-updates #-}
my module(s) compile very quickly. I'll wait and see if GHC 9 does better before I try to create a smaller case (and now I at least have a workaround).
Sebastian Graf
writes: Hi,
I'm not sure I see all the context of the conversation, but it is entirely possible that code with many local constraints regresses the pattern-match checker (which is accounted to Desugaring in the profile emitted by -v2), I'm afraid. That simply has to do with the fact that we now actually care about them, previously they were mostly discarded.
I'd be glad if you submitted a relatively isolated reproducer of what is fast with 8.8 and slow with 8.10 (even better 9.0). I hope that things have improved since we fixed https://gitlab.haskell.org/ghc/ghc/-/issues/17836, which is part of 9.0 but not of 8.10.
Cheers, Sebastian
Am Mo., 15. Feb. 2021 um 19:04 Uhr schrieb Troels Henriksen < athas@sigkill.dk>:
Carter Schonwald
writes: Ccing ghc devs since that’s a better forum perhaps Crazy theory:
this is a regression due the the partial changes to pattern matching coverage checking in 8.10 that finished / landed in ghc 9
Why: Desugaring is when pattern/ case statement translation happens I think? And the only obvious “big thing” is that you have some huge , albeit sane for a compiler, pattern matching
I’d first check if the new ghc 9 release doesn’t have that regression in build time that you experienced. And if it does file a ticket.
I may be totally wrong, but that seems like a decent likelihood !
You may be right! Another module that regressed is also mainly characterised by large-but-not-insane case expressions:
https://github.com/diku-dk/futhark/blob/d0839412bdd11884d75a1494dd5de5191833...
I'll try to split these modules up a little bit (I should have done so a while ago anyway) and maybe that will make the picture even clearer.
-- \ Troels /\ Henriksen _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
-- \ Troels /\ Henriksen

They already said something about waiting on dependencies to catch up with
ghc9, IIRC.
On Mon, Feb 15, 2021 at 2:22 PM Carter Schonwald
Don’t forget ghc 9 is already out! :)
On Mon, Feb 15, 2021 at 2:10 PM Troels Henriksen
wrote: It is very likely that issue 17386 is the issue. With
{-# OPTIONS_GHC -Wno-overlapping-patterns -Wno-incomplete-patterns -Wno-incomplete-uni-patterns -Wno-incomplete-record-updates #-}
my module(s) compile very quickly. I'll wait and see if GHC 9 does better before I try to create a smaller case (and now I at least have a workaround).
Sebastian Graf
writes: Hi,
I'm not sure I see all the context of the conversation, but it is entirely possible that code with many local constraints regresses the pattern-match checker (which is accounted to Desugaring in the profile emitted by -v2), I'm afraid. That simply has to do with the fact that we now actually care about them, previously they were mostly discarded.
I'd be glad if you submitted a relatively isolated reproducer of what is fast with 8.8 and slow with 8.10 (even better 9.0). I hope that things have improved since we fixed https://gitlab.haskell.org/ghc/ghc/-/issues/17836, which is part of 9.0 but not of 8.10.
Cheers, Sebastian
Am Mo., 15. Feb. 2021 um 19:04 Uhr schrieb Troels Henriksen < athas@sigkill.dk>:
Carter Schonwald
writes: Ccing ghc devs since that’s a better forum perhaps Crazy theory:
this is a regression due the the partial changes to pattern matching coverage checking in 8.10 that finished / landed in ghc 9
Why: Desugaring is when pattern/ case statement translation happens I think? And the only obvious “big thing” is that you have some huge , albeit sane for a compiler, pattern matching
I’d first check if the new ghc 9 release doesn’t have that regression in build time that you experienced. And if it does file a ticket.
I may be totally wrong, but that seems like a decent likelihood !
You may be right! Another module that regressed is also mainly characterised by large-but-not-insane case expressions:
https://github.com/diku-dk/futhark/blob/d0839412bdd11884d75a1494dd5de5191833...
I'll try to split these modules up a little bit (I should have done so
a
while ago anyway) and maybe that will make the picture even clearer.
-- \ Troels /\ Henriksen _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
-- \ Troels /\ Henriksen
_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
-- brandon s allbery kf8nh allbery.b@gmail.com

Troels Henriksen
It is very likely that issue 17386 is the issue. With
{-# OPTIONS_GHC -Wno-overlapping-patterns -Wno-incomplete-patterns -Wno-incomplete-uni-patterns -Wno-incomplete-record-updates #-}
my module(s) compile very quickly. I'll wait and see if GHC 9 does better before I try to create a smaller case (and now I at least have a workaround).
I have now tried it with GHC 9, and unfortunately it is still very slow. As time permits, I will see if I can come up with a self-contained module that illustrates the slowdown. I do have an idea for a optimisation: In all of the cases where coverage tracking takes a long time, I have a catch-all case at the bottom. I think that is a fairly common pattern, where a program tries to detect various special cases, before falling back to a general case. Perhaps coverage checking should have a short-circuiting check for whether there is an obvious catch-all case, and if so, not bother looking any closer? -- \ Troels /\ Henriksen

Hi Troels, Sorry to hear GHC 9 didn't fix your problems! Yes, please open an issue. Optimising for specific usage patterns might be feasible, although note that most often it's not the exhaustivity check that is causing problems, but the check for overlapping patterns. At the moment the checker doesn't take shortcuts if we have -Wincomplete-patterns, but -Wno-overlapping-patterns. Maybe it could? Let's see what is causing you problems and decide then. Cheers, Sebastian Am So., 28. März 2021 um 15:44 Uhr schrieb Troels Henriksen < athas@sigkill.dk>:
Troels Henriksen
writes: It is very likely that issue 17386 is the issue. With
{-# OPTIONS_GHC -Wno-overlapping-patterns -Wno-incomplete-patterns -Wno-incomplete-uni-patterns -Wno-incomplete-record-updates #-}
my module(s) compile very quickly. I'll wait and see if GHC 9 does better before I try to create a smaller case (and now I at least have a workaround).
I have now tried it with GHC 9, and unfortunately it is still very slow.
As time permits, I will see if I can come up with a self-contained module that illustrates the slowdown.
I do have an idea for a optimisation: In all of the cases where coverage tracking takes a long time, I have a catch-all case at the bottom. I think that is a fairly common pattern, where a program tries to detect various special cases, before falling back to a general case. Perhaps coverage checking should have a short-circuiting check for whether there is an obvious catch-all case, and if so, not bother looking any closer?
-- \ Troels /\ Henriksen
participants (4)
-
Brandon Allbery
-
Carter Schonwald
-
Sebastian Graf
-
Troels Henriksen