- ghc-devs - Haskell.org

RE: 8.2.1-rc2 upgrade report
by Simon Peyton Jones 13 Jun '17

13 Jun '17

Great. Could you put all this into a Trac ticket? Thanks! Simon From: Alberto Valverde [mailto:alberto@toscat.net] Sent: 08 June 2017 13:57 To: Simon Peyton Jones <simonpj(a)microsoft.com> Cc: GHC users <glasgow-haskell-users(a)haskell.org> Subject: Re: 8.2.1-rc2 upgrade report Hi Simon, Thanks for the pointer. I re-did both builds with -dshow-passes and made a small script to plot the results of the lines which summarize the elapsed time and allocated memory per phase and module. I've uploaded the raw logs, a plot of the results and the script I wrote to generate it to https://gist.githubusercontent.com/albertov/145ac5c01bfbadc5c9ff55e9c5c2e50e<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.githu…>. The plotted results live here https://gist.githubusercontent.com/albertov/145ac5c01bfbadc5c9ff55e9c5c2e50…<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.githu…> Apparently, the biggest slowdown in respect to 8.0.2 seems to occur in the SpecConstr and Simplifier passes in the Propag (where the "main" function is) and the Sigym4.Propag.Engine (where the main algorithm lives) modules. Any other tests that would be helpful for me to run? I'm not sure where to start to create a reproducible case but I'll see if I can come up with something soon... Alberto On Tue, Jun 6, 2017 at 1:58 PM, Simon Peyton Jones <simonpj(a)microsoft.com<mailto:simonpj@microsoft.com>> wrote: Thanks for the report. Going from 67G to 56G allocation is a very worthwhile improvement in runtime! Hurrah. However, trebling compile time is very bad. It is (I think) far from typical: generally 8.2 is faster at compiling than 8.0 so you must be hitting something weird. Anything you can do to make a reproducible case would be helpful. -dshow-passes shows the size of each intermediate form, which at least sometimes shows where the big changes are. Simon From: Glasgow-haskell-users [mailto:glasgow-haskell-users-bounces@haskell.org<mailto:glasgow-haskell-users-bounces@haskell.org>] On Behalf Of Alberto Valverde Sent: 06 June 2017 12:39 To: GHC users <glasgow-haskell-users(a)haskell.org<mailto:glasgow-haskell-users@haskell.org>> Subject: 8.2.1-rc2 upgrade report Hi, I've finally managed to upgrade all the dependencies of the proprietary app I mentioned some days ago in this list and there are good and bad differences I've noticed between 8.0.2 that I'd like to share. The bad ----------- * An optimized cold build (-O2) is about 3 times slower (~53s vs. ~2m55s) and consumes more memory (~2Gb vs. ~7Gb) at it's peak. The good ------------- * An un-optimized cold build (-O0) takes about the same time (~21s, phew! :) It's maybe even slightly faster with 8.2 (too few and badly taken measurements to really know, though) * The optimized executable is slightly faster and allocates less memory. For this app it makes up for the performance regression of the optimized build (which is almost always done by CI), IMHO. I did only a couple of runs and only wrote down [1] the last run results (which were similar to the previous results) so take these observations with a grain of salt (except maybe the optimized build slowdown, which doesn't have much margin for variance to be skewing the results). I also measured the peak memory usage by observing "top". In case gives a clue: The app is a multi-threaded 2D spread simulator which deals with many mmapped Storable mutable vectors and has been pretty optimized for countless hours (I mean by this that it has (too) many INLINE pragmas. Mostly on polymorphic functions to aid in their specialization). I think some of this information can be deduced from the results I'm linking at the footer. I believe the INLINEs are playing a big part of the slowdown since the slowest modules to compile are the "Main" ones which put everything together, along with the typical lens-th-heavy "Types" ones. I'd like to help by producing a reproducible and isolated benchmark or a better analysis or ... so someone more knowledgeable than me on GHC internals can someday hopefully attack the regression. Any pointers on what would help and where can I learn to do it? Thanks! [1] https://gist.github.com/albertov/46fbb13d940f67a569f9a25c1cb8154c<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.githu…>

2 1

Removing Hoopl dependency?
by Michal Terepeta 12 Jun '17

12 Jun '17

Hi all, I was looking at removing the `BlockId` type synonym in favor of Hoopl's `Label` (there was already a TODO and it is a bit confusing). But once I've started making the changes, I've realized that in a bunch of places this makes the code *less* readable. Mostly because of `CLabel` (sounds similar but is something quite different and having to rename local variables from `label` to `clabel` is not great). I started to look at alternatives and noticed that in general the interface between GHC and Hoopl is quite noisy and confusing: - Hoopl has `Label` which is GHC's `BlockId` but different than GHC's `CLabel` - Hoopl has `Unique` which is different than GHC's `Unique` - Hoopl has `Unique{Map,Set}` which are different than GHC's `Uniq{FM,Set}` - GHC has its own specialized copy of `Dataflow`, so `cmm/Hoopl` is needed just to filter the exposed functions (filter out some of the Hoopl's and add the GHC ones). - Working in `cmm/` requires constant switching between GHC code and Hoopl (`CmmNode`/`CmmGraph`/`CmmBlock` and dataflow stuff is in GHC, the actual implementation of `Block`/`Graph` are defined in Hoopl, etc.) GHC is actually using only a small subset of Hoopl (e.g., the fixpoint computation is copied/specialized: `cmm/Hoopl/Dataflow`). So I was wondering - maybe it's worth to simply drop the dependency on Hoopl? (and copy the code that is actually necessary in GHC) I've done an experiment in [1] (to see how much we'd need to actually copy) and I really like the result: - We can remove one external dependency and git submodule at the cost of only 5 new modules in `cmm/Hoopl` (net gain of only 4 modules: we add 5 new but can remove `cmm/Hoopl`, which is no longer needed) - We should be able to fix all of the above issues and make the code easier to understand (less code, everything in one repo, fewer concepts). - It's going to be easier to change things since we don't need to worry about changing the public interface of Hoopl (it's a standalone package on Hackage and other people already depend on the current behavior). What do you think? Does anyone think we shouldn't do this? Thanks, Michal [1] Branch: https://github.com/michalt/ghc/tree/hoopl/no-hoopl Diff: https://github.com/ghc/ghc/compare/master...michalt:hoopl/no-hoopl For now I just copied the code/updated imports and didn't do any cleanups, but I'd be happy to do them in subsequent PRs

8 22

8.2.1 Status
by Ben Gamari 12 Jun '17

12 Jun '17

Hello everyone, GHC 8.2.1 release candidate 2 has been out for a few weeks now and, while it turned up a number of issues, it seems that the release is (slowly) solidifying. Currently the ghc-8.2 branch has nearly 50 commits since the -rc2 tag, fixing many of the issues that have been reported. While we were hoping that 8.2 would require only two release candidates, it seems pretty clear that a third is in order. At this point, there are still a few more tickets in need of resolution before -rc3 release candidate can be cut. Hopefully these will be sorted out in the coming week, which means that we can cut a new source release by mid-next week, with the binary releases coming a week later. Unfortunately this all means that we'll realistically be looking at a 8.2.1 release date sometime near the beginning of July. As always, help is greatly appreciated; drop by #ghc if you would like to help squash some bugs. Cheers, - Ben

1 0

More TRAC ticket statuses?
by David Feuer 08 Jun '17

08 Jun '17

There are (at least) two situations that I don't think we currently have a good way to track: 1. A new bug has been verified, but we do not yet have an expect_broken test case. 2. A bug has been fixed, but we are waiting for a test case. Lacking (1) means that we have to manually dig through the ticket database and try things out in order to knock down tickets that should have already been closed. Lacking (2) means that a ticket could potentially get stuck in "new" status after someone's already done practically all the work required to fix them. What should we do about this? David

2 1

Re: Hunting down a compilation performance regression involving type families
by Ryan Scott 06 Jun '17

06 Jun '17

Ah yes, I believe you're referring to https://ghc.haskell.org/ trac/ghc/ticket/8767 (which I wasn't aware of). And yes, I do believe we'd need some combination of QuantifiedContexts/ImplicationConstraints to fully support everything. But in any case, don't let this discussion side-track too much from the matter at hand (the type family compilation slowdown). My suggestion _was_ somewhat tongue-in-cheek, after all ;) Ryan S. On Tue, Jun 6, 2017 at 11:01 AM, David Feuer <david(a)well-typed.com> wrote: > Edward Kmett has explained that this isn't sufficient when things go > higher order. His suggested improvement is > > liftCoercion :: Maybe (Coercion a b -> Coercion (f a) (f b)) > > > > David Feuer > Well-Typed, LLP > > -------- Original message -------- > From: Ryan Scott <ryan.gl.scott(a)gmail.com> > Date: 6/6/17 1:41 PM (GMT-05:00) > To: Richard Eisenberg <rae(a)cs.brynmawr.edu> > Cc: GHC developers <ghc-devs(a)haskell.org>, Eric Mertens < > emertens(a)gmail.com> > Subject: Re: Hunting down a compilation performance regression involving > type families > > Hrm. It's a shame that supporting this map/coerce RULE causes such pain. > > This makes me wonder: can we get rid of this RULE? Eric Mertens pointed out > a trick [1] that's used in the profunctors library to make mapping coerce > over certain Profunctors more efficient. To adapt this trick for Functor, > we'd need to add another class method: > > class Functor f where > fmap :: (a -> b) -> f a -> f b > (<#>) :: Coercible a b => (a -> b) -> f a -> f b > (<#>) = \f -> \p -> p `seq` fmap f p > > Now, when implementing Functor instances, if we are working with a datatype > whose role is representational or phantom, we can make (<#>) really fast: > > data List a = Nil | Cons a (List a) > instance Functor List where > fmap = ... > (<#>) = coerce > > Now, instead of relying on (map MkNewtype Nil) to rewrite to Nil, we can > just use (MkNewtype <#> Nil)! No map/coerce RULE necessary :) > > OK, I realize that suggesting that we remove the RULE is perhaps a touch > too far. But it does sting that we have to pay hefty compilation penalties > because of its existence... > > Ryan S. > ----- > [1] > http://hackage.haskell.org/package/profunctors-5.2/docs/ > Data-Profunctor-Unsafe.html#v:-35- > . > > On Wed, May 31, 2017 at 7:25 PM, Richard Eisenberg <rae(a)cs.brynmawr.edu> > wrote: > > > > > > On May 31, 2017, at 5:21 PM, Ryan Scott <ryan.gl.scott(a)gmail.com> > wrote: > > > Does you know what might be going on here? > > > > I think so, but I don't know how to fix it. > > > > The commit you found (thank you!) makes simple_opt_expr (the "simple > > optimizer", run directly after desugaring, even with -O0) a little more > > selective in what `case` expressions it throws away. Previous to that > > commit, the optimizer would throw away a `case error "deferred type > error" > > of _ -> ...` which is terrible. It seems that you have discovered that we > > are now too timid in throwing away unhelpful cases. It would be > interesting > > to know what the newly-retained cases look like, so that we might throw > > them away. > > > > But there remains a mystery: Why do we need this code at all? Reading > Note > > [Getting the map/coerce RULE to work] closely, it seems we need to > simplify > > > > forall a b (co :: a ~R# b). > > let dict = MkCoercible @* @a @b co in > > case Coercible_SCSel @* @a @b dict of > > _ [Dead] -> map @a @b (\(x :: a) -> case dict of > > MkCoercible (co :: a ~R# b) -> x |> co) = let dict = ... in ... > > > > to > > > > forall a b (co :: a ~R# b). > > map @a @b (\(x :: a) -> x |> co) = \(x :: [a]) -> x |> [co] > > > > Part of doing so is to drop the `case Coercible_SCSel ...`, which gets in > > the way. The mystery is why this needs special code -- shouldn't the > > eliminate-case-of-known-constructor do the trick? This would require > > unfolding Coercible_SCSel. Does that happen? It would seem not... but > maybe > > it should, which would remove the special-case code that I changed in > that > > commit, and quite likely would simplify much more code besides. > > > > So: Is Coercible_SCSel unfolded during simple_opt? If not, what wonderful > > or terrible things happen if we do? If so, why does > > case-of-known-constructor not work here? My guess is that answering these > > questions may solve the original problem, but this guess could be wrong. > > > > Richard > > > > >

1 0

Re: Hunting down a compilation performance regression involving type families
by David Feuer 06 Jun '17

06 Jun '17

Edward Kmett has explained that this isn't sufficient when things go higher order. His suggested improvement is liftCoercion :: Maybe (Coercion a b -> Coercion (f a) (f b)) David FeuerWell-Typed, LLP -------- Original message --------From: Ryan Scott <ryan.gl.scott(a)gmail.com> Date: 6/6/17 1:41 PM (GMT-05:00) To: Richard Eisenberg <rae(a)cs.brynmawr.edu> Cc: GHC developers <ghc-devs(a)haskell.org>, Eric Mertens <emertens(a)gmail.com> Subject: Re: Hunting down a compilation performance regression involving type families Hrm. It's a shame that supporting this map/coerce RULE causes such pain. This makes me wonder: can we get rid of this RULE? Eric Mertens pointed out a trick [1] that's used in the profunctors library to make mapping coerce over certain Profunctors more efficient. To adapt this trick for Functor, we'd need to add another class method: class Functor f where fmap :: (a -> b) -> f a -> f b (<#>) :: Coercible a b => (a -> b) -> f a -> f b (<#>) = \f -> \p -> p `seq` fmap f p Now, when implementing Functor instances, if we are working with a datatype whose role is representational or phantom, we can make (<#>) really fast: data List a = Nil | Cons a (List a) instance Functor List where fmap = ... (<#>) = coerce Now, instead of relying on (map MkNewtype Nil) to rewrite to Nil, we can just use (MkNewtype <#> Nil)! No map/coerce RULE necessary :) OK, I realize that suggesting that we remove the RULE is perhaps a touch too far. But it does sting that we have to pay hefty compilation penalties because of its existence... Ryan S. ----- [1] http://hackage.haskell.org/package/profunctors-5.2/docs/Data-Profunctor-Uns… . On Wed, May 31, 2017 at 7:25 PM, Richard Eisenberg <rae(a)cs.brynmawr.edu> wrote: > > > On May 31, 2017, at 5:21 PM, Ryan Scott <ryan.gl.scott(a)gmail.com> wrote: > > Does you know what might be going on here? > > I think so, but I don't know how to fix it. > > The commit you found (thank you!) makes simple_opt_expr (the "simple > optimizer", run directly after desugaring, even with -O0) a little more > selective in what `case` expressions it throws away. Previous to that > commit, the optimizer would throw away a `case error "deferred type error" > of _ -> ...` which is terrible. It seems that you have discovered that we > are now too timid in throwing away unhelpful cases. It would be interesting > to know what the newly-retained cases look like, so that we might throw > them away. > > But there remains a mystery: Why do we need this code at all? Reading Note > [Getting the map/coerce RULE to work] closely, it seems we need to simplify > > forall a b (co :: a ~R# b). > let dict = MkCoercible @* @a @b co in > case Coercible_SCSel @* @a @b dict of > _ [Dead] -> map @a @b (\(x :: a) -> case dict of > MkCoercible (co :: a ~R# b) -> x |> co) = let dict = ... in ... > > to > > forall a b (co :: a ~R# b). > map @a @b (\(x :: a) -> x |> co) = \(x :: [a]) -> x |> [co] > > Part of doing so is to drop the `case Coercible_SCSel ...`, which gets in > the way. The mystery is why this needs special code -- shouldn't the > eliminate-case-of-known-constructor do the trick? This would require > unfolding Coercible_SCSel. Does that happen? It would seem not... but maybe > it should, which would remove the special-case code that I changed in that > commit, and quite likely would simplify much more code besides. > > So: Is Coercible_SCSel unfolded during simple_opt? If not, what wonderful > or terrible things happen if we do? If so, why does > case-of-known-constructor not work here? My guess is that answering these > questions may solve the original problem, but this guess could be wrong. > > Richard > >

2 1

Hunting down a compilation performance regression involving type families
by Ryan Scott 06 Jun '17

06 Jun '17

Richard, In an effort to figure out why programs with lots of type families have become so much slower over the last few GHC releases, I've been looking at GHC Trac #12545 [1], which provides a very self-contained test case that became considerably slower to compile from GHC 7.10 to 8.0. Thanks to this test case, I found one of the sources of the slowdown: a single, very small commit of yours [2]. What's baffling, though, is that I can't figure out why that commit makes compilation so much slower. I tried making profiled builds of GHC before and after that commit and attaching SCCs to the new code, but to my surprise, their contribution to the overall compilation time was negligible. So I'm out of ideas on what might be causing this. The only hint I have is the output of -ddump-simpl-stats/-dshow-passes. Before that commit, there's a point where the terms, types, and coercions go up: *** Float out(FOS {Lam = Just 0, Consts = True, OverSatApps = False}): Result size of Float out(FOS {Lam = Just 0, Consts = True, OverSatApps = False}) = {terms: 20,769, types: 1,157,897, coercions: 3,635,961} *** Simplifier: Result size of Simplifier iteration=1 = {terms: 44,081, types: 2,451,155, coercions: 9,802,016} But after that commit, the terms and types jump up considerably higher! *** Simplifier: Result size of Simplifier = {terms: 16,429, types: 864,761, coercions: 2,184,448} *** Simplifier: Result size of Simplifier iteration=1 = {terms: 87,357, types: 4,366,893, coercions: 10,008,352} Here are the relevant bits from -ddump-simpl-stats. Before that commit: 150 PreInlineUnconditionally 40 PostInlineUnconditionally 220 UnfoldingDone 110 RuleFired 380 BetaReduction After that commit: 306 PreInlineUnconditionally 160 PostInlineUnconditionally 360 UnfoldingDone 308 RuleFired 1238 BetaReduction Does you know what might be going on here? Ryan S. ----- [1] https://ghc.haskell.org/trac/ghc/ticket/12545 [2] https://ghc.haskell.org/trac/ghc/changeset/1722fa106e10e63160bb2322e2ccb830…

2 2

Interested to help with error messages
by Siddharth Bhat 06 Jun '17

06 Jun '17

Hello all, Thanks for all the work that's put into GHC :) I've tried to get into GHC development before, but I was unsuccessful, mostly because I didn't dedicate enough time to understanding the problem at hand & exploring the codebase. I'd like to give it another shot. This time, I think I have a clear vision of what I want to help with: Have haskell's error messages be easier to read and understand. 1. Colors and layout to highlight important parts of the error messages 2. Clear formatting & naming of errors, so they're easily googleable, stack-overflow able, etc. 3. better hints with error messages, and perhaps integrated lints(?). 4. I don't know if this is already possible, but allowing GHC errors to be shipped off as JSON or something to interested tooling. I saw this ticket on trac: https://ghc.haskell.org/trac/ghc/ticket/8809 I would like to take this up, but I'd like help / pointers and stuff. I have GHC setup, I know how to use phabricator, but.. where do I start? :) Thanks, S~iddharth -- Sending this from my phone, please excuse any typos!

6 7

Trees that Grow / D3609
by Alan & Kim Zimmerman 05 Jun '17

05 Jun '17

This is a heads up that D3609[1], the first step toward implementing Trees that Grow[2] for the hsSyn AST is about to land in master. The key change is that it is replacing the HsSyn type parameter with one that is an index into a set of type families, enabling the extension points (for a future diff). Any current work in progress branches will need a one-time search and replace of all type parameters as follows RdrName -> GhcPs Name -> GhcRs Var -> GhcTc Id -> GhcTcId Any code that makes use of the GHC API and needs to compile with earlier versions too can do something like #if MIN_VERSION_ghc(8,3,0) type ParseI = GhcPs type RenameI = GhcRn type TypecheckI = GhcTc #else type ParseI = RdrName type RenameI = Name type TypecheckI = Var #endif and then replace the RdrName / Name / Var type params with ParseI / RenameI / TypecheckI. A real world example is in ghc-exactprint[3] Regards Alan [1] https://phabricator.haskell.org/D3609 [2] https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow [3] https://github.com/alanz/ghc-exactprint/commit/6a6e3cdeb40f9623e9a7fc6f1be9…

1 0

RE: Trees that Grow in the hsSyn AST
by Simon Peyton Jones 01 Jun '17

01 Jun '17

1. Which is better to start with: HsSyn or Core? Intuition suggests this sort of thing could be very helpful for making zapping more reliable and ensuring its efficiency, but there may be better reasons to start with HsSyn. Definitely HsSyn. It’s big, riddled with extra info, and has many purposes for different people. Core is small, tight, nailed down. I don’t want to mess with it. 2. If we're making intrusive changes to representations, would now be a sensible era to consider switching to a different variable representation (unbound, bound, abt, etc)? I don’t think so. The issues are quite orthogonal, and no one (to my knowledge) has proposed any vaguely plausible change to variable bindings that would scale to what GHC does. In contrast, this stuff is “just” re-engineering. Simon From: David Feuer [mailto:david@well-typed.com] Sent: 26 May 2017 01:11 To: Simon Peyton Jones <simonpj(a)microsoft.com>; Alan & Kim Zimmerman <alan.zimm(a)gmail.com>; ghc-devs(a)haskell.org Subject: RE: Trees that Grow in the hsSyn AST I haven't looked in detail yet, but there seem to be good ideas. I have two questions: 1. Which is better to start with: HsSyn or Core? Intuition suggests this sort of thing could be very helpful for making zapping more reliable and ensuring its efficiency, but there may be better reasons to start with HsSyn. 2. If we're making intrusive changes to representations, would now be a sensible era to consider switching to a different variable representation (unbound, bound, abt, etc)? David Feuer Well-Typed, LLP -------- Original message -------- From: Simon Peyton Jones via ghc-devs <ghc-devs(a)haskell.org<mailto:ghc-devs@haskell.org>> Date: 5/25/17 6:48 PM (GMT-05:00) To: Alan & Kim Zimmerman <alan.zimm(a)gmail.com<mailto:alan.zimm@gmail.com>>, ghc-devs(a)haskell.org<mailto:ghc-devs@haskell.org> Subject: RE: Trees that Grow in the hsSyn AST Folks Do take a look at this: · We propose to re-engineer HsSyn itself. This will touch a lot of code. · But it’s very neat, and will bring big long-term advantages · And we can do it a bit at a time The wiki page https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow has the details. It’s entirely an internal change, not a change to GHC’s specification, so it’s independent of the GHC proposals process. But I’d value the opinion of other GHC devs. Alan has done a prototype first step, which worked out rather well. Rather than having HsExpr Id (which we all know means “HsExpr after the typechecker” but tha’s a bit inexplicit, we have HsExpr GhcTc meaning “HsExpr after GHC’s Tc pass”. In some ways this is quite superficial, but it unlocks the Trees That Grow machiney. Please ask questions etc. Alan and Shayan can record the answers in the wiki. I’m inclined to go ahead with this, so yell soon if you disagree. Simon From: ghc-devs [mailto:ghc-devs-bounces@haskell.org] On Behalf Of Alan & Kim Zimmerman Sent: 24 May 2017 22:52 To: ghc-devs(a)haskell.org<mailto:ghc-devs@haskell.org> Subject: Trees that Grow in the hsSyn AST Hi all You may be aware that Shayan Najd presented the paper "Trees that Grow"[1] at HIW last year. Based on the following mandate > As in my previous email to Shayan (attached). Wiki page, describe goals, design, > approach. Point to prototype implementation. Seek comments. You can say that >I am supportive! > > Simon We have set up a Wiki page at [2] describing a prototype implementation of the first stage of this for the hsSyn AST, which is to change the polymorphic variable from one of RdrName / Name / Id to an index type. This is presented as a fabricator diff at [3]. Please take a look and provide feedback. Regards Alan [1] http://www.jucs.org/jucs_23_1/trees_that_grow/jucs_23_01_0042_0062_najd.pdf<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.jucs.or…<http://www.jucs.org/jucs_23_1/trees_that_grow/jucs_23_01_0042_0062_najd.pdf…>> [2] https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow [3] https://phabricator.haskell.org/D3609

2 2