June 2017 - ghc-devs - Haskell.org

external interpreter failing on Mac
by Richard Eisenberg 20 Jun '17

20 Jun '17

Hi devs, It seems every test run through the external interpreter is failing for me on a Mac: =====> TH_mkName(ext-interp) 1 of 1 [0, 0, 0] cd "./TH_mkName.run" && "/Users/rae/ghc/ghc/inplace/test spaces/ghc-stage2" -c TH_mkName.hs -dcore-lint -dcmm-lint -no-user-package-db -rtsopts -fno-warn-missed-specialisations -fshow-warning-groups -fdiagnostics-color=never -fno-diagnostics-show-caret -dno-debug-output -XTemplateHaskell -package template-haskell -fexternal-interpreter -v0 Compile failed (exit code 1) errors were: ghc-iserv.bin: lookupSymbol failed in relocateSection (RELOC_GOT) /Users/rae/ghc/ghc/libraries/integer-gmp/dist-install/build/HSinteger-gmp-1.0.0.1.o: unknown symbol `___gmp_rands' ghc-stage2: unable to load package `integer-gmp-1.0.0.1' *** unexpected failure for TH_mkName(ext-interp) Any advice? My most recent commit from master is ef07010cf4f480d9f595a71cf5b009884522a75e from Wed Jun 7. I'm on IRC today if you want to iterate that way. Thanks! Richard

3 3

#13804
by Simon Peyton Jones 17 Jun '17

17 Jun '17

Ben I've just pushed a fix to Trac #13804. It may belong on the 8.2 branch too because it fixes a regression wrt 8.0. I can't test on my Unix box, because I'm away from it for a few days, but I've validated on windows. Can you just check it's ok, and fix up or revert if not? Thanks SImon

2 1

WordX/IntX wrap Word#/Int#?
by Michal Terepeta 15 Jun '17

15 Jun '17

Hi all, I've just noticed that all `WordX` (and `IntX`) data types are actually implemented as wrappers around `Word#` (and `Int#`). This probably doesn't matter much if it's stored on the heap (due to pointer indirection and heap alignment), but it also means that: ``` data Foo = Foo {-# UNPACK #-} !Word8 {-# UNPACK #-} !Int8 ``` will actually take *a lot* of space: on 64 bit we'd need 8 bytes for header, 8 bytes for `Word8`, 8 bytes for `Int8`. Is there any reason for this? The only thing I can see is that this avoids having to add things like `Word8#` primitives into the compiler. (also the codegen would need to emit zero-extend moves when loading from memory, like `movzb{l,q}`) If we had things like `Word8#` we could also consider changing `Bool` to just wrap it (with the obvious encoding). Which would allow to both UNPACK `Bool` *and* save the size within the struct. (alternatively one could imagine a `Bool#` that would be just a byte) I couldn't find any discussion about this, so any pointers would be welcome. :) Thanks, Michal PS. I've had a look at it after reading about the recent implementation of struct field reordering optimization in rustc: http://camlorn.net/posts/April%202017/rust-struct-field-reordering.html

4 6

RE: Removing Hoopl dependency?
by ning w 14 Jun '17

14 Jun '17

Thanks Ben. Both A and B (mentioned in Simon's reply) are good alternatives as long as existing Hoopl users are NOT forced to upgrade when they upgrade Cable or Stackage. Otherwise, we will see more forks of Hoopl. Once Cable and Stackage gain the multi-version capability, A and B can be merged back to Hoopl as a major release. In my experience, Hoopl based optimizations/program analyses tend to use a lot of memory, but they are also easy to verify. So it's still a useful tool in some special use cases. If the performance of Hoopl can be improved, it will certainly be more useful. Cheers, Ning *From:* Ben Gamari <ben(a)smart-cactus.org> *Date:* June 12, 2017 at 11:05:51 AM PDT *To:* Simon Peyton Jones <simonpj(a)microsoft.com>, Sophie Taylor < sophie(a)traumapony.org>, Michal Terepeta <michal.terepeta(a)gmail.com>, ghc-devs <ghc-devs(a)haskell.org>, Ning Wang <email(a)ningwang.org> *Cc:* Kavon Farvardin <kavon(a)cs.uchicago.edu> *Subject:* *RE: Removing Hoopl dependency?* Simon Peyton Jones via ghc-devs <ghc-devs(a)haskell.org> writes: Snip That would leave Sophie free to do (B) free of the constraints of GHC depending on it; but we could always use it later. Does that sound plausible? Do we know of any other Hoopl users? CCing Ning, who is currently maintaining hoopl and I believe has some projects using it. Ning, you may want to have a look through this thread if you haven't already seen it. You can find the previous messages in the list archive [1]. Cheers, - Ben [1] May messages: https://mail.haskell.org/piper mail/ghc-devs/2017-May/014255.html June messages: https://mail.haskell.org/piper mail/ghc-devs/2017-June/014293.html

1 0

RE: 8.2.1-rc2 upgrade report
by Simon Peyton Jones 14 Jun '17

14 Jun '17

Great. Could you put all this into a Trac ticket? Thanks! Simon From: Alberto Valverde [mailto:alberto@toscat.net] Sent: 08 June 2017 13:57 To: Simon Peyton Jones <simonpj(a)microsoft.com> Cc: GHC users <glasgow-haskell-users(a)haskell.org> Subject: Re: 8.2.1-rc2 upgrade report Hi Simon, Thanks for the pointer. I re-did both builds with -dshow-passes and made a small script to plot the results of the lines which summarize the elapsed time and allocated memory per phase and module. I've uploaded the raw logs, a plot of the results and the script I wrote to generate it to https://gist.githubusercontent.com/albertov/145ac5c01bfbadc5c9ff55e9c5c2e50e<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.githu…>. The plotted results live here https://gist.githubusercontent.com/albertov/145ac5c01bfbadc5c9ff55e9c5c2e50…<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.githu…> Apparently, the biggest slowdown in respect to 8.0.2 seems to occur in the SpecConstr and Simplifier passes in the Propag (where the "main" function is) and the Sigym4.Propag.Engine (where the main algorithm lives) modules. Any other tests that would be helpful for me to run? I'm not sure where to start to create a reproducible case but I'll see if I can come up with something soon... Alberto On Tue, Jun 6, 2017 at 1:58 PM, Simon Peyton Jones <simonpj(a)microsoft.com<mailto:simonpj@microsoft.com>> wrote: Thanks for the report. Going from 67G to 56G allocation is a very worthwhile improvement in runtime! Hurrah. However, trebling compile time is very bad. It is (I think) far from typical: generally 8.2 is faster at compiling than 8.0 so you must be hitting something weird. Anything you can do to make a reproducible case would be helpful. -dshow-passes shows the size of each intermediate form, which at least sometimes shows where the big changes are. Simon From: Glasgow-haskell-users [mailto:glasgow-haskell-users-bounces@haskell.org<mailto:glasgow-haskell-users-bounces@haskell.org>] On Behalf Of Alberto Valverde Sent: 06 June 2017 12:39 To: GHC users <glasgow-haskell-users(a)haskell.org<mailto:glasgow-haskell-users@haskell.org>> Subject: 8.2.1-rc2 upgrade report Hi, I've finally managed to upgrade all the dependencies of the proprietary app I mentioned some days ago in this list and there are good and bad differences I've noticed between 8.0.2 that I'd like to share. The bad ----------- * An optimized cold build (-O2) is about 3 times slower (~53s vs. ~2m55s) and consumes more memory (~2Gb vs. ~7Gb) at it's peak. The good ------------- * An un-optimized cold build (-O0) takes about the same time (~21s, phew! :) It's maybe even slightly faster with 8.2 (too few and badly taken measurements to really know, though) * The optimized executable is slightly faster and allocates less memory. For this app it makes up for the performance regression of the optimized build (which is almost always done by CI), IMHO. I did only a couple of runs and only wrote down [1] the last run results (which were similar to the previous results) so take these observations with a grain of salt (except maybe the optimized build slowdown, which doesn't have much margin for variance to be skewing the results). I also measured the peak memory usage by observing "top". In case gives a clue: The app is a multi-threaded 2D spread simulator which deals with many mmapped Storable mutable vectors and has been pretty optimized for countless hours (I mean by this that it has (too) many INLINE pragmas. Mostly on polymorphic functions to aid in their specialization). I think some of this information can be deduced from the results I'm linking at the footer. I believe the INLINEs are playing a big part of the slowdown since the slowest modules to compile are the "Main" ones which put everything together, along with the typical lens-th-heavy "Types" ones. I'd like to help by producing a reproducible and isolated benchmark or a better analysis or ... so someone more knowledgeable than me on GHC internals can someday hopefully attack the regression. Any pointers on what would help and where can I learn to do it? Thanks! [1] https://gist.github.com/albertov/46fbb13d940f67a569f9a25c1cb8154c<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.githu…>

2 1

Removing Hoopl dependency?
by Michal Terepeta 13 Jun '17

13 Jun '17

Hi all, I was looking at removing the `BlockId` type synonym in favor of Hoopl's `Label` (there was already a TODO and it is a bit confusing). But once I've started making the changes, I've realized that in a bunch of places this makes the code *less* readable. Mostly because of `CLabel` (sounds similar but is something quite different and having to rename local variables from `label` to `clabel` is not great). I started to look at alternatives and noticed that in general the interface between GHC and Hoopl is quite noisy and confusing: - Hoopl has `Label` which is GHC's `BlockId` but different than GHC's `CLabel` - Hoopl has `Unique` which is different than GHC's `Unique` - Hoopl has `Unique{Map,Set}` which are different than GHC's `Uniq{FM,Set}` - GHC has its own specialized copy of `Dataflow`, so `cmm/Hoopl` is needed just to filter the exposed functions (filter out some of the Hoopl's and add the GHC ones). - Working in `cmm/` requires constant switching between GHC code and Hoopl (`CmmNode`/`CmmGraph`/`CmmBlock` and dataflow stuff is in GHC, the actual implementation of `Block`/`Graph` are defined in Hoopl, etc.) GHC is actually using only a small subset of Hoopl (e.g., the fixpoint computation is copied/specialized: `cmm/Hoopl/Dataflow`). So I was wondering - maybe it's worth to simply drop the dependency on Hoopl? (and copy the code that is actually necessary in GHC) I've done an experiment in [1] (to see how much we'd need to actually copy) and I really like the result: - We can remove one external dependency and git submodule at the cost of only 5 new modules in `cmm/Hoopl` (net gain of only 4 modules: we add 5 new but can remove `cmm/Hoopl`, which is no longer needed) - We should be able to fix all of the above issues and make the code easier to understand (less code, everything in one repo, fewer concepts). - It's going to be easier to change things since we don't need to worry about changing the public interface of Hoopl (it's a standalone package on Hackage and other people already depend on the current behavior). What do you think? Does anyone think we shouldn't do this? Thanks, Michal [1] Branch: https://github.com/michalt/ghc/tree/hoopl/no-hoopl Diff: https://github.com/ghc/ghc/compare/master...michalt:hoopl/no-hoopl For now I just copied the code/updated imports and didn't do any cleanups, but I'd be happy to do them in subsequent PRs

8 22

8.2.1 Status
by Ben Gamari 13 Jun '17

13 Jun '17

Hello everyone, GHC 8.2.1 release candidate 2 has been out for a few weeks now and, while it turned up a number of issues, it seems that the release is (slowly) solidifying. Currently the ghc-8.2 branch has nearly 50 commits since the -rc2 tag, fixing many of the issues that have been reported. While we were hoping that 8.2 would require only two release candidates, it seems pretty clear that a third is in order. At this point, there are still a few more tickets in need of resolution before -rc3 release candidate can be cut. Hopefully these will be sorted out in the coming week, which means that we can cut a new source release by mid-next week, with the binary releases coming a week later. Unfortunately this all means that we'll realistically be looking at a 8.2.1 release date sometime near the beginning of July. As always, help is greatly appreciated; drop by #ghc if you would like to help squash some bugs. Cheers, - Ben

1 0

More TRAC ticket statuses?
by David Feuer 08 Jun '17

08 Jun '17

There are (at least) two situations that I don't think we currently have a good way to track: 1. A new bug has been verified, but we do not yet have an expect_broken test case. 2. A bug has been fixed, but we are waiting for a test case. Lacking (1) means that we have to manually dig through the ticket database and try things out in order to knock down tickets that should have already been closed. Lacking (2) means that a ticket could potentially get stuck in "new" status after someone's already done practically all the work required to fix them. What should we do about this? David

2 1

Re: Hunting down a compilation performance regression involving type families
by Ryan Scott 07 Jun '17

07 Jun '17

Ah yes, I believe you're referring to https://ghc.haskell.org/ trac/ghc/ticket/8767 (which I wasn't aware of). And yes, I do believe we'd need some combination of QuantifiedContexts/ImplicationConstraints to fully support everything. But in any case, don't let this discussion side-track too much from the matter at hand (the type family compilation slowdown). My suggestion _was_ somewhat tongue-in-cheek, after all ;) Ryan S. On Tue, Jun 6, 2017 at 11:01 AM, David Feuer <david(a)well-typed.com> wrote: > Edward Kmett has explained that this isn't sufficient when things go > higher order. His suggested improvement is > > liftCoercion :: Maybe (Coercion a b -> Coercion (f a) (f b)) > > > > David Feuer > Well-Typed, LLP > > -------- Original message -------- > From: Ryan Scott <ryan.gl.scott(a)gmail.com> > Date: 6/6/17 1:41 PM (GMT-05:00) > To: Richard Eisenberg <rae(a)cs.brynmawr.edu> > Cc: GHC developers <ghc-devs(a)haskell.org>, Eric Mertens < > emertens(a)gmail.com> > Subject: Re: Hunting down a compilation performance regression involving > type families > > Hrm. It's a shame that supporting this map/coerce RULE causes such pain. > > This makes me wonder: can we get rid of this RULE? Eric Mertens pointed out > a trick [1] that's used in the profunctors library to make mapping coerce > over certain Profunctors more efficient. To adapt this trick for Functor, > we'd need to add another class method: > > class Functor f where > fmap :: (a -> b) -> f a -> f b > (<#>) :: Coercible a b => (a -> b) -> f a -> f b > (<#>) = \f -> \p -> p `seq` fmap f p > > Now, when implementing Functor instances, if we are working with a datatype > whose role is representational or phantom, we can make (<#>) really fast: > > data List a = Nil | Cons a (List a) > instance Functor List where > fmap = ... > (<#>) = coerce > > Now, instead of relying on (map MkNewtype Nil) to rewrite to Nil, we can > just use (MkNewtype <#> Nil)! No map/coerce RULE necessary :) > > OK, I realize that suggesting that we remove the RULE is perhaps a touch > too far. But it does sting that we have to pay hefty compilation penalties > because of its existence... > > Ryan S. > ----- > [1] > http://hackage.haskell.org/package/profunctors-5.2/docs/ > Data-Profunctor-Unsafe.html#v:-35- > . > > On Wed, May 31, 2017 at 7:25 PM, Richard Eisenberg <rae(a)cs.brynmawr.edu> > wrote: > > > > > > On May 31, 2017, at 5:21 PM, Ryan Scott <ryan.gl.scott(a)gmail.com> > wrote: > > > Does you know what might be going on here? > > > > I think so, but I don't know how to fix it. > > > > The commit you found (thank you!) makes simple_opt_expr (the "simple > > optimizer", run directly after desugaring, even with -O0) a little more > > selective in what `case` expressions it throws away. Previous to that > > commit, the optimizer would throw away a `case error "deferred type > error" > > of _ -> ...` which is terrible. It seems that you have discovered that we > > are now too timid in throwing away unhelpful cases. It would be > interesting > > to know what the newly-retained cases look like, so that we might throw > > them away. > > > > But there remains a mystery: Why do we need this code at all? Reading > Note > > [Getting the map/coerce RULE to work] closely, it seems we need to > simplify > > > > forall a b (co :: a ~R# b). > > let dict = MkCoercible @* @a @b co in > > case Coercible_SCSel @* @a @b dict of > > _ [Dead] -> map @a @b (\(x :: a) -> case dict of > > MkCoercible (co :: a ~R# b) -> x |> co) = let dict = ... in ... > > > > to > > > > forall a b (co :: a ~R# b). > > map @a @b (\(x :: a) -> x |> co) = \(x :: [a]) -> x |> [co] > > > > Part of doing so is to drop the `case Coercible_SCSel ...`, which gets in > > the way. The mystery is why this needs special code -- shouldn't the > > eliminate-case-of-known-constructor do the trick? This would require > > unfolding Coercible_SCSel. Does that happen? It would seem not... but > maybe > > it should, which would remove the special-case code that I changed in > that > > commit, and quite likely would simplify much more code besides. > > > > So: Is Coercible_SCSel unfolded during simple_opt? If not, what wonderful > > or terrible things happen if we do? If so, why does > > case-of-known-constructor not work here? My guess is that answering these > > questions may solve the original problem, but this guess could be wrong. > > > > Richard > > > > >

1 0

Re: Hunting down a compilation performance regression involving type families
by David Feuer 07 Jun '17

07 Jun '17

Edward Kmett has explained that this isn't sufficient when things go higher order. His suggested improvement is liftCoercion :: Maybe (Coercion a b -> Coercion (f a) (f b)) David FeuerWell-Typed, LLP -------- Original message --------From: Ryan Scott <ryan.gl.scott(a)gmail.com> Date: 6/6/17 1:41 PM (GMT-05:00) To: Richard Eisenberg <rae(a)cs.brynmawr.edu> Cc: GHC developers <ghc-devs(a)haskell.org>, Eric Mertens <emertens(a)gmail.com> Subject: Re: Hunting down a compilation performance regression involving type families Hrm. It's a shame that supporting this map/coerce RULE causes such pain. This makes me wonder: can we get rid of this RULE? Eric Mertens pointed out a trick [1] that's used in the profunctors library to make mapping coerce over certain Profunctors more efficient. To adapt this trick for Functor, we'd need to add another class method: class Functor f where fmap :: (a -> b) -> f a -> f b (<#>) :: Coercible a b => (a -> b) -> f a -> f b (<#>) = \f -> \p -> p `seq` fmap f p Now, when implementing Functor instances, if we are working with a datatype whose role is representational or phantom, we can make (<#>) really fast: data List a = Nil | Cons a (List a) instance Functor List where fmap = ... (<#>) = coerce Now, instead of relying on (map MkNewtype Nil) to rewrite to Nil, we can just use (MkNewtype <#> Nil)! No map/coerce RULE necessary :) OK, I realize that suggesting that we remove the RULE is perhaps a touch too far. But it does sting that we have to pay hefty compilation penalties because of its existence... Ryan S. ----- [1] http://hackage.haskell.org/package/profunctors-5.2/docs/Data-Profunctor-Uns… . On Wed, May 31, 2017 at 7:25 PM, Richard Eisenberg <rae(a)cs.brynmawr.edu> wrote: > > > On May 31, 2017, at 5:21 PM, Ryan Scott <ryan.gl.scott(a)gmail.com> wrote: > > Does you know what might be going on here? > > I think so, but I don't know how to fix it. > > The commit you found (thank you!) makes simple_opt_expr (the "simple > optimizer", run directly after desugaring, even with -O0) a little more > selective in what `case` expressions it throws away. Previous to that > commit, the optimizer would throw away a `case error "deferred type error" > of _ -> ...` which is terrible. It seems that you have discovered that we > are now too timid in throwing away unhelpful cases. It would be interesting > to know what the newly-retained cases look like, so that we might throw > them away. > > But there remains a mystery: Why do we need this code at all? Reading Note > [Getting the map/coerce RULE to work] closely, it seems we need to simplify > > forall a b (co :: a ~R# b). > let dict = MkCoercible @* @a @b co in > case Coercible_SCSel @* @a @b dict of > _ [Dead] -> map @a @b (\(x :: a) -> case dict of > MkCoercible (co :: a ~R# b) -> x |> co) = let dict = ... in ... > > to > > forall a b (co :: a ~R# b). > map @a @b (\(x :: a) -> x |> co) = \(x :: [a]) -> x |> [co] > > Part of doing so is to drop the `case Coercible_SCSel ...`, which gets in > the way. The mystery is why this needs special code -- shouldn't the > eliminate-case-of-known-constructor do the trick? This would require > unfolding Coercible_SCSel. Does that happen? It would seem not... but maybe > it should, which would remove the special-case code that I changed in that > commit, and quite likely would simplify much more code besides. > > So: Is Coercible_SCSel unfolded during simple_opt? If not, what wonderful > or terrible things happen if we do? If so, why does > case-of-known-constructor not work here? My guess is that answering these > questions may solve the original problem, but this guess could be wrong. > > Richard > >

2 1