June 2017 - ghc-tickets

[GHC] #8457: -ffull-laziness does more harm than good
by GHC 08 Dec '17

08 Dec '17

#8457: -ffull-laziness does more harm than good ------------------------------------+------------------------------------- Reporter: errge | Owner: Type: bug | Status: new Priority: high | Milestone: 7.8.1 Component: Compiler | Version: 7.7 Keywords: | Operating System: Unknown/Multiple Architecture: Unknown/Multiple | Type of failure: None/Unknown Difficulty: Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | ------------------------------------+------------------------------------- In this bug report I'd like to argue that `-ffull-laziness` shouldn't be turned on automatically with either `-O` nor `-O2`, because it's dangerous and can cause serious memory leaks which are hard to debug or prevent. I'll also try to show that its optimization benefits are negligible. Actually, my benchmarks show that it's beneficial to turn it off even in the cases where we don't hit a space leak. We've met this issue last week, but it had been reported several times before: e.g. #917 and #5262. A typical example is the following: {{{ #!haskell main :: IO () main = task () >> task () task :: () -> IO () task () = printvalues [1..1000000 :: Int] printvalues :: [Int] -> IO () printvalues (x:xs) = print x >> printvalues xs printvalues [] = return () }}} We succeed with `-O0`, but fail with `-O`: {{{ errge@curry:~/tmp $ ~/tmp/ghc/inplace/bin/ghc-stage2 -v0 -O0 -fforce- recomp lazy && ./lazy +RTS -t >/dev/null <<ghc: 1620098744 bytes, 3117 GCs, 32265/42580 avg/max bytes residency (3 samples), 2M in use, 0.00 INIT (0.00 elapsed), 1.28 MUT (1.28 elapsed), 0.02 GC (0.02 elapsed) :ghc>> errge@curry:~/tmp $ ~/tmp/ghc/inplace/bin/ghc-stage2 -v0 -O -fforce-recomp lazy && ./lazy +RTS -t >/dev/null <<ghc: 1444098612 bytes, 2761 GCs, 3812497/13044272 avg/max bytes residency (7 samples), 28M in use, 0.00 INIT (0.00 elapsed), 1.02 MUT (1.03 elapsed), 0.12 GC (0.12 elapsed) :ghc>> }}} 28M? What the leak!? Well, it's `-ffull-laziness`: {{{ errge@curry:~/tmp $ ~/tmp/ghc/inplace/bin/ghc-stage2 -v0 -O -fno-full- laziness -fforce-recomp lazy && ./lazy +RTS -t >/dev/null <<ghc: 1484098612 bytes, 2835 GCs, 34812/42580 avg/max bytes residency (2 samples), 1M in use, 0.00 INIT (0.00 elapsed), 1.04 MUT (1.04 elapsed), 0.02 GC (0.02 elapsed) :ghc>> }}} We get constant space and the fastest run-time too, since we spare some cycles on GC. Note, that in this instance we are trying to explicity disable sharing by using `()` as a fake argument for the function. Also note, that this function may easily be a utility function in a larger code base or in a library, therefore it's impractical to say that you shouldn't use it twice "too close together". Quoting from the GHC user guide: {{{ -O2: Means: “Apply every non-dangerous optimisation, even if it means significantly longer compile times.” The avoided “dangerous” optimisations are those that can make runtime or space worse if you're unlucky. They are normally turned on or off individually. At the moment, -O2 is unlikely to produce better code than -O. }}} This seems to be false at the moment. We decided to make a broader investigation into this issue and wanted to know if we can disable this optimization without too much pain. Came up with this benchmark plan: - let's benchmark GHC, - compile all stages with -O, but hack the stage1 compiler to emit `-t` statistics for every file compiled, - gather these statistics while compiling the libraries and the stage2 compiler. On the second run we compile the stage1 compiler with `-O -fno-full-laziness`, but leave everything else unchanged in the environment. When we have both results of the compilation of ~1600 files, we match them up and compute the (logarithmic) ratio of CPU and memory difference between compilations, the final results for our benchmark. The results and the raw data can be found at https://github.com/errge/notlazy. The overall compilation time dropped from 26:20 to 25:12, which is a 4% improvement. Investigating the full matching shows that this overall result is from small improvements all around the place. The results plotted: - https://github.com/errge/notlazy/blob/master/cpu.png - https://github.com/errge/notlazy/blob/master/mem.png The graphs show the logarithmic (100*log_10(new/orig)) ratio of change in cpu and memory consumption. Therefore negative results mean that the new compilation method is faster. As can be seen on the CPU graph, in most of the cases the difference is negligible (actually smaller than what can be measured on small files, this is why we have the spike at 0). In overall we see a small improvement in CPU, and there are some outliers in both directions, but there are more drastic improvement cases than drastic regressions. On the memory graph the situation is much more close to zero. There is one big positive memory outlier: `DsListComp.lhs`. It uses 69M originally and now uses 103M. But compiles in 2 seconds both ways and there are files in the source tree which requires 400M to compile, so this is not an issue. After all this, I'd like to hear other opinions about just disabling this optimization in `-O` and `-O2` and leaving it as an option that can be turned on when needed, my reasons once more: - it's unsafe, - it's hard to debug when you hit its issues, - the optimization doesn't seem to be very productive, - it's always easy to force sharing, but it's not easy to force copying. Apparently a Haskell programmer should be lazy, but never fully lazy. Research done by Gergely Risko <errge> and Mihaly Barasz <klao>, confirmed on two different machines with no other running processes. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8457> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 14

[GHC] #10535: Float out causes major space leak
by GHC 08 Dec '17

08 Dec '17

#10535: Float out causes major space leak -------------------------------------+------------------------------------- Reporter: AlexET | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.1 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Blocked By: Test Case: | Related Tickets: Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- This issue originally was on haskell-cafe https://mail.haskell.org/pipermail/haskell-cafe/2015-June/120113.html The following code has a space leak {{{ #!hs numbers = [1..200] replicateM' :: Monad m => Int -> m a -> m [a] replicateM' 0 xs = return [] replicateM' n xs = do a <- xs b <- replicateM' (n-1) xs return (a:b) test :: [[Int]] test = replicateM' 4 numbers main = print test }}} The recursive call in replicateM' gets floated out. This then causes a very large (200^3 elements) list to be kept in memory instead of being lazily produced and consumed each time. A similar thing happens to the replicateM in Control.Monad but it is harder to spot as a lot of specialisation and fusion goes on before the float-out happens This is similar to #7367 but in this case we have a huge space-leak instead of just adding allocations. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/10535> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 7

[GHC] #13110: GHC API allocates memory which is never GC'd
by GHC 27 Nov '17

27 Nov '17

#13110: GHC API allocates memory which is never GC'd -------------------------------------+------------------------------------- Reporter: DanielG | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: GHC API | Version: 8.0.1 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- I've run into a very strange sort of memory leak it seems. GHC seems to manage to allocate memory which I cannot get rid of no matter what I try even after all references have disappeared. The allocated memory doesn't show up in heap profiles but is still visibly allocated (as seen in ps or pmap). I discovered this while investigating reports of large memory usage in ghc-mod where this issue is forcing us to dump completion information in a separate process to avoid the allocated memory sticking around for the rest of the user's session. The attached test case demonstrates this problem by first calling `getModuleInfo` for all modules in all visible packages and then just looping forever in `main`. I would expect all memory GHC allocated to be GC'd at this point but this does not happen. When run as {{{ $ ./Leaky -hide-all-packages }}} the test case consumes around 30M of memory on my system, if we however load some large package, say GHC itself {{{ $ ./Leaky -hide-all-packages -package ghc }}} the program will consume around 2-300M of memory which are never deallocated. The behaviour is not related to the loaded package though I tried it with `Cabal` too with the same result though smaller memory usage. At first I thought this might be related to large CAFs with IORefs inside not being GCd for some reason so I tried calling `resetCAFs` from the RTS before entering the loop. This however makes no difference whatsoever. I also tried forcing a major GC just to be save to no avail. Next I speculated that maybe the allocated memory is beyond the GC's control (malloc()ed or something) but looking at the memory map of the process with `pmap $(pgrep Leaky)` that the majority of the memory usage I'm seeing comes from address `0x0000000200000000` which is where the RTS allocates memory AFAIK. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13110> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 20

[GHC] #13157: Opportunity to improve case-of-case
by GHC 24 Nov '17

24 Nov '17

#13157: Opportunity to improve case-of-case -------------------------------------+------------------------------------- Reporter: simonpj | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.0.1 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- Consider this code {{{ f g x = (case g x of True -> not False -> id) `seq` True }}} With GHC 8 (and -O) we get this Core {{{ f = \ (@ t_a1rk) (g_aqD :: t -> Bool) (x_aqE :: t) -> case case g_aqD x_aqE of { False -> id @ Bool; True -> not } of { __DEFAULT -> GHC.Types.True } }}} This is obviously bad. The reasoning is in the long, complicated `Note [Single-alternative cases]` in `Simplify.hs`. Happily with join points the entire Note becomes unnecessary; we can delete the special case that this Note describes, and the Note itself. Result: simpler compiler, and more optimal code. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13157> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 2

[GHC] #12159: Record-like GADTs with repeated fields (of same type) rejected
by GHC 22 Nov '17

22 Nov '17

#12159: Record-like GADTs with repeated fields (of same type) rejected -------------------------------------+------------------------------------- Reporter: heisenbug | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.0.1 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: GHC rejects Unknown/Multiple | valid program Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- I came across a curious bug with record-like GADTs and repeated fields. Consider following code: {{{#!hs {-# LANGUAGE GADTs #-} data Foo p where Bar :: { quux' :: Bool } -> Foo Char Baz :: { quux'' :: Bool } -> Foo Int quux :: Foo p -> Bool quux (Bar q) = q quux (Baz q) = q quuxSetter :: Foo p -> Bool -> Foo p quuxSetter old@Bar{} q = old{quux' = q} quuxSetter old@Baz{} q = old{quux'' = q} }}} This compiles and all is fine. *But* GHC is supposed to create the nice `quux` and `quuxSetter` accessors for me, right? So, let's try: {{{#!hs data Foo p where Bar :: { quux :: Bool } -> Foo Char Baz :: { quux :: Bool } -> Foo Int }}} It does not compile! Instead I get: {{{ T12159.hs:3:1: error: Constructors Bar and Baz have a common field quux , but have different result types In the data type declaration for Foo Failed, modules loaded: none. }}} This is not very polite :-) It should simply create the accessors like I did above. It obviously can be done! Testcase is attached. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12159> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 11

[GHC] #11721: GADT-syntax data constructors don't work well with TypeApplications
by GHC 20 Nov '17

20 Nov '17

#11721: GADT-syntax data constructors don't work well with TypeApplications -------------------------------------+------------------------------------- Reporter: goldfire | Owner: goldfire Type: bug | Status: new Priority: normal | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Keywords: | Operating System: Unknown/Multiple TypeApplications | Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- Consider {{{ data X a where MkX :: b -> Proxy a -> X a }}} According to the rules around specified vs. generalized variables around `TypeApplications`, the type of `MkX` should be {{{ MkX :: forall {k} (b :: *) (a :: k). b -> Proxy a -> X a }}} A few things to note: * The `k` isn't available for `TypeApplications` (that's why it's in braces), because it is not user-written. * The `b` is quantified before the `a`, because `b` comes before `a` in the user-written type signature for `MkX`. Both of these bullets are currently violated. GHCi reports `MkX`'s type as {{{ MkX :: forall k (a :: k) b. b -> Proxy a -> X a }}} It turns out that this is a hard to fix. The problem is that GHC expects data constructors to have their universal variables followed by their existential variables, always. And yet that's violated in the desired type for `MkX`. Furthermore, given the way that GHC deals with GADT return types ("rejigging", in technical parlance), it's inconvenient to get the specified/generalized distinction correct. Given time constraints, I'm afraid fixing this all won't make it for 8.0. Happily, there is are easy-to-articulate rules governing GHC's current (wrong) behavior. In a GADT-syntax data constructor: * All kind and type variables are considered specified and available for visible type application. * Universal variables always come first, in precisely the order they appear in the tycon. Note that universals that are constrained by a GADT return type are missing from the datacon. * Existential variables come next. Their order is determined by a user- written `forall`; or, if there is none, by taking the left-to-right order in the datacon's type and doing a stable topological sort. (This stable topological sort step is the same as for other user-written type signatures.) Despite the existence of these rules, it would still be better not to have special rules for GADT-syntax data constructors. This ticket is intended to capture that eventual goal. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11721> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 16

[GHC] #9118: Can't eta-reduce representational coercions
by GHC 19 Nov '17

19 Nov '17

#9118: Can't eta-reduce representational coercions ------------------------------------+------------------------------------- Reporter: goldfire | Owner: Type: feature request | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.8.2 Keywords: | Operating System: Unknown/Multiple Architecture: Unknown/Multiple | Type of failure: None/Unknown Difficulty: Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | ------------------------------------+------------------------------------- When I say {{{ import Data.Type.Coercion import Data.Coerce import Data.Proxy eta2 :: Coercible (f a) (g a) => Proxy a -> Coercion f g eta2 _ = Coercion }}} I get {{{ Could not coerce from ‘f’ to ‘g’ because ‘f’ and ‘g’ are different types. arising from a use of ‘Coercion’ from the context (Coercible (f a) (g a)) bound by the type signature for eta2 :: Coercible (f a) (g a) => Proxy a -> Coercion f g at /Users/rae/temp/Roles.hs:5:9-56 In the expression: Coercion In an equation for ‘eta2’: eta2 _ = Coercion }}} Unlike the case for #9116, this one is ''not'' expressible in Core. (At least, I don't see a way to do it.) So, to do this, we would have to update Core and then update the constraint solver. I ''do'' think this would be type safe. But, I also think it's reasonable to wait for someone with a real use case to shout before doing this. The only use case I have is to be able to do this: {{{ -- says that a's parameter is representational class Rep (a :: k1 -> k2) where co :: Coercible x y => Coercible (a x) (a x) instance Rep f => Rep (Compose f) where ... }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/9118> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 9

[GHC] #11392: Decide and document how semicolons are supposed to work in GHCi
by GHC 18 Nov '17

18 Nov '17

#11392: Decide and document how semicolons are supposed to work in GHCi -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: task | Status: new Priority: normal | Milestone: 8.2.1 Component: GHCi | Version: 7.10.3 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: #10663 Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- Currently there is a fair bit of confusion surrounding the intended function of semicolons in GHCi (see #10663 and Phab:D1726), * the users guide doesn't discuss the matter at all * sometimes the content after semicolons work as expected {{{ λ> let x=2; y=5 λ> x+y 7 }}} * other times it is silently ignored (see #10663), {{{ λ> import Data.List; 684 λ> }}} * other times it works but the parser state is subtly wrong, resulting in (arguably) incorrect line numbers later in the session, {{{ λ> data Infix a b = a :@: b; infixl 4 :@: λ> a <interactive>:3:1: error: Variable not in scope: a }}} * some times the only way to accomplish a given task is with a semicolon, {{{ Prelude> data Infix a b = a :@: b; infixl 4 :@: -- this is OK, but: Prelude> data Infix a b = a :@: b Prelude> infixl 4 :@: <interactive>:12:10: error: The fixity signature for `:@:' lacks an accompanying binding (The fixity signature must be given where `:@:' is declared) }}} This inconsistency is surprising and at times frustrating. Let's decide how we want semicolons to work and stick with it. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11392> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 5

[GHC] #9052: Support a "stable heap" which doesn't get garbage collected
by GHC 13 Nov '17

13 Nov '17

#9052: Support a "stable heap" which doesn't get garbage collected ----------------------------------------------+---------------------------- Reporter: ezyang | Owner: Type: feature request | simonmar Priority: normal | Status: new Component: Runtime System | Milestone: Keywords: | Version: 7.9 Architecture: Unknown/Multiple | Operating System: Difficulty: Moderate (less than a day) | Unknown/Multiple Blocked By: | Type of failure: Related Tickets: | None/Unknown | Test Case: | Blocking: ----------------------------------------------+---------------------------- Simon Marlow was explaining the concept on IRC. The problem we're trying to solve is you might have some very long-lived pointer data-structures, and you don't want to waste time traversing this structure on a major GC when you know that it is still going to be alive. Here's the basic idea: 1. We have a new generation on the heap called the 'stable heap' (along- side the nursery, gen0, gen1, etc). By default, no objects are automatically promoted from the old gen to the stable heap. 2. A user can request an object be moved to the stable heap, creating a stable pointer which points to the object. 3. During garbage collection, all blocks in the stable heap are always considered reachable. We don't include the stable pointers in the root set (so we don't trace the stable heap), however, write barriers are processed as usual, and we promote any objects pointed to by the stable heap to the stable heap. 4. Users manually request the stable heap to be garbage-collected, usually after deallocating stable pointers. In short, the stable heap is just another generation, but one which (1) is only collected by user request, and (2) never receives objects from automatic promotion. This project fits in very nicely with the existing GC infrastructure, so if you understand how everything is laid out this should be reasonably easy to implement. Could be a good ticket for someone who wants to get their hands dirty with the GC. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/9052> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 6

[GHC] #9420: Impredicative type instantiation without -XImpredicativeTypes
by GHC 11 Nov '17

11 Nov '17

#9420: Impredicative type instantiation without -XImpredicativeTypes -------------------------------------+------------------------------------- Reporter: goldfire | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.9 Keywords: | Operating System: Architecture: Unknown/Multiple | Unknown/Multiple Difficulty: Unknown | Type of failure: Blocked By: | None/Unknown Related Tickets: | Test Case: | Blocking: | Differential Revisions: -------------------------------------+------------------------------------- Consider this module: {{{ {-# LANGUAGE RankNTypes #-} module Bug where rank2 :: ((forall a. a -> a) -> b) -> () rank2 _ = () foo :: () -> () foo x = x quux :: (forall a. a -> a) -> Int quux _ = 5 bar = foo . rank2 $ quux }}} The `(.)` in the definition of `bar` requires an impredicative instantiation -- that is, one of its type variables is instantiated to a forall-type. Yet, the module compiles without `-XImpredicativeTypes`. To confirm this behavior, the following is an excerpt from `-ddump-simpl`: {{{ Bug.bar = break<7>() (break<6>() GHC.Base.. @ () @ () @ ((forall a_a1Tj. a_a1Tj -> a_a1Tj) -> GHC.Types.Int) Bug.foo (Bug.rank2 @ GHC.Types.Int)) Bug.quux }}} Note the third type argument to `(.)`. Fixing this would be a breaking change that could affect users. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/9420> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 4