May 2017 - ghc-tickets

[GHC] #8272: testing if SpLim=$rbp and Sp=$rsp changed performance at all
by GHC 15 Dec '17

15 Dec '17

#8272: testing if SpLim=$rbp and Sp=$rsp changed performance at all ----------------------------------------------+---------------------------- Reporter: carter | Owner: carter Type: task | Status: new Priority: normal | Milestone: 7.10.1 Component: Compiler | Version: 7.7 Keywords: callingConvention | Operating System: Architecture: x86_64 (amd64) | Unknown/Multiple Difficulty: Project (more than a week) | Type of failure: Blocked By: | None/Unknown Related Tickets: | Test Case: | Blocking: ----------------------------------------------+---------------------------- testing if SpLim=$rbp and Sp=$rsp changed performance at all would need a stack check but then push could be used to spill to the stack Idea via Nathan Howell. At the very least, the x86 PUSH instruction has a more succinct encoding than MOV. worth hacking out to see if this can measurable shift ghc perf on nofib or not. this would be part of a larger effort to explore ways to improve GHC's calling convention for modern hardware -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8272> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 16

[GHC] #11203: Kind inference with SigTvs is wrong
by GHC 15 Dec '17

15 Dec '17

#11203: Kind inference with SigTvs is wrong -------------------------------------+------------------------------------- Reporter: goldfire | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- Consider {{{#!hs data SameKind :: k -> k -> * data Q (a :: k1) (b :: k2) c = MkQ (SameKind a b) }}} This code should be rejected. Yet it is accepted. The problem is that, when kind-checking a datatype declaration without a CUSK, GHC uses `SigTv`s for user-written kind variables. `SigTv`s are allowed to unify with other `SigTv`s, leading to incorrect behavior here. The motivating scenario is this: {{{#!hs data T (a :: k1) x = MkT (S a ()) data S (b :: k2) y = MkS (T b ()) }}} This program should be accepted. Neither type has a CUSK and therefore is not generalized before kind-checking the group. GHC must then unify `k1` and `k2`. If they were skolems, this unification would fail and this pair of definitions would be rejected. This ticket is identical in spirit to #9201, but that example has a CUSK and thus works. The bug can happen only when there is no CUSK. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11203> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 12

[GHC] #12372: bug: documentation for Control.Monad.guard not useful after AMP
by GHC 14 Dec '17

14 Dec '17

#12372: bug: documentation for Control.Monad.guard not useful after AMP -------------------------------------+------------------------------------- Reporter: ntc2 | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: | Version: 8.0.1 libraries/base | Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: Documentation Unknown/Multiple | bug Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- Since the AMP refactor, the documentation for `Control.Monad.guard` [1] is no longer useful for beginners. It simply gives the definition of `guard`, but in prose: {{{ guard b is pure () if b is True, and empty if b is False. }}} (and better to just use Haskell instead of prose here, no?) To use `guard` in a `MonadPlus`, you now need to know that `Alternative` is a super class of `MonadPlus`, and that `mzero = zero`. The documentation [2] for `MonadPlus` doens't mention `mzero = zero` in the default definition -- you must look at the source for that -- and the docs for `guard` don't mention `MonadPlus`. The documentation for `Control.Monad.guard` should suggest use with `MonadPlus`, and give an example (compare with the very helpful example for `Control.Monad.when`). A non-monadic example would also be useful. [1] https://hackage.haskell.org/package/base-4.9.0.0/docs/Control- Monad.html#v:guard [2] https://hackage.haskell.org/package/base-4.9.0.0/docs/Control- Monad.html#t:MonadPlus -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12372> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 9

[GHC] #9364: GHCi (or haskeline?) confused by non-single-width characters
by GHC 13 Dec '17

13 Dec '17

#9364: GHCi (or haskeline?) confused by non-single-width characters -------------------------------------+------------------------------------- Reporter: cheater | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: GHCi | Version: 7.8.3 Keywords: linux, terminal, | Operating System: colour, color, prompt | Unknown/Multiple Architecture: Unknown/Multiple | Type of failure: Other Difficulty: Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Hi, there's an issue in the GHCi prompt. To reproduce: 1. Do everything in Linux (I don't have other operating systems); I'm using a version of the gnome 2 gnome-terminal. 2. Use the following to colourise the prompt in GHCi: echo -e :set prompt '"\033[32;1m%s\033[34;1m>\033[0m "' >> ~/.ghc/ghci.conf 3. Import quite a few modules (say, 5 to 10 should be enough) 4. Give the prompt enough text that it wraps around. It's best if you start the text with -- so that it's a comment and doesn't produce any output. For example, say I entered "-- bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", I now see the following: *Types Control.Monad.Writer Control.Monad Data.Functor Control.Applicative Data.Monoid> -- bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb (do note that the first set of b's is located on the same line as the prompt, and the second is located under the prompt) 5. Press Enter. You should now see: *Types Control.Monad.Writer Control.Monad Data.Functor Control.Applicative Data.Monoid> -- bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb *Types Control.Monad.Writer Control.Monad Data.Functor Control.Applicative Data.Monoid> 6. Press Ctrl-P or the up arrow key to select the previous history item. You will now see the following: *Types Control.Monad.Writer Control.Monad Data.Functor Control.Applicative Data.Monoid> -- bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb *Types Control.Monad.Writer Control.Monad Data.Functor Control.Applicative Data.Monoid> -- bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb 7. Press Ctrl-A or the Home cursor key to go to the beginning of the text entry, and then type X. You will now see the last prompt to be: *Types Control.Monad.Writer Control.Monad Data.Functor Control.Applicative Data.Monoid> -- bbbbbbbbbbbbX-- bbbbbbbbbbbbbbbbb bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb 8. If you press Enter, you will find out that prompt was actually (without line breaks): X-- bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb I tried resizing the terminal window to 45 chars to make the lines above shorter; I imported Data.Monoid and Data.Functor in a new GHCi. The prompt contained garbage, most likely badly wrapped escape codes. See attachment. As both likely are a single bug in disguise, I include both reports in this ticket. Creating a test case for whether things actually work in a terminal might be difficult, but creating test for whether things get wrapped correctly (and not testing how things work out in a terminal) should be simple if annoying. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/9364> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 7

[GHC] #13210: Can't run terminfo code in GHCi on Windows
by GHC 12 Dec '17

12 Dec '17

#13210: Can't run terminfo code in GHCi on Windows -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Runtime | Version: 8.0.2 System (Linker) | Keywords: | Operating System: Windows Architecture: | Type of failure: GHC rejects Unknown/Multiple | valid program Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- It's possible this is a duplicate of another runtime linker on Windows issue out there, but I'll post this just in case this really is a unique case. I was attempting to compile the `terminfo` library on Windows, and I actually made it surprisingly far--but I quickly stumbled when I actually tried to run some code in GHCi. First, make sure you have a native Windows version of `ncurses` installed. On MSYS2: {{{ $ pacman -S mingw-w64-x86_64-ncurses }}} And here's a stripped-down version of `terminfo`, which I'll use to demonstrate the bug: {{{#!hs module Main where import Control.Exception import Foreign import Foreign.C.String import Foreign.C.Types data TERMINAL newtype Terminal = Terminal (ForeignPtr TERMINAL) -- Use "unsafe" to make set_curterm faster since it's called quite a bit. foreign import ccall unsafe set_curterm :: Ptr TERMINAL -> IO (Ptr TERMINAL) foreign import ccall "&" del_curterm :: FunPtr (Ptr TERMINAL -> IO ()) foreign import ccall setupterm :: CString -> CInt -> Ptr CInt -> IO () setupTerm :: String -> IO Terminal setupTerm term = withCString term $ \c_term -> with 0 $ \ret_ptr -> do -- NOTE: I believe that for the way we use terminfo -- (i.e. custom output function) -- this parameter does not affect anything. let stdOutput = 1 -- Save the previous terminal to be restored after calling setupterm. old_term <- set_curterm nullPtr -- Call setupterm and check the return value. setupterm c_term stdOutput ret_ptr ret <- peek ret_ptr if (ret /=1) then throwIO $ SetupTermError $ "Couldn't look up terminfo entry " ++ show term else do cterm <- set_curterm old_term fmap Terminal $ newForeignPtr del_curterm cterm data SetupTermError = SetupTermError String instance Show SetupTermError where show (SetupTermError str) = "setupTerm: " ++ str instance Exception SetupTermError where main :: IO () main = do Terminal t <- setupTerm "xterm" print t }}} If you compile this code, it "works" (in the sense that it'll actually call the C code): {{{ $ ghc -lncursesw Terminfo.hs [1 of 1] Compiling Main ( Terminfo.hs, Terminfo.o ) Linking Terminfo.exe ... $ ./Terminfo Terminfo.exe: setupTerm: Couldn't look up terminfo entry "xterm" }}} But attempting to run the same code in GHCi is a disaster: {{{ $ runghc -lncursesw Terminfo.hs ghc.exe: C:/msys64/mingw64/lib/../lib/libncursesw.a: unknown symbol `nanosleep' ghc.exe: Could not on-demand load symbol '_nc_cookie_init' ghc.exe: C:/msys64/mingw64/lib/../lib/libncursesw.a: unknown symbol `_nc_cookie_init' ghc.exe: Could not on-demand load symbol '.refptr._nc_wacs' ghc.exe: C:/msys64/mingw64/lib/../lib/libncursesw.a: unknown symbol `.refptr._nc_wacs' ghc.exe: Could not on-demand load symbol '_nc_WIN_DRIVER' ghc.exe: C:/msys64/mingw64/lib/../lib/libncursesw.a: unknown symbol `_nc_WIN_DRIVER' ghc.exe: Could not on-demand load symbol '_nc_get_driver' ghc.exe: C:/msys64/mingw64/lib/../lib/libncursesw.a: unknown symbol `_nc_get_driver' ghc.exe: Could not on-demand load symbol '_nc_setupterm_ex' ghc.exe: C:/msys64/mingw64/lib/../lib/libncursesw.a: unknown symbol `_nc_setupterm_ex' ghc.exe: Could not on-demand load symbol 'newterm' ghc.exe: C:/msys64/mingw64/lib/../lib/libncursesw.a: unknown symbol `newterm' ghc.exe: Could not on-demand load symbol '.refptr._nc_globals' ghc.exe: C:/msys64/mingw64/lib/../lib/libncursesw.a: unknown symbol `.refptr._nc_globals' ghc.exe: Could not on-demand load symbol '_nc_err_abort' ghc.exe: C:/msys64/mingw64/lib/../lib/libncursesw.a: unknown symbol `_nc_err_abort' ghc.exe: Could not on-demand load symbol 'tparm' ghc.exe: C:/msys64/mingw64/lib/../lib/libncursesw.a: unknown symbol `tparm' ghc.exe: Could not on-demand load symbol '.refptr._nc_prescreen' ghc.exe: C:/msys64/mingw64/lib/../lib/libncursesw.a: unknown symbol `.refptr._nc_prescreen' ghc.exe: Could not on-demand load symbol '_nc_outch_sp' ghc.exe: C:/msys64/mingw64/lib/../lib/libncursesw.a: unknown symbol `_nc_outch_sp' ghc.exe: Could not on-demand load symbol '.refptr._nc_outch_sp' ghc.exe: C:/msys64/mingw64/lib/../lib/libncursesw.a: unknown symbol `.refptr._nc_outch_sp' ghc.exe: Could not on-demand load symbol '_nc_scrolln_sp' ghc.exe: C:/msys64/mingw64/lib/../lib/libncursesw.a: unknown symbol `_nc_scrolln_sp' ghc.exe: Could not on-demand load symbol '.refptr.SP' ghc.exe: C:/msys64/mingw64/lib/../lib/libncursesw.a: unknown symbol `.refptr.SP' ghc.exe: Could not on-demand load symbol '.refptr.cur_term' ghc.exe: C:/msys64/mingw64/lib/../lib/libncursesw.a: unknown symbol `.refptr.cur_term' ghc.exe: Could not on-demand load symbol '_nc_ospeed' ghc.exe: C:/msys64/mingw64/lib/../lib/libncursesw.a: unknown symbol `_nc_ospeed' ghc.exe: Could not on-demand load symbol 'set_curterm' Terminfo.hs: ByteCodeLink: can't find label During interactive linking, GHCi couldn't find the following symbol: set_curterm This may be due to you not asking GHCi to load extra object files, archives or DLLs needed by your current session. Restart GHCi, specifying the missing library using the -L/path/to/object/dir and -lmissinglibname flags, or simply by naming the relevant files on the GHCi command line. Alternatively, this link failure might indicate a bug in GHCi. If you suspect the latter, please send a bug report to: glasgow-haskell-bugs(a)haskell.org }}} Also reproducible on GHC HEAD. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13210> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 9

[GHC] #9669: Long compile time/high memory usage for modules with many deriving clauses
by GHC 11 Dec '17

11 Dec '17

#9669: Long compile time/high memory usage for modules with many deriving clauses -------------------------------------+------------------------------------- Reporter: snoyberg | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.8.3 Keywords: | Operating System: Linux Architecture: x86_64 (amd64) | Type of failure: Compile- Difficulty: Unknown | time performance bug Blocked By: | Test Case: Related Tickets: | Blocking: | Differential Revisions: -------------------------------------+------------------------------------- I've seen many different people complain about this in various different contexts. One of the most common I personally see is when using Persistent with a large number of entities, which results in multi-minute build times for that single module. Usually, these reports have come from private repositories, which makes for a difficult repro. Also, the usage of Template Haskell in those modules tends to confuse the issue. So I'd like to report this issue from a completely separate project, with the guess that this represents an issue that is affecting many other users. Steps to reproduce: * `cabal unpack fpco-api-1.2.0.1` * `cd fpco-api-1.2.0.1` * `cabal install` * `ghc --make -isrc/library src/library/FP/API/Types.hs` * `touch src/library/FP/API/Types.hs` Then, to see the time and memory usage of compiling just the one module, run: * `time ghc -O2 --make -isrc/library src/library/FP/API/Types.hs +RTS -s` On my system (16GB RAM, quadcore i7), the results are: {{{ $ time ghc -O2 --make -isrc/library src/library/FP/API/Types.hs +RTS -s [3 of 3] Compiling FP.API.Types ( src/library/FP/API/Types.hs, src/library/FP/API/Types.o ) 51,846,533,568 bytes allocated in the heap 6,799,246,288 bytes copied during GC 389,297,584 bytes maximum residency (22 sample(s)) 11,330,472 bytes maximum slop 1041 MB total memory in use (0 MB lost due to fragmentation) Tot time (elapsed) Avg pause Max pause Gen 0 2041 colls, 0 par 9.19s 9.19s 0.0045s 0.0998s Gen 1 22 colls, 0 par 4.33s 4.33s 0.1969s 0.5436s TASKS: 4 (1 bound, 3 peak workers (3 total), using -N1) SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled) INIT time 0.00s ( 0.00s elapsed) MUT time 34.21s ( 37.74s elapsed) GC time 13.52s ( 13.53s elapsed) EXIT time 0.07s ( 0.07s elapsed) Total time 47.81s ( 51.34s elapsed) Alloc rate 1,515,662,937 bytes per MUT second Productivity 71.7% of total user, 66.8% of total elapsed gc_alloc_block_sync: 0 whitehole_spin: 0 gen[0].sync: 0 gen[1].sync: 0 real 0m51.375s user 0m50.864s sys 0m0.456s }}} By contrast, with optimizations turned off: {{{ $ time ghc -O0 --make -isrc/library src/library/FP/API/Types.hs +RTS -s [3 of 3] Compiling FP.API.Types ( src/library/FP/API/Types.hs, src/library/FP/API/Types.o ) 12,767,593,936 bytes allocated in the heap 1,078,202,664 bytes copied during GC 179,551,768 bytes maximum residency (13 sample(s)) 6,684,544 bytes maximum slop 439 MB total memory in use (0 MB lost due to fragmentation) Tot time (elapsed) Avg pause Max pause Gen 0 623 colls, 0 par 1.24s 1.24s 0.0020s 0.0651s Gen 1 13 colls, 0 par 0.76s 0.76s 0.0586s 0.2397s TASKS: 4 (1 bound, 3 peak workers (3 total), using -N1) SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled) INIT time 0.00s ( 0.00s elapsed) MUT time 8.28s ( 9.23s elapsed) GC time 2.00s ( 2.00s elapsed) EXIT time 0.05s ( 0.05s elapsed) Total time 10.34s ( 11.28s elapsed) Alloc rate 1,542,443,897 bytes per MUT second Productivity 80.6% of total user, 73.9% of total elapsed gc_alloc_block_sync: 0 whitehole_spin: 0 gen[0].sync: 0 gen[1].sync: 0 real 0m11.304s user 0m11.108s sys 0m0.172s }}} and -O1: {{{ $ time ghc -O1 --make -isrc/library src/library/FP/API/Types.hs +RTS -s [3 of 3] Compiling FP.API.Types ( src/library/FP/API/Types.hs, src/library/FP/API/Types.o ) 45,550,443,664 bytes allocated in the heap 5,721,700,512 bytes copied during GC 358,036,456 bytes maximum residency (21 sample(s)) 9,167,176 bytes maximum slop 906 MB total memory in use (0 MB lost due to fragmentation) Tot time (elapsed) Avg pause Max pause Gen 0 1642 colls, 0 par 7.70s 7.70s 0.0047s 0.1031s Gen 1 21 colls, 0 par 3.68s 3.69s 0.1756s 0.4968s TASKS: 4 (1 bound, 3 peak workers (3 total), using -N1) SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled) INIT time 0.00s ( 0.00s elapsed) MUT time 30.21s ( 33.09s elapsed) GC time 11.38s ( 11.39s elapsed) EXIT time 0.07s ( 0.07s elapsed) Total time 41.68s ( 44.55s elapsed) Alloc rate 1,507,571,106 bytes per MUT second Productivity 72.7% of total user, 68.0% of total elapsed gc_alloc_block_sync: 0 whitehole_spin: 0 gen[0].sync: 0 gen[1].sync: 0 real 0m44.577s user 0m44.164s sys 0m0.368s }}} Note that this module is essentially just 900 lines of data declarations, with a large number of derived classes, and a few manually written helper functions and instances. This high memory usage hasn't just resulted in user complaints: automated Jenkins and Travis jobs will often times fail without optimizations disabled, which can be problematic for proper testing and production code. In the case of fpco-api, I've worked around this by adding `-O0` to the cabal file, but it would be much nicer to not have to do that. For some other examples of complaints along these lines: * https://groups.google.com/d/msg/yesodweb/MX1bnOFu8Hc/VLQXhnvpIkMJ * https://groups.google.com/d/msg/yesodweb/XPWixNjuOnM/FN26bmkudgwJ I believe there are a few other threads discussing this, if it would be helpful. I did my testing on GHC 7.8.3 64-bit, Ubuntu 12.04. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/9669> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 19

[GHC] #13604: regression in ghc 8.2.1-rc1 (8.2.0.2017040)
by GHC 11 Dec '17

11 Dec '17

#13604: regression in ghc 8.2.1-rc1 (8.2.0.2017040) -------------------------------------+------------------------------------- Reporter: George | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: GHCi | Version: 8.0.1 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: Runtime Unknown/Multiple | performance bug Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- In 8.2.1-rc1 loading a file compiled with -O2 into ghci results in ghci recompiling the file into interpreted byte code. In 8.0.2 it simply loads the compiled object file. 8.2.1 {{{ ghc -dynamic -O2 eh2.hs [1 of 1] Compiling Main ( eh2.hs, eh2.o ) Linking eh2 ... bash-3.2$ ghci -ignore-dot-ghci GHCi, version 8.2.0.20170404: http://www.haskell.org/ghc/ :? for help Prelude> :load eh2 [1 of 1] Compiling Main ( eh2.hs, interpreted ) [flags changed] Ok, modules loaded: Main. }}} 8.0.2 {{{ ghc --version The Glorious Glasgow Haskell Compilation System, version 8.0.2 bash-3.2$ pwd /Users/gcolpitts/haskell bash-3.2$ ghc -dynamic -O2 eh2.hs [1 of 1] Compiling Main ( eh2.hs, eh2.o ) Linking eh2 ... bash-3.2$ ghci -ignore-dot-ghci GHCi, version 8.0.2: http://www.haskell.org/ghc/ :? for help Prelude> :load eh2 Ok, modules loaded: Main (eh2.o). }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13604> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 52

[GHC] #8457: -ffull-laziness does more harm than good
by GHC 08 Dec '17

08 Dec '17

#8457: -ffull-laziness does more harm than good ------------------------------------+------------------------------------- Reporter: errge | Owner: Type: bug | Status: new Priority: high | Milestone: 7.8.1 Component: Compiler | Version: 7.7 Keywords: | Operating System: Unknown/Multiple Architecture: Unknown/Multiple | Type of failure: None/Unknown Difficulty: Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | ------------------------------------+------------------------------------- In this bug report I'd like to argue that `-ffull-laziness` shouldn't be turned on automatically with either `-O` nor `-O2`, because it's dangerous and can cause serious memory leaks which are hard to debug or prevent. I'll also try to show that its optimization benefits are negligible. Actually, my benchmarks show that it's beneficial to turn it off even in the cases where we don't hit a space leak. We've met this issue last week, but it had been reported several times before: e.g. #917 and #5262. A typical example is the following: {{{ #!haskell main :: IO () main = task () >> task () task :: () -> IO () task () = printvalues [1..1000000 :: Int] printvalues :: [Int] -> IO () printvalues (x:xs) = print x >> printvalues xs printvalues [] = return () }}} We succeed with `-O0`, but fail with `-O`: {{{ errge@curry:~/tmp $ ~/tmp/ghc/inplace/bin/ghc-stage2 -v0 -O0 -fforce- recomp lazy && ./lazy +RTS -t >/dev/null <<ghc: 1620098744 bytes, 3117 GCs, 32265/42580 avg/max bytes residency (3 samples), 2M in use, 0.00 INIT (0.00 elapsed), 1.28 MUT (1.28 elapsed), 0.02 GC (0.02 elapsed) :ghc>> errge@curry:~/tmp $ ~/tmp/ghc/inplace/bin/ghc-stage2 -v0 -O -fforce-recomp lazy && ./lazy +RTS -t >/dev/null <<ghc: 1444098612 bytes, 2761 GCs, 3812497/13044272 avg/max bytes residency (7 samples), 28M in use, 0.00 INIT (0.00 elapsed), 1.02 MUT (1.03 elapsed), 0.12 GC (0.12 elapsed) :ghc>> }}} 28M? What the leak!? Well, it's `-ffull-laziness`: {{{ errge@curry:~/tmp $ ~/tmp/ghc/inplace/bin/ghc-stage2 -v0 -O -fno-full- laziness -fforce-recomp lazy && ./lazy +RTS -t >/dev/null <<ghc: 1484098612 bytes, 2835 GCs, 34812/42580 avg/max bytes residency (2 samples), 1M in use, 0.00 INIT (0.00 elapsed), 1.04 MUT (1.04 elapsed), 0.02 GC (0.02 elapsed) :ghc>> }}} We get constant space and the fastest run-time too, since we spare some cycles on GC. Note, that in this instance we are trying to explicity disable sharing by using `()` as a fake argument for the function. Also note, that this function may easily be a utility function in a larger code base or in a library, therefore it's impractical to say that you shouldn't use it twice "too close together". Quoting from the GHC user guide: {{{ -O2: Means: “Apply every non-dangerous optimisation, even if it means significantly longer compile times.” The avoided “dangerous” optimisations are those that can make runtime or space worse if you're unlucky. They are normally turned on or off individually. At the moment, -O2 is unlikely to produce better code than -O. }}} This seems to be false at the moment. We decided to make a broader investigation into this issue and wanted to know if we can disable this optimization without too much pain. Came up with this benchmark plan: - let's benchmark GHC, - compile all stages with -O, but hack the stage1 compiler to emit `-t` statistics for every file compiled, - gather these statistics while compiling the libraries and the stage2 compiler. On the second run we compile the stage1 compiler with `-O -fno-full-laziness`, but leave everything else unchanged in the environment. When we have both results of the compilation of ~1600 files, we match them up and compute the (logarithmic) ratio of CPU and memory difference between compilations, the final results for our benchmark. The results and the raw data can be found at https://github.com/errge/notlazy. The overall compilation time dropped from 26:20 to 25:12, which is a 4% improvement. Investigating the full matching shows that this overall result is from small improvements all around the place. The results plotted: - https://github.com/errge/notlazy/blob/master/cpu.png - https://github.com/errge/notlazy/blob/master/mem.png The graphs show the logarithmic (100*log_10(new/orig)) ratio of change in cpu and memory consumption. Therefore negative results mean that the new compilation method is faster. As can be seen on the CPU graph, in most of the cases the difference is negligible (actually smaller than what can be measured on small files, this is why we have the spike at 0). In overall we see a small improvement in CPU, and there are some outliers in both directions, but there are more drastic improvement cases than drastic regressions. On the memory graph the situation is much more close to zero. There is one big positive memory outlier: `DsListComp.lhs`. It uses 69M originally and now uses 103M. But compiles in 2 seconds both ways and there are files in the source tree which requires 400M to compile, so this is not an issue. After all this, I'd like to hear other opinions about just disabling this optimization in `-O` and `-O2` and leaving it as an option that can be turned on when needed, my reasons once more: - it's unsafe, - it's hard to debug when you hit its issues, - the optimization doesn't seem to be very productive, - it's always easy to force sharing, but it's not easy to force copying. Apparently a Haskell programmer should be lazy, but never fully lazy. Research done by Gergely Risko <errge> and Mihaly Barasz <klao>, confirmed on two different machines with no other running processes. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8457> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 14

[GHC] #10535: Float out causes major space leak
by GHC 08 Dec '17

08 Dec '17

#10535: Float out causes major space leak -------------------------------------+------------------------------------- Reporter: AlexET | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.1 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Blocked By: Test Case: | Related Tickets: Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- This issue originally was on haskell-cafe https://mail.haskell.org/pipermail/haskell-cafe/2015-June/120113.html The following code has a space leak {{{ #!hs numbers = [1..200] replicateM' :: Monad m => Int -> m a -> m [a] replicateM' 0 xs = return [] replicateM' n xs = do a <- xs b <- replicateM' (n-1) xs return (a:b) test :: [[Int]] test = replicateM' 4 numbers main = print test }}} The recursive call in replicateM' gets floated out. This then causes a very large (200^3 elements) list to be kept in memory instead of being lazily produced and consumed each time. A similar thing happens to the replicateM in Control.Monad but it is harder to spot as a lot of specialisation and fusion goes on before the float-out happens This is similar to #7367 but in this case we have a huge space-leak instead of just adding allocations. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/10535> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 7

[GHC] #13110: GHC API allocates memory which is never GC'd
by GHC 27 Nov '17

27 Nov '17

#13110: GHC API allocates memory which is never GC'd -------------------------------------+------------------------------------- Reporter: DanielG | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: GHC API | Version: 8.0.1 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- I've run into a very strange sort of memory leak it seems. GHC seems to manage to allocate memory which I cannot get rid of no matter what I try even after all references have disappeared. The allocated memory doesn't show up in heap profiles but is still visibly allocated (as seen in ps or pmap). I discovered this while investigating reports of large memory usage in ghc-mod where this issue is forcing us to dump completion information in a separate process to avoid the allocated memory sticking around for the rest of the user's session. The attached test case demonstrates this problem by first calling `getModuleInfo` for all modules in all visible packages and then just looping forever in `main`. I would expect all memory GHC allocated to be GC'd at this point but this does not happen. When run as {{{ $ ./Leaky -hide-all-packages }}} the test case consumes around 30M of memory on my system, if we however load some large package, say GHC itself {{{ $ ./Leaky -hide-all-packages -package ghc }}} the program will consume around 2-300M of memory which are never deallocated. The behaviour is not related to the loaded package though I tried it with `Cabal` too with the same result though smaller memory usage. At first I thought this might be related to large CAFs with IORefs inside not being GCd for some reason so I tried calling `resetCAFs` from the RTS before entering the loop. This however makes no difference whatsoever. I also tried forcing a major GC just to be save to no avail. Next I speculated that maybe the allocated memory is beyond the GC's control (malloc()ed or something) but looking at the memory map of the process with `pmap $(pgrep Leaky)` that the majority of the memory usage I'm seeing comes from address `0x0000000200000000` which is where the RTS allocates memory AFAIK. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13110> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 20