- ghc-tickets - Haskell.org

[GHC] #13630: ByteString pinned memory can be leaky
by GHC 04 Jan '19

04 Jan '19

#13630: ByteString pinned memory can be leaky -------------------------------------+------------------------------------- Reporter: nh2 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Runtime | Version: 8.0.1 System | Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- My question on IRC: {{{ How does memory allocation for pinned blocks work? Let's say pinned blocks are 4KB in size, and I allocate first a 3 KB ByteString A and then an 8-byte ByteString B. Now I GC A, no longer need it. Then according to https://ghc.haskell.org/trac/ghc/wiki/Commentary/Rts/Storage/GC/Pinned "a single pinned object keeps alive the whole block in which it resides", my small ByteString B keeps the entire block alive. But what happens with the 3KB in the front of that block? Will it be re-used by the next ByteString allocation (say 1KB)? In other words, how smart is allocatePinned as an allocator? }}} The answer: {{{ slyfox: nh2: allocatePinned it quite dump. it only allocated from free tail space }}} {{{ nh2: slyfox: that sounds like a huge potential for memory leak then slyfox: yes, fragmentation is quite bad for bytestrings }}} So it seems that I can get into the unfortunate situation where a super short `ByteString` of a few bytes can waste an entire 4 KB block of memory; some migth call this a leak. One idea to solve it seems to be to change standard `ByteStrings` to not pinned, and to allocate pinned ones explicitly when needed. This seems to be an often-discussed topic and not trivial because many `ByteString` functions are implemented using libc FFI functions. However, it seems there will always be _some_ need for pinned memory, so we should better have an efficient way to manage it in any case. Efficient here means, for example, to re-use freed memory inside a block instead of only using free tail space. It seems that `jemalloc` has a feature to use given blocks of memory and provide a `malloc()` functionality inside them: https://stackoverflow.com/questions/30836359/jemalloc-mmap-and-shared- memory Perhaps this could be used to provide GHC with a simple method to use pinned memory more efficiently? -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13630> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 4

[GHC] #14797: High-residency modules during GHC build
by GHC 04 Jan '19

04 Jan '19

#14797: High-residency modules during GHC build -------------------------------------+------------------------------------- Reporter: tdammers | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.2.1 Keywords: | Operating System: Linux Architecture: x86_64 | Type of failure: Runtime (amd64) | performance bug Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- While trying to figure out memory allocation failures on the CI server, I tracked allocations and residency on a per-module basis, and noticed that some modules require a lot more memory to compile than others. I've compiled a digest of the most memory-hungry compilation steps: {{{ "/usr/local/bin/ghc" ... compiler/stage1/build/OptCoercion.o: Max. residency: 991356584 / In use: 2094M "inplace/bin/ghc-stage1" ... compiler/stage2/build/OptCoercion.dyn_o: Max. residency: 714940528 / In use: 1471M "/usr/local/bin/ghc" ... compiler/stage1/build/X86/CodeGen.o: Max. residency: 679947656 / In use: 1497M "inplace/bin/ghc-stage1" ... libraries/Cabal/Cabal/dist- install/build/Distribution/SPDX/LicenseId.dyn_o: Max. residency: 653177496 / In use: 1355M "inplace/bin/ghc-stage1" ... compiler/stage2/build/X86/CodeGen.dyn_o: Max. residency: 641515344 / In use: 1425M "/usr/local/bin/ghc" ... -ilibraries/parsec: Max. residency: 593335568 / In use: 1355M "/usr/local/bin/ghc" ... libraries/Cabal/Cabal/dist- boot/build/Distribution/SPDX/LicenseId.o: Max. residency: 575034968 / In use: 1137M "inplace/bin/ghc-stage1" ... compiler/stage2/build/DynFlags.dyn_o: Max. residency: 402734408 / In use: 1059M "/usr/local/bin/ghc" ... compiler/stage1/build/DynFlags.o: Max. residency: 343154784 / In use: 892M "inplace/bin/ghc-stage1" ... libraries/template-haskell/dist- install/build/Language/Haskell/TH/Syntax.dyn_o: Max. residency: 266744312 / In use: 720M "/usr/local/bin/ghc" ... compiler/stage1/build/Parser.o: Max. residency: 258613584 / In use: 679M "inplace/bin/ghc-stage1" ... compiler/stage2/build/CmmOpt.dyn_o: Max. residency: 254639624 / In use: 568M "inplace/bin/ghc-stage1" ... compiler/stage2/build/Parser.dyn_o: Max. residency: 236774592 / In use: 635M "inplace/bin/ghc-stage1" ... ghc/stage2/build/GHCi/UI.dyn_o: Max. residency: 234055792 / In use: 609M "/usr/local/bin/ghc" ... compiler/stage1/build/HsExpr.o: Max. residency: 233724400 / In use: 657M "/usr/local/bin/ghc" ... compiler/stage1/build/CmmOpt.o: Max. residency: 208378888 / In use: 436M "/usr/local/bin/ghc" ... compiler/stage1/build/HscMain.o: Max. residency: 197177248 / In use: 500M "inplace/bin/ghc-stage1" ... compiler/stage2/build/HsExpr.dyn_o: Max. residency: 192780496 / In use: 521M "inplace/bin/ghc-stage1" ... libraries/Cabal/Cabal/dist- install/build/Distribution/Simple/Setup.dyn_o: Max. residency: 191649952 / In use: 521M "/usr/local/bin/ghc" ... compiler/stage1/build/HsDecls.o: Max. residency: 183441120 / In use: 508M "inplace/bin/ghc-stage1" ... compiler/stage2/build/TcSplice.dyn_o: Max. residency: 176802000 / In use: 445M "inplace/bin/ghc-stage1" ... compiler/stage2/build/TcRnDriver.dyn_o: Max. residency: 172892128 / In use: 458M "inplace/bin/ghc-stage1" ... compiler/stage2/build/HscMain.dyn_o: Max. residency: 172802688 / In use: 470M "inplace/bin/ghc-stage1" ... libraries/ghci/dist- install/build/GHCi/TH/Binary.dyn_o: Max. residency: 169775368 / In use: 455M "inplace/bin/ghc-stage1" ... compiler/stage2/build/GhcMake.dyn_o: Max. residency: 167305408 / In use: 431M "inplace/bin/ghc-stage1" ... compiler/stage2/build/PrimOp.dyn_o: Max. residency: 162532408 / In use: 455M "/usr/local/bin/ghc" ... compiler/stage1/build/HscTypes.o: Max. residency: 160814736 / In use: 421M "inplace/bin/ghc-stage1" ... compiler/stage2/build/HsDecls.dyn_o: Max. residency: 160445552 / In use: 426M "inplace/bin/ghc-stage1" ... libraries/Cabal/Cabal/dist- install/build/Language/Haskell/Extension.dyn_o: Max. residency: 159817712 / In use: 363M "/usr/local/bin/ghc" ... compiler/stage1/build/TcSplice.o: Max. residency: 156738656 / In use: 375M "/usr/local/bin/ghc" ... compiler/stage1/build/TcRnDriver.o: Max. residency: 153910768 / In use: 421M "inplace/bin/ghc-stage1" ... libraries/containers/dist- install/build/Data/Sequence/Internal.dyn_o: Max. residency: 151318224 / In use: 405M "inplace/bin/ghc-stage1" ... compiler/stage2/build/MatchLit.dyn_o: Max. residency: 150681176 / In use: 382M "/usr/local/bin/ghc" ... compiler/stage1/build/ByteCodeGen.o: Max. residency: 144579176 / In use: 389M "/usr/local/bin/ghc" ... compiler/stage1/build/GhcMake.o: Max. residency: 144411312 / In use: 379M "/usr/local/bin/ghc" ... compiler/stage1/build/MatchLit.o: Max. residency: 142596736 / In use: 382M "inplace/bin/ghc-stage1" ... compiler/stage2/build/StgCmmPrim.dyn_o: Max. residency: 141484616 / In use: 340M "inplace/bin/ghc-stage1" ... libraries/ghci/dist- install/build/GHCi/Message.dyn_o: Max. residency: 139224136 / In use: 363M "/usr/local/bin/ghc" ... compiler/stage1/build/StgCmmPrim.o: Max. residency: 137390744 / In use: 346M "inplace/bin/ghc-stage1" ... compiler/stage2/build/TcExpr.dyn_o: Max. residency: 136387128 / In use: 370M "inplace/bin/ghc-stage1" ... compiler/stage2/build/Lexer.dyn_o: Max. residency: 135228120 / In use: 337M "inplace/bin/ghc-stage1" ... compiler/stage2/build/TcTyClsDecls.dyn_o: Max. residency: 134817024 / In use: 369M "/usr/local/bin/ghc" ... compiler/stage1/build/MkIface.o: Max. residency: 134039144 / In use: 339M "inplace/bin/ghc-stage1" ... compiler/stage2/build/RnExpr.dyn_o: Max. residency: 133947600 / In use: 345M "inplace/bin/ghc-stage1" ... compiler/stage2/build/HscTypes.dyn_o: Max. residency: 133938840 / In use: 366M "inplace/bin/ghc-stage1" ... compiler/stage2/build/TcGenDeriv.dyn_o: Max. residency: 133744056 / In use: 320M "/usr/local/bin/ghc" ... compiler/stage1/build/TcExpr.o: Max. residency: 131766896 / In use: 336M "inplace/bin/ghc-stage1" ... compiler/stage2/build/GHC.dyn_o: Max. residency: 130998912 / In use: 277M "inplace/bin/ghc-stage1" ... compiler/stage2/build/ByteCodeGen.dyn_o: Max. residency: 129499464 / In use: 348M "inplace/bin/ghc-stage1" ... compiler/stage2/build/InteractiveEval.dyn_o: Max. residency: 128902944 / In use: 286M }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14797> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 2

Re: [GHC] #7275: Give more detailed information about PINNED data in a heap profile
by GHC 04 Jan '19

04 Jan '19

#7275: Give more detailed information about PINNED data in a heap profile -------------------------------------+------------------------------------- Reporter: edsko | Owner: (none) Type: feature request | Status: new Priority: normal | Milestone: Component: Profiling | Version: 7.6.1 Resolution: | Keywords: newcomer Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: Other | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by mpickering): * keywords: => newcomer Comment: Would be very useful to implement this and there is a clear implementation plan indicated by Ian in comment:2. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/7275#comment:15> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 0

[GHC] #11962: Support induction recursion
by GHC 04 Jan '19

04 Jan '19

#11962: Support induction recursion -------------------------------------+------------------------------------- Reporter: goldfire | Owner: Type: feature | Status: new request | Priority: normal | Milestone: Component: Compiler | Version: 8.1 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- Now that we have `-XTypeInType`, let's take it a step further and allow induction recursion. This feature exists in Agda and Idris. Here is a short example of what can be done in Agda: {{{ mutual -- Codes for types. data U : Set where nat : U pi : (a : U) → (El a → U) → U -- A function that interprets codes as actual types. El : U → Set El nat = ℕ El (pi a b) = (x : El a) → El (b x) }}} Note that the `U` datatype and the `El` function depend on each other. But if you look more closely, the header for `U` does not depend on `El`; only the constructors of `U` depend on `El`. So if we typecheck `U : Set` first, then `El : U → Set`, then the constructors of `U`, then the equations of `El`, we're OK. Translation into Haskell: {{{ import Data.Kind data family Sing (a :: k) -- we still require singletons data U :: Type where Nat :: U Pi :: Sing (a :: U) -> (El a -> U) -> U type family El (u :: U) :: Type where El 'Nat = Int El (Pi a b) = forall (x :: El a). Sing x -> El (b x) }}} This currently errors with {{{ • Type constructor ‘U’ cannot be used here (it is defined and used in the same recursive group) • In the kind ‘U’ }}} It needn't error. (I'm cheating a bit here, because for unrelated reasons, we can't return a `forall` on the right-hand side of a type family. But that's not the issue at hand.) I have some very sketchy notes on how we might do this [wiki:DependentHaskell/Internal#Separatingtypesignaturesfromdefinitions here]. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11962> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 12

[GHC] #16062: Improve -dynamic-too progress messages
by GHC 04 Jan '19

04 Jan '19

#16062: Improve -dynamic-too progress messages -------------------------------------+------------------------------------- Reporter: osa1 | Owner: (none) Type: task | Status: new Priority: lowest | Milestone: Component: Compiler | Version: 8.7 Keywords: newcomer | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- With `-dynamic-too` we run the code generator twice and generate two object files: `.o` and `.o_dyn`. However the progress messages do not reflect that: {{{ $ ghc Main.hs -dynamic-too [1 of 1] Compiling Lib ( Main.hs, Main.o ) }}} I think a better message would be: {{{ [1 of 1] Compiling Lib ( Main.hs, Main.o, Main.dyn_o ) }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/16062> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 1

[GHC] #15877: --flavour=quick followed by --flavour=prof fails
by GHC 04 Jan '19

04 Jan '19

#15877: --flavour=quick followed by --flavour=prof fails -------------------------------------+------------------------------------- Reporter: mpickering | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.3 Component: Build System | Version: 8.6.2 (Hadrian) | Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- Starting from a clean tree I first built successfully with `--flavour=quick`, I then changed to `--flavour=prof` and the build failed as follows: {{{ /root/ghc/_build/stage1/rts/build/libCffi_p.a: copyFile: does not exist (No such file or directory) shakeArgsWith 0.010s 0% Function shake 0.435s 0% Database read 0.670s 0% With database 0.035s 0% Running rules 123.401s 99% ========================= Total 124.551s 100% Error when running Shake build system: at src/Rules.hs:(32,19)-(45,17): at src/Rules.hs:45:5-17: * Depends on: _build/stage1/lib/package.conf.d/rts-1.0.conf * Raised the exception: ExitFailure 1 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/15877> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 2

[GHC] #16124: prof_hc_hb way failures on CI
by GHC 04 Jan '19

04 Jan '19

#16124: prof_hc_hb way failures on CI -------------------------------------+------------------------------------- Reporter: osa1 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Profiling | Version: 8.7 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: #15508 Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- Reported by Gabor in https://gitlab.haskell.org/ghc/ghc/merge_requests/67#note_1845 It seems like we have non-deterministic `prof_hc_hb` problems that are caught by the CI sometimes: - https://gitlab.haskell.org/adamse/ghc/-/jobs/8518 - https://gitlab.haskell.org/ghc/ghc/-/jobs/8489 Another ticket for profiling build problems: #15508. I fixed a few bugs in that ticket, but there are at least two more bugs to fix. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/16124> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 0

[GHC] #15155: How untagged pointers sneak into banged fields
by GHC 03 Jan '19

03 Jan '19

#15155: How untagged pointers sneak into banged fields -------------------------------------+------------------------------------- Reporter: heisenbug | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.4.2 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: 14677 | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- (''N.B.'' I am writing this up from memory, and cannot verify it just now, maybe someone can lend a hand, otherwise I'll do it ASAP!) Here is a way how untagged pointers to strict data can be created in banged (strict) constructor fields. This reproduction recipe **depends on the patch from #14677 applied**. We have 3 modules `A`, `B` and `C`: {{{#!hs module A where data A = X | Y | Z a = Z }}} {{{#!hs module B where import A newtype B = B A b = B a }}} {{{#!hs {-# language MagicHash #-} module C where import A import B import GHC.Exts data C = C !B c = C b main = do print (I# (reallyUnsafePtrEquality# a (coerce b))) -- prints 0, b is softlink print (I# (dataToTag# c)) -- prints 0: not entered yet print (case c of C b' -> I# (dataToTag# b')) -- prints 0? print (case c of C (B a') -> I# (dataToTag# a')) -- prints 3 }}} ------------------- == Why this happens `B.b` is a newtype to `A.a` so one would expect that both alias the same memory location (a ''hardlink'' in filesystem parlance). But currently reexports are implemented with a special type of closure `IND_STATIC` (a ''softlink'') which needs to be entered to obtain the actual (tagged pointer). The `IND_STATIC` closure's pointer is never tagged (otherwise it would never be entered, instead interpreted as a honest-to-goodness `A.A`, which causes the symptoms seen in #14677). With #14677 applied to GHC, the unfolding of `B.b` is correctly read when compiling `C` (with `-O1` and better) and thus the compiler knows that it should be a tagged pointer value. Thus the construction of `C.c` shortcuts the entering of `B.b` when filling the strict field, and (because `B.b` being a softlink, thus untagged) the field ends up carrying a 0 tag. ------------------------- == How can this be fixed? I see two possibilities one conservative and one invasive. === Conservative When seeing a coercion unfolding of a tagged value being used to initialise a strict field, do not skip the evaluatedness check, but cater for the possibility of an `IND_STATIC` closure. Check the closure type, and if confirmed, steal the pointee and use that. === Invasive Get rid of the `IND_STATIC` closures altogether. For ''intra-module'' softlinks we can have proper hardlinks (assembler `.equiv` directives, or LLVM `alias`es). ''Inter-module'' softlinks can also be eliminated by linker scripts. This would however cause more build artifacts, so I don't know how hairy it would turn out. OTOH, it would reduce binary size by eliminating indirection closures and potential dereferencing code. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/15155> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 18

[GHC] #15617: Unboxed tuples/sum error message on `a = show 5` in expression evaluation and interactive modes
by GHC 03 Jan '19

03 Jan '19

#15617: Unboxed tuples/sum error message on `a = show 5` in expression evaluation and interactive modes -------------------------------------+------------------------------------- Reporter: ChaiTRex | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.6.1-beta1 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: Poor/confusing Unknown/Multiple | error message Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- With GHC 8.4.3 (`ghc` below on both Ubuntu 16.04.5 and Ubuntu 18.04.1) and recent GHC (`~/dev/ghc/inplace/bin/ghc-stage2` below on Ubuntu 16.04.5; not `HEAD` but close at `ff29fc84c03c800cfa04c2a00eb8edf6fa5f4183`), I get errors for `a = show 5`. I run the following commands, showing that `show 5` is usually fine: {{{ $ ghc -fobject-code -O2 -e 'show 5' "5" $ ghc -fobject-code -O2 -e 'let a = show 5 in a' "5" }}} But not with `a = show 5`: {{{ $ ghc -fobject-code -O2 -e 'a = show 5' <interactive>: Error: bytecode compiler can't handle unboxed tuples and sums. Possibly due to foreign import/export decls in source. Workaround: use -fobject-code, or compile this module to .o separately. }}} Running with `ghci` gives the same error: {{{ $ ghci -fobject-code -O2 GHCi, version 8.4.3: http://www.haskell.org/ghc/ :? for help Prelude> a = show 5 Error: bytecode compiler can't handle unboxed tuples and sums. Possibly due to foreign import/export decls in source. Workaround: use -fobject-code, or compile this module to .o separately. Prelude> Leaving GHCi. }}} Both errors stop when optimization is turned off: {{{ $ ghc -fobject-code -O0 -e 'a = show 5' $ ghci -fobject-code -O0 GHCi, version 8.4.3: http://www.haskell.org/ghc/ :? for help Prelude> a = show 5 Prelude> Leaving GHCi. }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/15617> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 17

[GHC] #16119: Hide the gnarly levity polymorphism stuff in the signatures of `undefined`, `throw`, etc
by GHC 02 Jan '19

02 Jan '19

#16119: Hide the gnarly levity polymorphism stuff in the signatures of `undefined`, `throw`, etc -------------------------------------+------------------------------------- Reporter: jberryman | Owner: (none) Type: feature | Status: new request | Priority: normal | Milestone: Component: | Version: 8.6.3 libraries/base | Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- In base-4.12, `throw` goes from {{{#!hs throw :: Exception e => e -> a }}} to {{{#!hs throw :: forall (r :: RuntimeRep). forall (a :: TYPE r). forall e. Exception e => e -> a }}} http://hackage.haskell.org/package/base-4.12.0.0/docs/Control-Exception- Base.html#v:throw Similar story for `undefined`. Even as a fairly experienced haskell programmer I find the signature difficult to parse, leaving aside the meaning of `RuntimeRep` and `TYPE`. The amount of research an expert in Haskell 98 would have to do to thoroughly understand that signature is unreasonable (the various meanings of `forall`, `DataKinds`, as well as the levity polymorphism stuff). I don't have a proposed solution, but a constraint synonym or something that hid `RuntimeRep` and `TYPE`, and which could be documented with "This means a type with any runtime representation; everything is fine" would be okay. Relevant: https://ghc.haskell.org/trac/ghc/ticket/12030 https://ghc.haskell.org/trac/ghc/ticket/11549 -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/16119> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

1 2