- ghc-devs - Haskell.org

A small but useful tool for performance characterisation
by Ben Gamari 04 Jan '20

04 Jan '20

Hi everyone, I have recently been doing a fair amount of performance characterisation and have long wanted a convenient means of collecting GHC runtime statistics for later analysis. For this I quickly developed a small wrapper utility [1]. To see what it does, let's consider an example. Say we made a change to GHC which we believe might affect the runtime performance of Program.hs. We could quickly check this by running, $ ghc-before/_build/stage1/bin/ghc -O Program.hs $ ghc_perf.py -o before.json ./Program $ ghc-before/_build/stage1/bin/ghc -O Program.hs $ ghc_perf.py -o after.json ./Program This will produce two files, before.json and after.json, which contain the various runtime statistics emitted by +RTS -s --machine-readable. These files are in the same format as is used by my nofib branch [2] and therefore can be compared using `nofib-compare` from that branch. In addition to being able to collect runtime metrics, ghc_perf is also able to collect performance counters (on Linux only) using perf. For instance, $ ghc_perf.py -o program.json \ -e instructions,cycles,cache-misses ./Program will produce program.json containing not only RTS statistics but also event counts from the perf instructions, cycles, and cache-misses events. Alternatively, passing simply `ghc_perf.py --perf` enables a reasonable default set of events (namely instructions, cycles, cache-misses, branches, and branch-misses). Finally, ghc_perf can also handle repeated runs. For instance, $ ghc_perf.py -o program.json -r 5 --summarize \ -e instructions,cycles,cache-misses ./Program will run Program 5 times, emit all of the collected samples to program.json, and produce a (very basic) statistical summary of what it collected on stdout. Note that there are a few possible TODOs that I've been considering: * I chose JSON as the output format to accomodate structured data (e.g. capture experimental parameters in a structured way). However, in practice this choice has lead to significantly more inconvenience than I would like, especially given that so far I've only used the format to capture basic key/value pairs. Perhaps reverting to CSV would be preferable. * It might be nice to also add support for cachegrind. Anyways, I hope that others find this as useful as I have. Cheers, - Ben [1] https://gitlab.haskell.org/bgamari/ghc-utils/blob/master/ghc_perf.py [2] https://gitlab.haskell.org/ghc/nofib/merge_requests/24

2 2

[ANNOUNCE] GHC 8.8.2-rc1 is now available
by Ben Gamari 02 Jan '20

02 Jan '20

Hello everyone, The GHC team is proud to announce the first release candidate of GHC 8.8.2. The source distribution, binary distributions, and documentation are available at https://downloads.haskell.org/~ghc/8.8.2-rc1 Release notes are also available [1]. This release fixes a handful of issues affecting 8.8.1: - A bug (#17088) in the compacting garbage collector resulting in segmentations faults under specific circumstances. Note that this may affect user programs even if they did not explicitly request the compacting GC (using the -c RTS flag) since GHC may fallback to compacting collection during times of high memory pressure. - A code generation bug (#17334) resulting in GHC panics has been fixed. - A bug in the `process` library causing builds using `hsc2hs` to fail non-deterministically on Windows has been fixed (Trac #17480) - A typechecker bug (#12088) resulting in programs being unexpectedly rejected has been fixed. - A bug in the implementation of compact normal forms resulting in segmentation faults in some uses (#17044) has been fixed. - A bug causing GHC to incorrectly complain about incompatible LLVM versions when using LLVM 7.0.1 has been fixed (#16990). As always, if anything looks amiss do let us know. Happy compiling! Cheers, - Ben [1] https://downloads.haskell.org/ghc/8.8.2-rc1/docs/html/users_guide/8.8.2-not…

4 3

[ANNOUNCE] Glasgow Haskell Compiler 8.10.1-alpha2 released
by Ben Gamari 02 Jan '20

02 Jan '20

Hello all, The GHC team is happy to announce the availability of the first alpha release in the GHC 8.10 series. Source and binary distributions are available at the usual place: https://downloads.haskell.org/ghc/8.10.1-alpha2/ GHC 8.10.1 will bring a number of new features including: * The new UnliftedNewtypes extension allowing newtypes around unlifted types. * The new StandaloneKindSignatures extension allows users to give top-level kind signatures to type, type family, and class declarations. * A new warning, -Wderiving-defaults, to draw attention to ambiguous deriving clauses * A number of improvements in code generation, including changes * A new GHCi command, :instances, for listing the class instances available for a type. * An upgraded Windows toolchain lifting the MAX_PATH limitation * A new, low-latency garbage collector. * Improved support profiling, including support for sending profiler samples to the eventlog, allowing correlation between the profile and other program events This release is the second alpha of the 8.10 pre-release cycle. The release will be a release candidate in roughly three weeks. If all goes well the final release will be cut roughly two weeks after the candidate, in mid-January. This being an alpha release, there are a few issues that are still outstanding: * The new Alpine Linux binary distribution is not present due to an apparent correctness issue [1]; any help Alpine users can offer here would be greatly appreciated. * We have yet to sort out compliance with Apple's notarization requirements [2] which will be likely be necessary for users of macOS Catalina. However, until this has been sorted out Catalina users can exempt the current macOS binary distribution from the notarization requirement themselves by running `xattr -cr .` on the unpacked tree before running `make install`. * There is one remaining non-regression correctness issue which we plan to fix for the final 8.10.1 but that is not fixed in this release. Please do test this release and let us know if you encounter any other issues. Cheers, - Ben [1] https://gitlab.haskell.org/ghc/ghc/issues/17508 [2] https://gitlab.haskell.org/ghc/ghc/issues/17418

3 2

Implementation of Type level warnings
by Rowan Goemans 30 Dec '19

30 Dec '19

Hello everyone, A couple of days ago I tried to implement type level warnings [1]. I have got something now that sort of works but issues the same warning multiple times and not on the correct location. It's currently a WIP merge request [2]. I would like some help/pointers to continue the implementation The approach I have taken is to issue the warning once TypeWarning is rewritten. However,this has severe problems since now a warning is issued multiple times and at different locations due to therewritinghappening multiple times. I don't know what the best location is to do this check and issue the warning. Prior to constraint solving isn't a good place since the TypeWarning could be nested deeply in some constraint. Afterwards also isn't doable since then the TypeWarning constraint is gone. An alternative could be to not implement any instance in for TypeWarning similar to TypeError and special case it somehow. Any help is appreciated! [1] https://gitlab.haskell.org/ghc/ghc/issues/17027 [2] https://gitlab.haskell.org/ghc/ghc/merge_requests/2361 Best regards, RowanGoemans

2 1

HEAD bootstrap error macOS
by Gabor Greif 25 Dec '19

25 Dec '19

Recently I started getting following error: ``` [nix-shell:~/ghc]$ "inplace/bin/ghc-stage1" -optc-Wall -optc-Wall -optc-Wextra -optc-Wstrict-prototypes -optc-Wmissing-prototypes -optc-Wmissing-declarations -optc-Winline -optc-Wpointer-arith -optc-Wmissing-noreturn -optc-Wnested-externs -optc-Wredundant-decls -optc-Wno-aggregate-return -optc-Iincludes -optc-Iincludes/dist -optc-Iincludes/dist-derivedconstants/header -optc-Iincludes/dist-ghcconstants/header -optc-Iincludes/dist-install/build -optc-Irts -optc-Irts/dist/build -optc-DCOMPILING_RTS -optc-DFS_NAMESPACE=rts -optc-fno-strict-aliasing -optc-fno-common -optc-DDTRACE -optc-Irts/dist/build/./autogen -optc-Wno-unknown-pragmas -optc-O2 -optc-fomit-frame-pointer -optc-g -optc-DRtsWay=\"rts_thr\" -static -optc-DTHREADED_RTS -O0 -H64m -Wall -Iincludes -Iincludes/dist -Iincludes/dist-derivedconstants/header -Iincludes/dist-ghcconstants/header -Iincludes/dist-install/build -Irts -Irts/dist/build -DCOMPILING_RTS -DFS_NAMESPACE=rts -this-unit-id rts -dcmm-lint -DDTRACE -i -irts -irts/dist/build -Irts/dist/build -irts/dist/build/./autogen -Irts/dist/build/./autogen -O2 -Wcpp-undef -Wnoncanonical-monad-instances -c rts/sm/Scav_thr.c -o rts/dist/build/sm/Scav_thr.thr_o -v Glasgow Haskell Compiler, Version 8.11.0.20191222, stage 1 booted by GHC version 8.6.5 *** initializing package database: Using binary package database: /Users/ggreif/ghc/inplace/lib/package.conf.d/package.cache package flags [] loading package database /Users/ggreif/ghc/inplace/lib/package.conf.d wired-in package ghc-prim mapped to ghc-prim-0.6.1 wired-in package integer-wired-in mapped to integer-gmp-1.0.2.0 wired-in package base mapped to base-4.14.0.0 wired-in package rts mapped to rts wired-in package template-haskell mapped to template-haskell-2.16.0.0 wired-in package ghc mapped to ghc-8.11.0.20191222 !!! initializing package database: finished in 8.51 milliseconds, allocated 8.770 megabytes Created temporary directory: /var/folders/39/lr50t0q96tx5qp0mzht34r240000gn/T/ghc55616_0 *** systool:cc: *** C Compiler: /nix/store/mq8dqgqf2qqkp77pzf3jwc05px8rkbny-clang-wrapper-9.0.0/bin/cc -x c rts/sm/Scav_thr.c -o /var/folders/39/lr50t0q96tx5qp0mzht34r240000gn/T/ghc55616_0/ghc_1.s -fno-common -U__PIC__ -D__PIC__ -Wimplicit -S -O2 -include /Users/ggreif/ghc/includes/ghcversion.h -Iincludes -Iincludes/dist -Iincludes/dist-derivedconstants/header -Iincludes/dist-ghcconstants/header -Iincludes/dist-install/build -Irts -Irts/dist/build -Irts/dist/build -Irts/dist/build/./autogen -I/Users/ggreif/ghc/libraries/base/include -I/Users/ggreif/ghc/libraries/base/dist-install/build/include -I/Users/ggreif/ghc/libraries/base/dist-install/build/dist-install/build/include -I/Users/ggreif/ghc/libraries/integer-gmp/include -I/Users/ggreif/ghc/libraries/integer-gmp/dist-install/build/include -I/Users/ggreif/ghc/libraries/integer-gmp/dist-install/build/dist-install/build/include -I/Users/ggreif/ghc/rts/dist/build -I/Users/ggreif/ghc/includes -I/Users/ggreif/ghc/includes/dist-derivedconstants/header -I/Users/ggreif/ghc/includes/dist-install/build -Xpreprocessor -DCOMPILING_RTS -Xpreprocessor '-DFS_NAMESPACE=rts' -Xpreprocessor -DDTRACE -Wall -Wall -Wextra -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Winline -Wpointer-arith -Wmissing-noreturn -Wnested-externs -Wredundant-decls -Wno-aggregate-return -Iincludes -Iincludes/dist -Iincludes/dist-derivedconstants/header -Iincludes/dist-ghcconstants/header -Iincludes/dist-install/build -Irts -Irts/dist/build -DCOMPILING_RTS '-DFS_NAMESPACE=rts' -fno-strict-aliasing -fno-common -DDTRACE -Irts/dist/build/./autogen -Wno-unknown-pragmas -O2 -fomit-frame-pointer -g '-DRtsWay="rts_thr"' -DTHREADED_RTS In file included from rts/sm/Scav_thr.c:3:0: error: In file included from rts/sm/Scav.c:51:0: error: In file included from rts/sm/GCUtils.h:18:0: error: rts/sm/GCTDecl.h:113:1: error: error: register '%r13' unsuitable for global register variables on this target | 113 | GCT_REG_DECL(gc_thread*, gct, REG_Base); | ^ GCT_REG_DECL(gc_thread*, gct, REG_Base); ^ rts/sm/GCTDecl.h:37:56: error: note: expanded from macro 'GCT_REG_DECL' | 37 | #define GCT_REG_DECL(type,name,reg) register type name REG(reg); | ^ #define GCT_REG_DECL(type,name,reg) register type name REG(reg); ^ includes/stg/MachRegs.h:160:24: error: note: expanded from macro 'REG' | 160 | #define REG(x) __asm__("%" #x) | ^ #define REG(x) __asm__("%" #x) ^ 1 error generated. *** Deleting temp files: Deleting: /var/folders/39/lr50t0q96tx5qp0mzht34r240000gn/T/ghc55616_0/ghc_1.s /var/folders/39/lr50t0q96tx5qp0mzht34r240000gn/T/ghc55616_0/ghc_2.rsp Warning: deleting non-existent /var/folders/39/lr50t0q96tx5qp0mzht34r240000gn/T/ghc55616_0/ghc_1.s *** Deleting temp dirs: Deleting: /var/folders/39/lr50t0q96tx5qp0mzht34r240000gn/T/ghc55616_0 `cc' failed in phase `C Compiler'. (Exit code: 1) ``` My $CC is clang-9 (but I also tried others). Has anybody seen something similar? Cheers nd thanks, Gabor

2 5

Perf metrics
by Simon Peyton Jones 20 Dec '19

20 Dec '19

When I run `sh validate -legacy`, towards the end I see stuff like this Performance Metrics (test environment: local): Conversions(normal) runtime/bytes allocated 107696.000 DeriveNull(normal) runtime/bytes allocated 112050960.000 InlineArrayAlloc(normal) runtime/bytes allocated 1600041088.000 InlineByteArrayAlloc(normal) runtime/bytes allocated 1440041088.000 InlineCloneArrayAlloc(normal) runtime/bytes allocated 1600041248.000 ManyAlternatives(normal) compile_time/bytes allocated 840985728.000 ManyConstructors(normal) compile_time/bytes allocated 4540766560.000 MethSharing(normal) runtime/peak_megabytes_allocated 2.000 MethSharing(normal) runtime/bytes allocated 480098136.000 MultiLayerModules(normal) compile_time/bytes allocated 5856970504.000 ... It is intermingled with other apparently unrelated output. What should I conclude form this? Is it good or bad? By what amount have these figures changed, and relative to what? How can I run a single perf test? Thanks Simon

2 3

Re: haskell/bytestring stalled?
by Simon Jakobi 19 Dec '19

19 Dec '19

Hi Herbert, I apologize for the overly harsh and accusatory tone in my last email. After looking through the existing PRs and issues, I was admittedly somewhat disappointed over the current situation with bytestring issues and PRs, but I shouldn't have expressed my feelings in this way. Also, thanks a lot for responding so quickly today to my many comments on issues and PRs. To be clear, I'm not interested in putting blame on you or Duncan or anyone else. I do however strongly believe that bytestring needs a more responsive maintainer: 1. Contributors IMO deserve to get a response within weeks or ideally days. 2. bytestring users (and that transitively includes all GHC users) could benefit massively if some of the current PRs were finally merged. For example in [#175], Sylvain Henry reports improvements of **11%** for some compilation tasks when he used the PR to build GHC. And I admittedly don't want to ask you, Herbert, to lead the effort of clearing up the maintenance backlog in bytestring. Of course, _any_ Haskell project you're involved in benefits from your massive experience! However, I feel that since your involvement is so crucial in so many other essential projects in the Haskell ecosystem, if you would shift your focus towards bytestring, these other projects would suffer! I also believe that projects like cabal or Hackage need your domain knowledge more dearly than bytestring does. So to ask you to allocate more time to bytestring, would feel a bit like a zero-sum game, or possibly a negative-sum game. A second, more personal reason, why I would prefer to see a different maintainer for bytestring, is your preference for coordinating things in private communication. The many PRs and issues in the bytestring project, and this email thread demonstrate that there are a lot of people who would like to be involved in improving bytestring. Private coordination would exclude most of them, and make it more difficult to make use of all the help that this project currently needs. I understand that your preference for private communication is related to your experiences with previous discussions over Haskell projects, and I am sorry that it has come so far, but I believe that a more open and transparent style of communication is more healthy. So, while I'm very grateful for your work on bytestring, and while I hope that you stay involved in the project, I hope that that someone else can take over official maintainer duties. Now I am not sure who could take on that job. Clearly we need someone whom Duncan trusts. That's why I wonder whether someone involved in the performance work on GHC could help. Ultimately I don't think that their time involvement would need to be very high – good judgment and responsiveness seem more important to me. I'm CC'ing the ghc-devs list, hoping that someone with the right experience and availability would come forward. Thanks, Simon [#175] https://github.com/haskell/bytestring/pull/175#issuecomment-567233984

2 2

Invitation to GHC Week
by Matthew Pickering 19 Dec '19

19 Dec '19

Dear all, I am happy to announce GHC Week, a developers meeting focusing on GHC, organized by myself and AndreasK. We are renting a big holiday cottage in rural Belgium during March so that participants can hack on GHC uninterrupted for a whole week. The place has a large number of beds and a nice open living area for collaboration and relaxation. Dates: 13th - 20th March 2020 Location: Annevoie, Belgium If you are interested in attending GHC Week then please consult the linked webpage which contains information about how to attend. There are only a limited number of spaces and we already have a number of confirmed attendees. Hopefully no one will be disappointed but if all 700 subscribers to ghc-devs want to attend there will not be enough space! We would particularly like to encourage undergraduate students who are interested in working on GHC to attend GHC Week. Thanks to our sponsors, Haskell.org and Well-Typed we are in a position to not only make the event itself very affordable, but also to cover travel and accommodation costs for those in need. This is an exceptional opportunity to learn about GHC from experienced developers. So if you can, make sure you don’t miss it! For more details please consult the following page: https://andreaspk.github.io/ghc-week Cheers, Matt & Andreas

1 1

Typo in closure name in rts/Prelude.h
by Csaba Hruska 15 Dec '19

15 Dec '19

Hello, Could you confirm if *base_GHCziWeak_rrts/win32/libHSbase.def_closure* <https://gitlab.haskell.org/ghc/ghc/blob/master/rts/Prelude.h#L30> is a typo in rts/Prelude.h? I guess it is a typo because *base_GHCziWeak_runFinalizerBatch_closure* does exists in base/GHC/Weak.hs. <https://gitlab.haskell.org/ghc/ghc/blob/master/libraries/base/GHC/Weak.hs#L…> The following files have the wrong name: - rts/package.conf.in - rts/rts.cabal.in - rts/Prelude.h - rts/win32/libHSbase.def Regards, Csaba

2 3

[ANNOUNCE] GHC 8.8.1-alpha2 is now available
by Ben Gamari 12 Dec '19

12 Dec '19

Hello everyone, The GHC team is pleased to announce the second and likely last alpha release of GHC 8.8.1. The source distribution, binary distributions, and documentation are available at https://downloads.haskell.org/~ghc/8.8.1-alpha2 A draft of the release notes is also available [1]. This release is the culmination of over 3000 commits by over one hundred contributors and has several new features and numerous bug fixes relative to GHC 8.6: * Profiling now works correctly on 64-bit Windows (although still may be problematic on 32-bit Windows due to platform limitations; see #15934) * A new code layout algorithm for amd64's native code generator * The introduction of a late lambda-lifting pass which may reduce allocations significantly for some programs. * Further work on Trees That Grow, enabling improved code re-use of the Haskell AST in tooling * More locations where users can write `forall` (GHC Proposal #0007) * Further work on the Hadrian build system This release brings a number of fixes since alpha 1: * A number of linker fixes (#16779, #16784) * The process, binary, Cabal, time, terminfo libraries have all been bumped to their final release versions * A regression rendering TemplateHaskell unusable in cross-compiled configurations has been fixed (#16331) * A regression causing compiler panics on compilation of some programs has been fixed (#16449) * -Wmissing-home-modules now handles hs-boot files correctly (#16551) * A regression causing some programs to fail at runtime has been fixed (#16066) Due to on-going work on our release and testing infrastructure this cycle is proceeding at a pace significantly slower than expected. However, we anticipate that this investment will allow us to release a more reliable, easier-to-install compiler on the planned six-month release cadence in the future. As always, if anything looks amiss do let us know. Happy compiling! Cheers, - Ben [1] https://downloads.haskell.org/ghc/8.8.1-alpha2/docs/html/users_guide/8.8.1-…

8 18