January 2021 - ghc-devs

The future of Phabricator
by Ben Gamari 18 Sep '21

18 Sep '21

TL;DR. For several reasons I think we should consider alternatives to Phabricator. My view is that GitLab seems like the best option. Hello everyone, Over the past year I have been growing increasingly weary of our continued dependence on Phabricator. Without a doubt, its code review interface is the best I have used. However, for a myriad of reasons I am recently questioning whether it is still the best tool for GHC's needs. # The problem There are a number of reasons why I am currently uncertain about Phabricator. For one, at this point we have no options for support in the event that something goes wrong as the company responsible for Phabricator, Phacility, has closed their support channels to non-paying customers. Furthermore, in the past year or two Phacility has been placing their development resources in the parts their customers pay them for, which appear to be much different that the parts that we actively use. For this reason, some parts that we rely on seem oddly half-finished. This concern was recently underscored by some rather unfortunate misfeatures in Harbormaster which resulted in broken CI after the Hadrian merge and now apparent bugs which have made it difficult to migrate to the CircleCI integration we previously prepared. Perhaps most importantly, in our recent development priorities survey our use of Phabricator was the most common complaint by a fair margin, both in case of respondents who have contributed patches and those who have not. On the whole, past contributors and potential future contributors alike have strongly indicated that they want a more Git-like experience. Of course, this wasn't terribly surprising; this is just the most recent case where contributors have made this preference known. Frankly, in a sense it is hard to blame them. The fact that users need to install a PHP tool, Arcanist, to contribute anything but documentation patches has always seemed like unnecessary friction to me and I would be quite happy to be rid of it. Indeed we have had a quite healthy number of GitHub documentation patches since we started accepting them. This makes me thing that there may indeed be potential contributoes that we are leaving on the table. # What to do With Rackspace support ending at the end of year, now may be a good time to consider whether we really want to continue on this road. Phabricator is great at code review but I am less and less certain that it is worth the maintenance uncertainty and potential lost contributors that it costs. Moreover, good alternatives seem closer at-hand than they were when we deployed Phabricator. ## Move to GitHub When people complain about our infrastructure, they often use GitHub as the example of what they would like to see. However, realistically I have a hard time seeing GitHub as a viable option. Its feature set is simply insufficient enough to handle the needs of a larger project like GHC without significant external tooling (as seen in the case of Rust-lang). The concrete reasons have been well-documented in previous discussions but, to summarize, * its review functionality is extremely painful to use with larger patches * rebased patches lead to extreme confusion and lost review comments * it lacks support for pre-receive hooks, which serve as a last line of defense to prevent unintended submodule changes * its inability to deal with external tickets is problematic * there is essentially no possibility that we could eventually migrate GHC's tickets to GitHub's ticket tracker without considerable data loss (much less manage the ticket load that would result), meaning that we would forever be stuck with maintaining Trac. * on a personal note, its search functionality has often left me empty-handed On the whole, these issues seem pretty hard to surmount. ## Move to GitLab In using GitLab for another project over the past months I have been positively surprised by its quality. It handles rebased merge requests far better than GitHub, has functional search, and quite a usable review interface. Furthermore, upstream has been extremely responsive to suggestions for improvement [1]. Even out-of-the-box it seems to be flexible enough to accommodate our needs, supporting integration with external issue trackers, having reasonable release management features, and support for code owners to automate review triaging (replacing much of the functionality of Phabricator's Herald). Finally, other FOSS projects' [3] recent migrations from Phabrictor to GitLab have shown that GitLab-the-company is quite willing to offer help when needed. I took some time this weekend to try setting up a quick GHC instance [2] to play around with. Even after just a few hours of playing around I think the result is surprisingly usable. Out of curiosity I also played around with importing some tickets from Trac (building on Matt Pickering's Trac-to-Maniphest migration tool). With relatively little effort I was even able to get nearly all of our tickets (as of 9 months ago) imported while preserving ticket numbers (although there are naturally a few wrinkles that would need to be worked out). Naturally, I think we should treat the question of ticket tracker migration as an orthogonal one to code review, but it is good to know that this is possible. ## Continue with Phabricator Continuing with Phabricator is of course an option. Its review functionality is great and it has served us reasonably well. However, compared to GitLab and even GitHub of today its features seem less distinguished than they once did. Moreover, the prospect of having to maintain a largely stagnant product with no support strikes me as a slightly dangerous game to play. Working around the issues we have recently encountered has already cost a non-negligible amount of time. # The bottom line If it wasn't clear already, I think that we should strongly consider a move to GitLab. At this point it seems clear that it isn't going to vanish, has a steady pace of development, is featureful, and available. However, these are just my thoughts. What do you think? Cheers, - Ben [1] 11.4 will ship with a file tree view in the code review interface, which I reported (https://gitlab.com/gitlab-org/gitlab-ce/issues/46474) as being is one of the Phabricator features I missed the most during review [2] https://gitlab.home.smart-cactus.org/ghc/ghc/issues/14641 [3] The GNOME and freedesktop.org projects have recently migrated, the former from a hodge-podge of self-hosted services and the latter from Phabricator

17 40

New implementation for `ImpredicativeTypes`
by Alejandro Serrano Mena 08 Sep '21

08 Sep '21

Hi all, As I mentioned some time ago, we have been busy working on a new implementation of `ImpredicativeTypes` for GHC. I am very thankful to everybody who back then sent us examples of impredicativity which would be nice to support, as far as we know this branch supports all of them! :) If you want to try it, at https://gitlab.haskell.org/trupill/ghc/commit/a3f95a0fe0f647702fd7225fa719a… you can find the result of the pipeline, which includes builds for several platforms (click on the "Artifacts" button, the one which looks like a cloud, to get them). The code is being developed at https://gitlab.haskell.org/trupill/ghc. Any code should run *unchanged* except for some eta-expansion required for some specific usage patterns of higher-rank types. Please don't hesitate to ask any questions or clarifications about it. A merge request for tracking this can be found at https://gitlab.haskell.org/ghc/ghc/merge_requests/1659 Kind regards, Alejandro

8 23

Potential improvement in compacting GC
by Ömer Sinan Ağacan 02 Sep '21

02 Sep '21

Hello, I recently implemented the algorithm used by GHC's compacting GC in another project. The algorithm (after marking) makes two passes over the heap /generation. In GHC, these passes are implemented in [1] and in the next function. In my implementation I tried 3 ways of implementing these passes, one of which is the same as GHC's code, and benchmarked each version. I found that the fastest implementation is not what's used in GHC, but it could easily be used. I should mention that my code targets Wasm, and I benchmarked Wasm instructions executed. Not CPU cycles, CPU instructions, or anything like that. It's very likely that the results will be different when benchmarking code running on actual hardware. Secondly, in my case the heap is mostly dead (residency is low). In GHC, compaction for the oldest generation is enabled when residency passes a threshold, so the assumption is the heap is mostly live. I'm guessing this should also make some difference. Anyway, the first implementation I tried was similar to GHC's scan, but I did scan += object_size(scan); instead of bumping scan by one, as GHC does in [2]. This was the slowest version. Second implementation did the same as GHC (bumped scan by one). This was faster, but still slower than the next version. What I found to be the best is scanning the bitmap, not the heap. The bitmap iterator reads one word at a time. In each iteration it checks if the bitmap word is 0. In GHC, in the best case this can skip 16 words on heap on 32-bit systems, and 32 words on 64-bit. Reminder: we need two bits per object in the bitmap, see [3]. (this is not the case in my implementation so the payoff is better) When the bitmap word is not 0 I use "count trailing zeros" (or "count leading zeros" depending on the bitmap implementation) to get the number of words to skip. This is a single instruction on Wasm and x86 (TZCNT or LZCNT, available via __builtin_ctzl and __builtin_clzl in gcc). So instead of skipping one word at a time, this can potentially skip 16 words (or 32 on 64-bit architectures). When that's not possible, it can still skip multiple words by using ctz/clz. Ömer [1]: https://github.com/ghc/ghc/blob/bd877edd9499a351db947cd51ed583872b2facdf/rt… [2]: https://github.com/ghc/ghc/blob/bd877edd9499a351db947cd51ed583872b2facdf/rt… [3]: https://github.com/ghc/ghc/blob/bd877edd9499a351db947cd51ed583872b2facdf/rt…

5 6

Inspecting function arguments in GHCi
by Andrew Kvapil 02 Feb '21

02 Feb '21

Hello, I'm interested in inspecting the strictness of functions at runtime and the depth of thunks "in the wild." For this reason I'm modifying GHC 8.10.2, essentially to add additional information to breakpoints. I'd like to reuse the logic behind GHCi's :print command (pprintClosureCommand, obtainTermFromId, ...) for which I suppose I need Id's. Those however don't exist for destructuring patterns, such as those in the following equations: last [x] = x last (_:xs) = last xs So I'm wondering where would be a good place in the pipeline to transform patterns like these into at-patterns, to give them Id's. However, the breakpoint logic only looks at the free variables of the right-hand sides and not transitively, which means that e.g. in the following example neither ':print arg1' nor ':print as' works when the interpreter hits a breakpoint in the top level expression on the RHS: qsort arg1@(a:as) = qsort left ++ [a] ++ qsort right where (left, right) = (filter (<=a) as, filter (>a) as) Thus I'd also like to know how to extend the free var logic for Tickish that eventually leads to CgBreakInfo and :print's ability to inspect these bindings at runtime. My goal would be to determine to what extent was a thunk evaluated during function application. Any advice would be greatly appreciated! Regards, Andrew Kvapil

4 5

Storage layout of integral types
by Stefan Schulze Frielinghaus 01 Feb '21

01 Feb '21

Hi all, I'm wondering what the supposed storage layout of integral types is. In particular for integral types with size less than the size of a word. For example, on a 64bit machine is a 32bit integer supposed to be written as a whole word and therefore as 64 bits or just as 32bits in the payload of a closure? I'm asking because since commit be5d74ca I see differently aligned integers in the payload of a closure on a 64bit big-endian machine. For example, in the following code an Int32 object is created which contains the actual integer in the high part of the payload (the snippet comes from the add operator GHC.Int.$fNumInt32_$c+_entry): Hp = Hp + 16; ... I64[Hp - 8] = GHC.Int.I32#_con_info; I32[Hp] = _scz7::I32; whereas e.g. in function rts_getInt32 the opposite is assumed and the actual integer is expected in the low part of the payload: HsInt32 rts_getInt32 (HaskellObj p) { // See comment above: // ASSERT(p->header.info == I32zh_con_info || // p->header.info == I32zh_static_info); return (HsInt32)(HsInt)(UNTAG_CLOSURE(p)->payload[0]); } The same seems to be the case for the interpreter and foreign calls (case bci_CCALL) where integral arguments are passed in the low part of a whole word. Currently, my intuition is that the payload of a closure for an integral type with size smaller than WordSize is written as a whole word where the subword is aligned according to the machines endianness. Can someone confirm this? If that is indeed true, then rts_getInt32 seems to be correct but not the former. Otherwise the converse seems to be the case. Cheers, Stefan

1 1

-ddump-json
by Richard Eisenberg 29 Jan '21

29 Jan '21

Hi devs, In my work with Alfredo at revising our error message infrastructure, we ran across some code that renders error messages as JSON. Given that our data structures are changing, it seems natural to change the JSON output, too, but it's unclear whether that's wise. The manual currently lists -ddump-json in the chapter on "Debugging the compiler", suggesting that a change is fine, but I'm not yet convinced. So: - Is there someone in charge of -ddump-json? Matthew Pickering authored -ddump-json in https://gitlab.haskell.org/ghc/ghc/-/commit/91691117fc194c525f58ccd5b266dd1… <https://gitlab.haskell.org/ghc/ghc/-/commit/91691117fc194c525f58ccd5b266dd1…>. - Are there clients of -ddump-json? - Is there a specification of -ddump-json? - There will likely be a desire to evolve this feature. What is the process for doing so? Thanks! Richard

5 5

safe to upgrade my Mac?
by Richard Eisenberg 28 Jan '21

28 Jan '21

Hi devs, I'm currently running MacOS High Sierra, 10.13.6. Things are fine, but I'd like to upgrade to get Dark Mode and (hopefully) to speed up Mail. (That is, upgrading is nice, but not at all necessary.) If I upgrade, will GHC hate me? That is, will GHC 8.10 continue to work? Will I continue to be able to compile GHC? Will it be as performant? (A few years ago, I had a strange issue on a secondary computer where any binary built by GHC was horribly slow; we never got to the bottom of it.) All advice is welcome. Thanks! Richard

4 3

[ANNOUNCE] Glasgow Haskell Compiler 9.0.1-rc1 released
by Ben Gamari 26 Jan '21

26 Jan '21

Hello all, The GHC team is very pleased to announce the availability of the first release candidate of GHC 9.0.1 series. Source and binary distributions are available at the usual place: https://downloads.haskell.org/ghc/9.0.1-rc1/ This release candidate comes quite a bit later than expected after difficulty finding a performance neutral fix for a critical soundness bug, #17760. See [1] for details on the solution, particularly if you are a library maintainer currently using the touch# primop or withForeignPtr. Nevertheless, this release has nevertheless seen a considerable amount of testing and consequently we hope that this should be the last pre-release before the final release. In addition to numerous bug fixes, GHC 9.0.1 will bring a number of new features: * A first cut of the new LinearTypes language extension [2], allowing use of linear function syntax and linear record fields. * A new bignum library (ghc-bignum), allowing GHC to be more easily used with integer libraries other than GMP. * Improvements in code generation, resulting in considerable performance improvements in some programs. * Improvements in pattern-match checking, allowing more precise detection of redundant cases and reduced compilation time. * Implementation of the "simplified subsumption" proposal [3] simplifying the type system and paving the way for QuickLook impredicativity in GHC 9.2. * Implementation of the QualifiedDo extension [4], allowing more convenient overloading of `do` syntax. * Improvements in compilation time. And many more. See the release notes [5] for a full accounting of the changes in this release. As always, do test this release and open tickets for whatever issues you encounter. Cheers, - Ben [1] https://gitlab.haskell.org/ghc/ghc/-/wikis/migration/9.0#ghc-prim-07 [2] https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0111-l… [3] https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0287-s… [4] https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0216-q… [5] https://downloads.haskell.org/ghc/9.0.1-rc1/docs/html/users_guide/9.0.1-not…

3 2

[hadrian] HEAD link failure
by Shayne Fletcher 25 Jan '21

25 Jan '21

Building `HEAD` just now with `hadrian/build-stack --flavour=quickest -j`, I get a link failure for stage1 `bin/ghc`: ``` ld: library not found for -lHSrts-1.0_thr_l ``` I'll try another flavour or something but in the meantime is this ringing any bells? -- Shayne Fletcher

2 1

Benchmarking experiences: Cabal test vs compiling nofib/spectral/simple/Main.hs
by Andreas Klebinger 23 Jan '21

23 Jan '21

Hello Devs, When I started to work on GHC a few years back the Wiki recommended using nofib/spectral/simple/Main.hs as a test case for compiler performance changes. I've been using this ever since. "Recently" the cabal-test (compiling cabal-the-library) has become sort of a default benchmark for GHC performance. I've used the Cabal test as well and it's probably a better test case than nofib/spectral/simple/Main.hs. I've started using both usually using spectral/simple to benchmark intermediate changes and then looking at the cabal test for the final patch at the end. So far I have rarely seen a large difference between using cabal or spectral/simple. Sometimes the magnitude of the effect was different between the two, but I've never seen one regress/improve while the other didn't. Since the topic came up recently in a discussion I wonder if others use similar means to quickly bench ghc changes and what your experiences were in terms of simpler benchmarks being representative compared to the cabal test. Cheers, Andreas

3 2