February 2021 - ghc-devs

The future of Phabricator
by Ben Gamari 18 Sep '21

18 Sep '21

TL;DR. For several reasons I think we should consider alternatives to Phabricator. My view is that GitLab seems like the best option. Hello everyone, Over the past year I have been growing increasingly weary of our continued dependence on Phabricator. Without a doubt, its code review interface is the best I have used. However, for a myriad of reasons I am recently questioning whether it is still the best tool for GHC's needs. # The problem There are a number of reasons why I am currently uncertain about Phabricator. For one, at this point we have no options for support in the event that something goes wrong as the company responsible for Phabricator, Phacility, has closed their support channels to non-paying customers. Furthermore, in the past year or two Phacility has been placing their development resources in the parts their customers pay them for, which appear to be much different that the parts that we actively use. For this reason, some parts that we rely on seem oddly half-finished. This concern was recently underscored by some rather unfortunate misfeatures in Harbormaster which resulted in broken CI after the Hadrian merge and now apparent bugs which have made it difficult to migrate to the CircleCI integration we previously prepared. Perhaps most importantly, in our recent development priorities survey our use of Phabricator was the most common complaint by a fair margin, both in case of respondents who have contributed patches and those who have not. On the whole, past contributors and potential future contributors alike have strongly indicated that they want a more Git-like experience. Of course, this wasn't terribly surprising; this is just the most recent case where contributors have made this preference known. Frankly, in a sense it is hard to blame them. The fact that users need to install a PHP tool, Arcanist, to contribute anything but documentation patches has always seemed like unnecessary friction to me and I would be quite happy to be rid of it. Indeed we have had a quite healthy number of GitHub documentation patches since we started accepting them. This makes me thing that there may indeed be potential contributoes that we are leaving on the table. # What to do With Rackspace support ending at the end of year, now may be a good time to consider whether we really want to continue on this road. Phabricator is great at code review but I am less and less certain that it is worth the maintenance uncertainty and potential lost contributors that it costs. Moreover, good alternatives seem closer at-hand than they were when we deployed Phabricator. ## Move to GitHub When people complain about our infrastructure, they often use GitHub as the example of what they would like to see. However, realistically I have a hard time seeing GitHub as a viable option. Its feature set is simply insufficient enough to handle the needs of a larger project like GHC without significant external tooling (as seen in the case of Rust-lang). The concrete reasons have been well-documented in previous discussions but, to summarize, * its review functionality is extremely painful to use with larger patches * rebased patches lead to extreme confusion and lost review comments * it lacks support for pre-receive hooks, which serve as a last line of defense to prevent unintended submodule changes * its inability to deal with external tickets is problematic * there is essentially no possibility that we could eventually migrate GHC's tickets to GitHub's ticket tracker without considerable data loss (much less manage the ticket load that would result), meaning that we would forever be stuck with maintaining Trac. * on a personal note, its search functionality has often left me empty-handed On the whole, these issues seem pretty hard to surmount. ## Move to GitLab In using GitLab for another project over the past months I have been positively surprised by its quality. It handles rebased merge requests far better than GitHub, has functional search, and quite a usable review interface. Furthermore, upstream has been extremely responsive to suggestions for improvement [1]. Even out-of-the-box it seems to be flexible enough to accommodate our needs, supporting integration with external issue trackers, having reasonable release management features, and support for code owners to automate review triaging (replacing much of the functionality of Phabricator's Herald). Finally, other FOSS projects' [3] recent migrations from Phabrictor to GitLab have shown that GitLab-the-company is quite willing to offer help when needed. I took some time this weekend to try setting up a quick GHC instance [2] to play around with. Even after just a few hours of playing around I think the result is surprisingly usable. Out of curiosity I also played around with importing some tickets from Trac (building on Matt Pickering's Trac-to-Maniphest migration tool). With relatively little effort I was even able to get nearly all of our tickets (as of 9 months ago) imported while preserving ticket numbers (although there are naturally a few wrinkles that would need to be worked out). Naturally, I think we should treat the question of ticket tracker migration as an orthogonal one to code review, but it is good to know that this is possible. ## Continue with Phabricator Continuing with Phabricator is of course an option. Its review functionality is great and it has served us reasonably well. However, compared to GitLab and even GitHub of today its features seem less distinguished than they once did. Moreover, the prospect of having to maintain a largely stagnant product with no support strikes me as a slightly dangerous game to play. Working around the issues we have recently encountered has already cost a non-negligible amount of time. # The bottom line If it wasn't clear already, I think that we should strongly consider a move to GitLab. At this point it seems clear that it isn't going to vanish, has a steady pace of development, is featureful, and available. However, these are just my thoughts. What do you think? Cheers, - Ben [1] 11.4 will ship with a file tree view in the code review interface, which I reported (https://gitlab.com/gitlab-org/gitlab-ce/issues/46474) as being is one of the Phabricator features I missed the most during review [2] https://gitlab.home.smart-cactus.org/ghc/ghc/issues/14641 [3] The GNOME and freedesktop.org projects have recently migrated, the former from a hodge-podge of self-hosted services and the latter from Phabricator

17 40

New implementation for `ImpredicativeTypes`
by Alejandro Serrano Mena 08 Sep '21

08 Sep '21

Hi all, As I mentioned some time ago, we have been busy working on a new implementation of `ImpredicativeTypes` for GHC. I am very thankful to everybody who back then sent us examples of impredicativity which would be nice to support, as far as we know this branch supports all of them! :) If you want to try it, at https://gitlab.haskell.org/trupill/ghc/commit/a3f95a0fe0f647702fd7225fa719a… you can find the result of the pipeline, which includes builds for several platforms (click on the "Artifacts" button, the one which looks like a cloud, to get them). The code is being developed at https://gitlab.haskell.org/trupill/ghc. Any code should run *unchanged* except for some eta-expansion required for some specific usage patterns of higher-rank types. Please don't hesitate to ask any questions or clarifications about it. A merge request for tracking this can be found at https://gitlab.haskell.org/ghc/ghc/merge_requests/1659 Kind regards, Alejandro

8 23

Potential improvement in compacting GC
by Ömer Sinan Ağacan 02 Sep '21

02 Sep '21

Hello, I recently implemented the algorithm used by GHC's compacting GC in another project. The algorithm (after marking) makes two passes over the heap /generation. In GHC, these passes are implemented in [1] and in the next function. In my implementation I tried 3 ways of implementing these passes, one of which is the same as GHC's code, and benchmarked each version. I found that the fastest implementation is not what's used in GHC, but it could easily be used. I should mention that my code targets Wasm, and I benchmarked Wasm instructions executed. Not CPU cycles, CPU instructions, or anything like that. It's very likely that the results will be different when benchmarking code running on actual hardware. Secondly, in my case the heap is mostly dead (residency is low). In GHC, compaction for the oldest generation is enabled when residency passes a threshold, so the assumption is the heap is mostly live. I'm guessing this should also make some difference. Anyway, the first implementation I tried was similar to GHC's scan, but I did scan += object_size(scan); instead of bumping scan by one, as GHC does in [2]. This was the slowest version. Second implementation did the same as GHC (bumped scan by one). This was faster, but still slower than the next version. What I found to be the best is scanning the bitmap, not the heap. The bitmap iterator reads one word at a time. In each iteration it checks if the bitmap word is 0. In GHC, in the best case this can skip 16 words on heap on 32-bit systems, and 32 words on 64-bit. Reminder: we need two bits per object in the bitmap, see [3]. (this is not the case in my implementation so the payoff is better) When the bitmap word is not 0 I use "count trailing zeros" (or "count leading zeros" depending on the bitmap implementation) to get the number of words to skip. This is a single instruction on Wasm and x86 (TZCNT or LZCNT, available via __builtin_ctzl and __builtin_clzl in gcc). So instead of skipping one word at a time, this can potentially skip 16 words (or 32 on 64-bit architectures). When that's not possible, it can still skip multiple words by using ctz/clz. Ömer [1]: https://github.com/ghc/ghc/blob/bd877edd9499a351db947cd51ed583872b2facdf/rt… [2]: https://github.com/ghc/ghc/blob/bd877edd9499a351db947cd51ed583872b2facdf/rt… [3]: https://github.com/ghc/ghc/blob/bd877edd9499a351db947cd51ed583872b2facdf/rt…

5 6

Pattern matching desugaring regression? Re: [Haskell-cafe] Why does my module take so long to compile?
by Carter Schonwald 28 Mar '21

28 Mar '21

Ccing ghc devs since that’s a better forum perhaps Crazy theory: this is a regression due the the partial changes to pattern matching coverage checking in 8.10 that finished / landed in ghc 9 Why: Desugaring is when pattern/ case statement translation happens I think? And the only obvious “big thing” is that you have some huge , albeit sane for a compiler, pattern matching I’d first check if the new ghc 9 release doesn’t have that regression in build time that you experienced. And if it does file a ticket. I may be totally wrong, but that seems like a decent likelihood ! On Mon, Feb 15, 2021 at 12:20 PM Troels Henriksen <athas(a)sigkill.dk> wrote: > Carter Schonwald <carter.schonwald(a)gmail.com> writes: > > > How big are your data types in the file In question? Do you ghc generics > or > > deriving or template Haskell? Could you share a link to the file in > > question ? > > The file does not define any large data types itself, but it operates on > some fairly large data types (an AST, it's a compiler). But so do many > other modules that work just fine. It uses no generics, nontrivial > deriving, or Template Haskell. It's this file: > > https://github.com/diku-dk/futhark/blob/master/src/Futhark/Pass/ExtractKern… > > I also found a handful of other modules in my project that are > significantly slower to compile in GHC 8.10, and seemingly also because > of the desugarer, but none of them have any obvious smoking guns like > generics or TH. > > The only commonality I can find is that the affected modules contain > functions with a relatively large typeclass context. I use > ConstraintKinds to make them more concise, and I guess unfolded there > may be 30-40 equality/class constraints in total. Like this: > > type DistLore lore = > ( Bindable lore, > HasSegOp lore, > BinderOps lore, > LetDec lore ~ Type, > ExpDec lore ~ (), > BodyDec lore ~ () > ) > > where the 'Bindable' constraint in particular then implies a further > handful of "~" constraints: > > class > ( ASTLore lore, > FParamInfo lore ~ DeclType, > LParamInfo lore ~ Type, > RetType lore ~ DeclExtType, > BranchType lore ~ ExtType, > SetType (LetDec lore) > ) => > Bindable lore > where > ... > > FParamInfo/LParamInfo/etc are type families. Are such constraints > particularly slow? > > -- > \ Troels > /\ Henriksen >

4 7

Plan for GHC 9.2
by Ben Gamari 02 Mar '21

02 Mar '21

tl;dr. Provisional release schedule for 9.2 enclosed. Please discuss, especially if you have something you would like merged for 9.2.1. Hello all, With GHC 9.0.1 at long-last out the door, it is time that we start turning attention to GHC 9.2. I would like to avoid making the mistake made in the 9.0 series in starting the fork in a state that required a significant amount of backporting to be releaseable. Consequently, I want to make sure that we have a fork schedule that is realistic given the things that need to be merged for 9.2. These include: * Update haddock submodule in `master` (Ben) * Bumping bytestring to 0.11 (#19091, Ben) * Finishing the rework of sized integer primops (#19026, John Ericson) * Merge of ghc-exactprint into GHC? (Alan Zimmerman, Henry) * Merge BoxedRep (#17526, Ben) * ARM NCG backend and further stabilize Apple ARM support? (Moritz) * Some form of coercion zapping (Ben, Simon, Richard) * Tag inference analysis and tag check elision (Andreas) If you see something that you would like to see in 9.2.1 please do holler. Otherwise, if you see your name in this list it would be great if you could let me know when you think your project may be in a mergeable state. Ideally we would strive for a schedule like the following: 4 February 2021: We are here ~4 weeks pass 3 March 2021: Release branch forked 1 week passes 10 March 2021: Alpha 1 released 3 weeks pass 31 March 2021: Alpha 2 released 2 weeks pass 14 April 2021: Alpha 3 released 2 weeks pass 28 April 2021: Alpha 4 released 1 week passes 5 May 2021: Beta 1 released 1 week passes 12 May 2021: Release candidate 1 released 2 weeks pass 26 May 2021: Final release This provides ample time for stabilization while avoiding deviation from the usual May release timeframe. However, this would require that we move aggressively to start getting the tree into shape since the fork would be less than four weeks away. I would appreciate contributors' thoughts on the viability of this timeline. Cheers, - Ben

6 7

initIface* usage in plugins
by Josh Meredith 27 Feb '21

27 Feb '21

Hi, I'm trying to use initIfaceLoad and initIfaceLcl to lookup core `Name`s within a plugin. My understanding is that functions such as `lookupIfaceTop` are used for this, but I'm running into an issue that I suspect is caused by some `Name`s being inappropriate for this function, so they fail with the error `Iface id out of scope: ...`. Is there a robust way to select which `Name` lookup function to use based on a core binding/expression? Thanks, Josh

3 2

MR template text
by Simon Peyton Jones 23 Feb '21

23 Feb '21

I often see MRs in my inbox that say Thank you for your contribution to GHC! Please take a few moments to verify that your commits fulfill the following: [ ] are either individually buildable or squashed This is because the author hasn't changed the Description of the MR, but rather has left the template text unchanged. As a way to "nudge" authors to give reviewers more information, I suggest replacing the template text with the draft below. Does anyone have any views, for or against? Simon Proposed new template PLEASE REPLACE ALL OF THIS TEXT with a description of your merge request, including * A description of what the Merge Request does. For a single-commit MR, a copy of the commit message is often perfect. * A reference (e.g. #19415) to the ticket that led to this MR, and that describes the problem that this MR solves. Almost all MRs need a ticket, except the tiniest changes (e.g. code formatting) - A ticket describes a *problem* - A merge request describes a *solution* to that problem. While you are encouraged to write a good MR Description, it's not compulsory. You could just be putting up the MR to share with a colleague, for example. But if you want (a) to get code reviews from others, or (b) to land the patch in GHC, please do follow these guidelines. For general style guidance see https://gitlab.haskell.org/ghc/ghc/wikis/commentary/coding-style For completeness, the current template is Thank you for your contribution to GHC! Please take a few moments to verify that your commits fulfill the following: * [ ] are either individually buildable or squashed * [ ] have commit messages which describe *what they do* (referring to [Notes][notes] and tickets using `#NNNN` syntax when appropriate) * [ ] have added source comments describing your change. For larger changes you likely should add a [Note][notes] and cross-reference it from the relevant places. * [ ] add a [testcase to the testsuite](https://gitlab.haskell.org/ghc/ghc/wikis/building/running-tests/…. * [ ] replace this message with a description motivating your change If you have any questions don't hesitate to open your merge request and inquire in a comment. If your patch isn't quite done yet please do add prefix your MR title with `WIP:`. [notes]: https://gitlab.haskell.org/ghc/ghc/wikis/commentary/coding-style#comments-i…

4 5

Changes to performance testing?
by Ben Gamari 23 Feb '21

23 Feb '21

Hi all, Recently our performance tests have been causing quite some pain. One reason for this is due to our new Darwin runners (see #19025), which (surprisingly) differ significantly in their performance characteristics (perhaps due to running Big Sur or using native tools provided by nix?). However, this is further exacerbated by the fact that there are quite a few people working on compiler performance currently (horray!). This leads to the following failure mode during Marge jobs: 1. Merge request A improves test T1234 by 0.5%, which is within the test's acceptance window and therefore CI passes 2. Merge request B *also* improves test T1234 by another 0.5%, which similarly passes CI 3. Marge tries to merge MRs A and B in a batch but finds that the combined 1% improvement in T1234 is *outside* the acceptance window. Consequently, the batch fails. This is quite painful, especially given that it creates work for those trying to improve GHC (as the saying goes: no good deed goes unpunished). To mitigate this I would suggest that we allow performance test failures in marge-bot pipelines. A slightly weaker variant of this idea would instead only allow performance *improvements*. I suspect the latter would get most of the benefit, while eliminating the possibility that a large regression goes unnoticed. Thoughts? Cheers, - Ben

4 4

On CI
by Moritz Angermann 22 Feb '21

22 Feb '21

Friends, I've been looking at CI recently again, as I was facing CI turnaround times of 9-12hs; and this just keeps dragging out and making progress hard. The pending pipeline currently has 2 darwin, and 15 windows builds waiting. Windows builds on average take ~220minutes. We have five builders, so we can expect this queue to be done in ~660 minutes assuming perfect scheduling and good performance. That is 11hs! The next windows build can be started in 11hs. Please check my math and tell me I'm wrong! If you submit a MR today, with some luck, you'll be able to know if it will be mergeable some time tomorrow. At which point you can assign it to marge, and marge, if you are lucky and the set of patches she tries to merge together is mergeable, will merge you work into master probably some time on Friday. If a job fails, well you have to start over again. What are our options here? Ben has been pretty clear about not wanting a broken commit for windows to end up in the tree, and I'm there with him. Cheers, Moritz

8 15

Weekly show & tell video meeting
by Csaba Hruska 16 Feb '21

16 Feb '21

Hello, Would you be interested in a weekly show & tell video meeting? The topic would be Haskell & compilers in general. Either GHC or non-GHC related. Regards, Csaba

2 3