December 2020 - ghc-devs

The future of Phabricator
by Ben Gamari 18 Sep '21

18 Sep '21

TL;DR. For several reasons I think we should consider alternatives to Phabricator. My view is that GitLab seems like the best option. Hello everyone, Over the past year I have been growing increasingly weary of our continued dependence on Phabricator. Without a doubt, its code review interface is the best I have used. However, for a myriad of reasons I am recently questioning whether it is still the best tool for GHC's needs. # The problem There are a number of reasons why I am currently uncertain about Phabricator. For one, at this point we have no options for support in the event that something goes wrong as the company responsible for Phabricator, Phacility, has closed their support channels to non-paying customers. Furthermore, in the past year or two Phacility has been placing their development resources in the parts their customers pay them for, which appear to be much different that the parts that we actively use. For this reason, some parts that we rely on seem oddly half-finished. This concern was recently underscored by some rather unfortunate misfeatures in Harbormaster which resulted in broken CI after the Hadrian merge and now apparent bugs which have made it difficult to migrate to the CircleCI integration we previously prepared. Perhaps most importantly, in our recent development priorities survey our use of Phabricator was the most common complaint by a fair margin, both in case of respondents who have contributed patches and those who have not. On the whole, past contributors and potential future contributors alike have strongly indicated that they want a more Git-like experience. Of course, this wasn't terribly surprising; this is just the most recent case where contributors have made this preference known. Frankly, in a sense it is hard to blame them. The fact that users need to install a PHP tool, Arcanist, to contribute anything but documentation patches has always seemed like unnecessary friction to me and I would be quite happy to be rid of it. Indeed we have had a quite healthy number of GitHub documentation patches since we started accepting them. This makes me thing that there may indeed be potential contributoes that we are leaving on the table. # What to do With Rackspace support ending at the end of year, now may be a good time to consider whether we really want to continue on this road. Phabricator is great at code review but I am less and less certain that it is worth the maintenance uncertainty and potential lost contributors that it costs. Moreover, good alternatives seem closer at-hand than they were when we deployed Phabricator. ## Move to GitHub When people complain about our infrastructure, they often use GitHub as the example of what they would like to see. However, realistically I have a hard time seeing GitHub as a viable option. Its feature set is simply insufficient enough to handle the needs of a larger project like GHC without significant external tooling (as seen in the case of Rust-lang). The concrete reasons have been well-documented in previous discussions but, to summarize, * its review functionality is extremely painful to use with larger patches * rebased patches lead to extreme confusion and lost review comments * it lacks support for pre-receive hooks, which serve as a last line of defense to prevent unintended submodule changes * its inability to deal with external tickets is problematic * there is essentially no possibility that we could eventually migrate GHC's tickets to GitHub's ticket tracker without considerable data loss (much less manage the ticket load that would result), meaning that we would forever be stuck with maintaining Trac. * on a personal note, its search functionality has often left me empty-handed On the whole, these issues seem pretty hard to surmount. ## Move to GitLab In using GitLab for another project over the past months I have been positively surprised by its quality. It handles rebased merge requests far better than GitHub, has functional search, and quite a usable review interface. Furthermore, upstream has been extremely responsive to suggestions for improvement [1]. Even out-of-the-box it seems to be flexible enough to accommodate our needs, supporting integration with external issue trackers, having reasonable release management features, and support for code owners to automate review triaging (replacing much of the functionality of Phabricator's Herald). Finally, other FOSS projects' [3] recent migrations from Phabrictor to GitLab have shown that GitLab-the-company is quite willing to offer help when needed. I took some time this weekend to try setting up a quick GHC instance [2] to play around with. Even after just a few hours of playing around I think the result is surprisingly usable. Out of curiosity I also played around with importing some tickets from Trac (building on Matt Pickering's Trac-to-Maniphest migration tool). With relatively little effort I was even able to get nearly all of our tickets (as of 9 months ago) imported while preserving ticket numbers (although there are naturally a few wrinkles that would need to be worked out). Naturally, I think we should treat the question of ticket tracker migration as an orthogonal one to code review, but it is good to know that this is possible. ## Continue with Phabricator Continuing with Phabricator is of course an option. Its review functionality is great and it has served us reasonably well. However, compared to GitLab and even GitHub of today its features seem less distinguished than they once did. Moreover, the prospect of having to maintain a largely stagnant product with no support strikes me as a slightly dangerous game to play. Working around the issues we have recently encountered has already cost a non-negligible amount of time. # The bottom line If it wasn't clear already, I think that we should strongly consider a move to GitLab. At this point it seems clear that it isn't going to vanish, has a steady pace of development, is featureful, and available. However, these are just my thoughts. What do you think? Cheers, - Ben [1] 11.4 will ship with a file tree view in the code review interface, which I reported (https://gitlab.com/gitlab-org/gitlab-ce/issues/46474) as being is one of the Phabricator features I missed the most during review [2] https://gitlab.home.smart-cactus.org/ghc/ghc/issues/14641 [3] The GNOME and freedesktop.org projects have recently migrated, the former from a hodge-podge of self-hosted services and the latter from Phabricator

17 40

New implementation for `ImpredicativeTypes`
by Alejandro Serrano Mena 08 Sep '21

08 Sep '21

Hi all, As I mentioned some time ago, we have been busy working on a new implementation of `ImpredicativeTypes` for GHC. I am very thankful to everybody who back then sent us examples of impredicativity which would be nice to support, as far as we know this branch supports all of them! :) If you want to try it, at https://gitlab.haskell.org/trupill/ghc/commit/a3f95a0fe0f647702fd7225fa719a… you can find the result of the pipeline, which includes builds for several platforms (click on the "Artifacts" button, the one which looks like a cloud, to get them). The code is being developed at https://gitlab.haskell.org/trupill/ghc. Any code should run *unchanged* except for some eta-expansion required for some specific usage patterns of higher-rank types. Please don't hesitate to ask any questions or clarifications about it. A merge request for tracking this can be found at https://gitlab.haskell.org/ghc/ghc/merge_requests/1659 Kind regards, Alejandro

8 23

[ANNOUNCE] Glasgow Haskell Compiler 9.0.1-rc1 released
by Ben Gamari 26 Jan '21

26 Jan '21

Hello all, The GHC team is very pleased to announce the availability of the first release candidate of GHC 9.0.1 series. Source and binary distributions are available at the usual place: https://downloads.haskell.org/ghc/9.0.1-rc1/ This release candidate comes quite a bit later than expected after difficulty finding a performance neutral fix for a critical soundness bug, #17760. See [1] for details on the solution, particularly if you are a library maintainer currently using the touch# primop or withForeignPtr. Nevertheless, this release has nevertheless seen a considerable amount of testing and consequently we hope that this should be the last pre-release before the final release. In addition to numerous bug fixes, GHC 9.0.1 will bring a number of new features: * A first cut of the new LinearTypes language extension [2], allowing use of linear function syntax and linear record fields. * A new bignum library (ghc-bignum), allowing GHC to be more easily used with integer libraries other than GMP. * Improvements in code generation, resulting in considerable performance improvements in some programs. * Improvements in pattern-match checking, allowing more precise detection of redundant cases and reduced compilation time. * Implementation of the "simplified subsumption" proposal [3] simplifying the type system and paving the way for QuickLook impredicativity in GHC 9.2. * Implementation of the QualifiedDo extension [4], allowing more convenient overloading of `do` syntax. * Improvements in compilation time. And many more. See the release notes [5] for a full accounting of the changes in this release. As always, do test this release and open tickets for whatever issues you encounter. Cheers, - Ben [1] https://gitlab.haskell.org/ghc/ghc/-/wikis/migration/9.0#ghc-prim-07 [2] https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0111-l… [3] https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0287-s… [4] https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0216-q… [5] https://downloads.haskell.org/ghc/9.0.1-rc1/docs/html/users_guide/9.0.1-not…

3 2

can't get raw logs -- server down?
by Richard Eisenberg 24 Dec '20

24 Dec '20

Hi devs, I have a failing CI run at https://gitlab.haskell.org/ghc/ghc/-/jobs/533674 <https://gitlab.haskell.org/ghc/ghc/-/jobs/533674>. I can't repro the problem locally, so I wanted to see the raw log. But clicking the raw log button (leading to https://gitlab.haskell.org/ghc/ghc/-/jobs/533674/raw <https://gitlab.haskell.org/ghc/ghc/-/jobs/533674/raw>) gets nothing. It looks like a server is down. Help? Thanks! Richard

2 2

Troubles when building ghc
by Neophytos Michael 24 Dec '20

24 Dec '20

I am having some trouble building ghc on ubuntu. I have ghc 8.8.4 installed on the machine. I pulled from the main repository and followed the instructions here: https://gitlab.haskell.org/ghc/ghc/-/wikis/building/hadrian Building with: ./hadrian/build -j it builds for a long time until it finally fails with the message below. I would appreciate some advice on how to fix this. Thank you, Neo =========================== Exit code: 1 Stderr and Stdout: _build/stage1/compiler/build/GHC/Parser/Lexer.hs:3495:3: error: Variable not in scope: int16ToInt# :: Int# -> Int# | 3495 | int16ToInt# | ^^^^^^^^^^^ _build/stage1/compiler/build/GHC/Parser/Lexer.hs:3520:3: error: Variable not in scope: int32ToInt# :: Int# -> Int# | 3520 | int32ToInt# | ^^^^^^^^^^^ =========================== On that line (in Lexer.hs) we see: #if __GLASGOW_HASKELL__ >= 901 int16ToInt# #endif (indexInt16OffAddr# arr off) #endif so we are in the "then" clause and somehow "int16ToInt#" is nowhere to be found. -- Neo

3 3

Fat interface files?
by Ben Gamari 21 Dec '20

21 Dec '20

Hi Edward, While chatting with the ghc-ide folks recently I realized that it would be useful to be able to preserve Core such that compilation can be restarted (e.g. to be pushed down the bytecode pipeline to evaluate TH splices). As I recall this is precisely what you implemented in your "fat interface file" work. Do you recall what the state of this work was? Do you have a branch with last-known-good work? Do you recall any tricky questions that remained outstanding? Cheers, - Ben

5 6

[ANNOUNCE] Glasgow Haskell Compiler 8.10.3 released
by Ben Gamari 20 Dec '20

20 Dec '20

Hello all, The GHC team is happy to announce the release of GHC 8.10.3. Source and binary distributions are available at the usual place: https://downloads.haskell.org/ghc/8.10.3/ GHC 8.10.3 fixes a number of issues in present in GHC 8.10.2 including: * Numerous stability improves on Windows * More robust support for architectures with weak memory ordering guarantees (e.g. modern ARM hardware). * GHC can now split dynamic objects to accomodate macOS' RPATH size limitation when building large projects (#14444) * Several correctness bugs in the new low-latency garbage collector * Many, many other bug-fixes Note that at the moment we still require that macOS Catalina users exempt the binary distribution from the notarization requirement by running `xattr -cr .` on the unpacked tree before running `make install`. This situation will hopefully be improved for GHC 9.0.1 with the resolution of #17418 [1]. Cheers, - Ben [1] https://gitlab.haskell.org/ghc/ghc/issues/17418

3 3

Status and GHC's treatment of Haddock
by Ben Gamari 15 Dec '20

15 Dec '20

Hi all, Sadly `master` has been broken for much of the day. The causal-chain is roughly the following: 1. Last week a haddock merge-request was prematurely merged by a haddock developer, despite not being compilable, to the `ghc-head` branch. 2. marge-bot failed during a merge on Sunday 3. On Monday I finished merging the half-merged batch manually, not aware of the inconsistent submodule state. However, while doing so some unintended changes crept in, relying on an unmerged haddock commit. 4. in an attempt to fix the inconsistent submodule state another developer merged the unmerged commit 5. I notice the unintentional commits on `master` and start to back them out 6. I note that even after backing out the unintentional commits the tree *still* doesn't build due to haddock 7. I perform a fair amount of head-scratching trying to figure out what is going on; eventually I find the non-building commit. 8. I eventually sort out what needs to be backed-out and in what order I take a few lessons from this: * manual merges, no matter how "straightforward", are always a vector for mistakes. I should have triple-checked my work before pushing manually pushing the half-merged branch. * haddock submodule changes going in prematurely come with a very high cost, especially if they don't build. It turns out I had run into the non-compilable haddock commit a few times last week but had chalked it up to in-flight changes. On the whole, I am really starting to wonder whether we can't improve our model for dealing with haddock; it is currently quite easy for haddock and GHC to enter an inconsistent state. Moreover, the fact that `haddock`'s `ghc-head` branch runs ahead of `master` means that we won't realize the problem until it's too late. One small improvement would be to move Haddock to GitLab. This would at least ensure that there is a clear connection between haddock MRs and their corresponding GHC changes. I have suggested this to Alex. Beyond this, it would be great if someone could step up to finish the hi-haddock change [1]. I suspect this refactoring has the potential to greatly improve this situation: the interface file AST tends to be more stable that the Core AST and has significantly less surface area. After this is in place Haddock could then use either its own AST or a frozen HsSyn AST from GHC (faciliated by TTG) internally, greatly reducing the coupling between Haddock and GHC. Any takers? Cheers, - Ben

1 0

Nested constructed product results?
by Alexis King 15 Dec '20

15 Dec '20

Hi all, I spent some time today looking into the performance of a program involving a parser type that looks something like this: data AnnotatedParser a = AnnotatedParser { annotation :: Annotation , parser :: String -> (Maybe a, String) } The `Annotation` records metadata about the structure of an `AnnotatedParser` that can be accessed statically (that is, without having to run the parser on some input). `AnnotatedParser`s are built from various primitive constructors and composed using various combinators. These combinators end up looking something like this: (<+>) :: AnnotatedParser a -> AnnotatedParser b -> AnnotatedParser (a, b) AnnotatedParser ann1 f <+> AnnotatedParser ann2 g = AnnotatedParser { annotation = Seq ann1 ann2 , parser = \s1 -> let !(a, s2) = f s1 !(b, s3) = g s2 in ((,) <$> a <*> b, s3) } Use of these combinators leads to the construction and subsequent case analysis of numerous `AnnotatedParser` closures. Happily, constructed product result[1] analysis kicks in and rewrites such combinators to cut down on the needless boxing, leading to worker/wrapper splits like this: $w<+> :: Annotation -> (String -> (Maybe a, String)) -> Annotation -> (String -> (Maybe b, String)) -> (# Annotation, String -> (Maybe (a, b), String) #) $w<+> ann1 f ann2 g = (# Seq ann1 ann2 , \s1 -> let !(a, s2) = f s1 !(b, s3) = g s2 in ((,) <$> a <*> b, s3) #) <+> :: AnnotatedParser a -> AnnotatedParser b -> AnnotatedParser (a, b) <+> (AnnotatedParser ann1 f) (AnnotatedParser ann2 g) = case $w<+> ann1 f ann2 g of (# a, b #) -> AnnotatedParser a b {-# INLINE <+> #-} This is great, and it cuts down on allocation significantly, but there is still something unsatisfying about it: the `parser` function inside the record is not affected by CPR! This is a shame, because essentially all use sites immediately deconstruct the pair, making it a prime candidate for unboxing. Ideally, we’d like to get this, instead: $w<+> :: Annotation -> (String -> (Maybe a, String)) -> Annotation -> (String -> (Maybe b, String)) -> (# Annotation, String -> (# Maybe (a, b), String #) #) $w<+> ann1 f ann2 g = (# Seq ann1 ann2 , \s1 -> let !(a, s2) = f s1 !(b, s3) = g s2 in (# (,) <$> a <*> b, s3 #) #) In practice, little combinators like `$w<+>` are marked INLINE, so `f` and `g` are usually known rather than unknown calls. This nested CPR transformation would allow the tuple construction to fuse with the tuple deconstruction, eliminating quite a lot of unnecessary boxing/unboxing. Unfortunately, it seems as though GHC’s implementation of CPR is entirely first-order: although function arguments are given rich demand signatures, results are only described one level deep. But as the above example hopefully illustrates, that’s leaving significant optimization opportunities on the table! Hence, my questions: 1. Has this notion of “nested CPR” been explored at all before? 2. Does such an extension to CPR sound worth its weight? I peeked a little at GHC.Types.Cpr and GHC.Core.Opt.CprAnal, and it seems quite manageable to me, but I haven’t actually looked into an implementation attempt just yet. I’m mostly interested in whether others have thought about something like this and/or run into similar issues in the past, or if this is really an unusual construction. Thanks, Alexis [1]: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/07/cpr.pdf

2 1

[ANNOUNCEMENT] Change in MR landing procedure
by Ben Gamari 10 Dec '20

10 Dec '20

tl;dr. On Thursday 10 Dec, Marge-bot will be relocated to a retirement facility. Her duties will henceforth be carried out by GitLab's Merge Train functionality. Hi all, When GHC moved to GitLab two years ago the platform lacked a critical feature: the ability to manage a queue of merge requests slated to be merged. For the past two years we have been using @marge-bot for this functionality. While Marge has served us well, she also introduces her fair share of failure modes. In the meantime GitLab has grown proper support for our use-case in the form of the "merge train" mechanism [1]. In light of Marge's growing intransigence [2], I will grant her a well-earned retirement and begin using merge trains for merging MRs starting on Thursday. As a result, the workflow for adding a merge request to the merge queue will become a bit more straightforward: Instead of setting the MR's "assignee" to @marge-bot rather push the blue "Start/add to merge train" button. This adds the merge request to the merge queue. This is documented in the Wiki [3]. Apart from freeing us from the need to maintain Marge, this change also enables us to use GitLab's branch-squashing functionality, which Marge did not support. Do let me know if you have any questions or concerns. So long, Marge, and thanks for all of the commits. Cheers, - Ben [1] https://docs.gitlab.com/ee/ci/merge_request_pipelines/pipelines_for_merged_… [2] https://gitlab.haskell.org/ghc/ghc/-/issues/19034 [3] https://gitlab.haskell.org/ghc/ghc/-/wikis/Contributing-a-Patch

2 2