May 2021 - ghc-devs - Haskell.org

The future of Phabricator
by Ben Gamari 19 Sep '21

19 Sep '21

TL;DR. For several reasons I think we should consider alternatives to Phabricator. My view is that GitLab seems like the best option. Hello everyone, Over the past year I have been growing increasingly weary of our continued dependence on Phabricator. Without a doubt, its code review interface is the best I have used. However, for a myriad of reasons I am recently questioning whether it is still the best tool for GHC's needs. # The problem There are a number of reasons why I am currently uncertain about Phabricator. For one, at this point we have no options for support in the event that something goes wrong as the company responsible for Phabricator, Phacility, has closed their support channels to non-paying customers. Furthermore, in the past year or two Phacility has been placing their development resources in the parts their customers pay them for, which appear to be much different that the parts that we actively use. For this reason, some parts that we rely on seem oddly half-finished. This concern was recently underscored by some rather unfortunate misfeatures in Harbormaster which resulted in broken CI after the Hadrian merge and now apparent bugs which have made it difficult to migrate to the CircleCI integration we previously prepared. Perhaps most importantly, in our recent development priorities survey our use of Phabricator was the most common complaint by a fair margin, both in case of respondents who have contributed patches and those who have not. On the whole, past contributors and potential future contributors alike have strongly indicated that they want a more Git-like experience. Of course, this wasn't terribly surprising; this is just the most recent case where contributors have made this preference known. Frankly, in a sense it is hard to blame them. The fact that users need to install a PHP tool, Arcanist, to contribute anything but documentation patches has always seemed like unnecessary friction to me and I would be quite happy to be rid of it. Indeed we have had a quite healthy number of GitHub documentation patches since we started accepting them. This makes me thing that there may indeed be potential contributoes that we are leaving on the table. # What to do With Rackspace support ending at the end of year, now may be a good time to consider whether we really want to continue on this road. Phabricator is great at code review but I am less and less certain that it is worth the maintenance uncertainty and potential lost contributors that it costs. Moreover, good alternatives seem closer at-hand than they were when we deployed Phabricator. ## Move to GitHub When people complain about our infrastructure, they often use GitHub as the example of what they would like to see. However, realistically I have a hard time seeing GitHub as a viable option. Its feature set is simply insufficient enough to handle the needs of a larger project like GHC without significant external tooling (as seen in the case of Rust-lang). The concrete reasons have been well-documented in previous discussions but, to summarize, * its review functionality is extremely painful to use with larger patches * rebased patches lead to extreme confusion and lost review comments * it lacks support for pre-receive hooks, which serve as a last line of defense to prevent unintended submodule changes * its inability to deal with external tickets is problematic * there is essentially no possibility that we could eventually migrate GHC's tickets to GitHub's ticket tracker without considerable data loss (much less manage the ticket load that would result), meaning that we would forever be stuck with maintaining Trac. * on a personal note, its search functionality has often left me empty-handed On the whole, these issues seem pretty hard to surmount. ## Move to GitLab In using GitLab for another project over the past months I have been positively surprised by its quality. It handles rebased merge requests far better than GitHub, has functional search, and quite a usable review interface. Furthermore, upstream has been extremely responsive to suggestions for improvement [1]. Even out-of-the-box it seems to be flexible enough to accommodate our needs, supporting integration with external issue trackers, having reasonable release management features, and support for code owners to automate review triaging (replacing much of the functionality of Phabricator's Herald). Finally, other FOSS projects' [3] recent migrations from Phabrictor to GitLab have shown that GitLab-the-company is quite willing to offer help when needed. I took some time this weekend to try setting up a quick GHC instance [2] to play around with. Even after just a few hours of playing around I think the result is surprisingly usable. Out of curiosity I also played around with importing some tickets from Trac (building on Matt Pickering's Trac-to-Maniphest migration tool). With relatively little effort I was even able to get nearly all of our tickets (as of 9 months ago) imported while preserving ticket numbers (although there are naturally a few wrinkles that would need to be worked out). Naturally, I think we should treat the question of ticket tracker migration as an orthogonal one to code review, but it is good to know that this is possible. ## Continue with Phabricator Continuing with Phabricator is of course an option. Its review functionality is great and it has served us reasonably well. However, compared to GitLab and even GitHub of today its features seem less distinguished than they once did. Moreover, the prospect of having to maintain a largely stagnant product with no support strikes me as a slightly dangerous game to play. Working around the issues we have recently encountered has already cost a non-negligible amount of time. # The bottom line If it wasn't clear already, I think that we should strongly consider a move to GitLab. At this point it seems clear that it isn't going to vanish, has a steady pace of development, is featureful, and available. However, these are just my thoughts. What do you think? Cheers, - Ben [1] 11.4 will ship with a file tree view in the code review interface, which I reported (https://gitlab.com/gitlab-org/gitlab-ce/issues/46474) as being is one of the Phabricator features I missed the most during review [2] https://gitlab.home.smart-cactus.org/ghc/ghc/issues/14641 [3] The GNOME and freedesktop.org projects have recently migrated, the former from a hodge-podge of self-hosted services and the latter from Phabricator

17 40

New implementation for `ImpredicativeTypes`
by Alejandro Serrano Mena 08 Sep '21

08 Sep '21

Hi all, As I mentioned some time ago, we have been busy working on a new implementation of `ImpredicativeTypes` for GHC. I am very thankful to everybody who back then sent us examples of impredicativity which would be nice to support, as far as we know this branch supports all of them! :) If you want to try it, at https://gitlab.haskell.org/trupill/ghc/commit/a3f95a0fe0f647702fd7225fa719a… you can find the result of the pipeline, which includes builds for several platforms (click on the "Artifacts" button, the one which looks like a cloud, to get them). The code is being developed at https://gitlab.haskell.org/trupill/ghc. Any code should run *unchanged* except for some eta-expansion required for some specific usage patterns of higher-rank types. Please don't hesitate to ask any questions or clarifications about it. A merge request for tracking this can be found at https://gitlab.haskell.org/ghc/ghc/merge_requests/1659 Kind regards, Alejandro

8 23

Potential improvement in compacting GC
by Ömer Sinan Ağacan 03 Sep '21

03 Sep '21

Hello, I recently implemented the algorithm used by GHC's compacting GC in another project. The algorithm (after marking) makes two passes over the heap /generation. In GHC, these passes are implemented in [1] and in the next function. In my implementation I tried 3 ways of implementing these passes, one of which is the same as GHC's code, and benchmarked each version. I found that the fastest implementation is not what's used in GHC, but it could easily be used. I should mention that my code targets Wasm, and I benchmarked Wasm instructions executed. Not CPU cycles, CPU instructions, or anything like that. It's very likely that the results will be different when benchmarking code running on actual hardware. Secondly, in my case the heap is mostly dead (residency is low). In GHC, compaction for the oldest generation is enabled when residency passes a threshold, so the assumption is the heap is mostly live. I'm guessing this should also make some difference. Anyway, the first implementation I tried was similar to GHC's scan, but I did scan += object_size(scan); instead of bumping scan by one, as GHC does in [2]. This was the slowest version. Second implementation did the same as GHC (bumped scan by one). This was faster, but still slower than the next version. What I found to be the best is scanning the bitmap, not the heap. The bitmap iterator reads one word at a time. In each iteration it checks if the bitmap word is 0. In GHC, in the best case this can skip 16 words on heap on 32-bit systems, and 32 words on 64-bit. Reminder: we need two bits per object in the bitmap, see [3]. (this is not the case in my implementation so the payoff is better) When the bitmap word is not 0 I use "count trailing zeros" (or "count leading zeros" depending on the bitmap implementation) to get the number of words to skip. This is a single instruction on Wasm and x86 (TZCNT or LZCNT, available via __builtin_ctzl and __builtin_clzl in gcc). So instead of skipping one word at a time, this can potentially skip 16 words (or 32 on 64-bit architectures). When that's not possible, it can still skip multiple words by using ctz/clz. Ömer [1]: https://github.com/ghc/ghc/blob/bd877edd9499a351db947cd51ed583872b2facdf/rt… [2]: https://github.com/ghc/ghc/blob/bd877edd9499a351db947cd51ed583872b2facdf/rt… [3]: https://github.com/ghc/ghc/blob/bd877edd9499a351db947cd51ed583872b2facdf/rt…

5 6

GHC and the future of Freenode
by Ben Gamari 17 Jun '21

17 Jun '21

Hi all, As you may have heard the Freenode IRC network, long the home of #ghc and several other prominent Haskell channels, appears to be in the middle of a rather nasty hostile takeover [1,2,3,4]. As a consequence, it seems it will be necessary to migrate the #ghc community elsewhere. The next question is, of course, where will this be. One option is Liberachat, the spiritual successor of Freenode. However, in recent years I have also heard an increasingly loud choir of users, contributors, and potential-contributors note how archane IRC feels when compared to other modern chat platforms. Using IRC effectively in a collaborative environment essentially requires that all parties use a bouncer; however, this is (understandably) isn't something that most users are willing to do. Consequently, I think it would be wise to expand our search space to include other FOSS platforms. At the moment, I can see the following options: 1. Remain on IRC and move to Liberachat, the spiritual successor of Freenode 2. Remain on IRC and move to OFTC, another widely used network 3. Move to [Matrix] 4. Move to [Zulip] My sense is that of the non-IRC options Matrix is the truest successor to IRC, being a federated protocol with a wide array of (if somewhat immature) clients. I know some of our contributors already use it and in principle one could configure an IRC-to-Matrix bridge for those existing contributors who would rather continue using IRC. Zulip, while also being FOSS, is far more centralized than Matrix and appears to be more of a chat web application than an open protocol. Do you know of any other options? Thoughts? Cheers, - Ben [1]: https://gist.github.com/joepie91/df80d8d36cd9d1bde46ba018af497409 [2]: https://fuchsnet.ch/freenode-resign-letter.txt [3]: https://gist.github.com/aaronmdjones/1a9a93ded5b7d162c3f58bdd66b8f491 [4]: https://mniip.com/freenode.txt [Matrix]: https://www.matrix.org/ [Zulip]:

20 27

GitLab downtime
by Ben Gamari 02 Jun '21

02 Jun '21

Hi all, Today I'm going to be working on migrating our GitLab instance to a new machine. Starting in likely an hour there will be a few hours of downtime while the migration proceeds. I will send an email when the downtime begins and concludes. Apologies for the late notice! Cheers, - Ben

6 10

GHC releases and versions
by Sylvain Henry 29 May '21

29 May '21

Hi devs, We currently have 4 branches of GHC in flight: 8.10, 9.0, 9.2 and HEAD Latest releases: - 8.10.4: 2021/02/06 - 9.0.1: 2021/02/04 - 9.2.1-alpha2: 2021/04/23 Considering: 1) 8.10 branch should be stable but a lot of stuff has been merged for 8.10.5. To the point that 8.10.5 should probably be a "major release in the 8.10 series". 2) 9.0.1 is the latest major release but it shouldn't be used before 9.0.2 is released because of bugs and regressions (9.0.2 branch contains a fix for a critical bug in 9.0.1 [1] since March). 3) We might release 9.2.1 and 9.0.2 approximately at the same time which will be quite confusing for users ("9.0.2 in the 9.0 series and 9.2.1 in the 9.2 series"). 4) The first major number is meaningless. Proposition: Switch to A.B.C[.D] version scheme where: - A: major release ("series") - B: major release in the A series if B>0 and C=0; beta release if B=0 - C: bugfix release for A.B (if C>0) or beta release number (if B=0) - D: date when building in tree, not for releases It might be clearer exposed like this: showVersion = \case [a,b,c,d] -> "Dev version of " ++ showVersion [a,b,c] ++ " built on " ++ show d [a,0,c] -> "beta " ++ show c ++ " in series " ++ show a [a,b,0] -> "Major release " ++ show [a,b] ++ " in series " ++ show a [a,b,c] -> "Bugfix release " ++ show c ++ " for " ++ show [a,b] _ -> undefined > showVersion [9,0,1,20211028] "Dev version of beta 1 in series 9 built on 20211028" > showVersion [9,0,1] "beta 1 in series 9" > showVersion [9,0,2] "beta 2 in series 9" > showVersion [9,1,0] "Major release [9,1] in series 9" > showVersion [9,1,1] "Bugfix release 1 for [9,1]" > showVersion [9,2,0] "Major release [9,2] in series 9" > showVersion [10,1,0] "Major release [10,1] in series 10" Effects: 1) We could use C for bugfix versions which are to be released much faster than major versions. 2) B would be used for the old series we maintain. We backport a lot more stuff than before in older releases it seems, so it would be more PVP compliant to bump a major version number. 3) A would be used for the usual 6-month major releases. 4) We could make major releases in the 8 series (e.g. 8.10.5 could be released as 8.11.0) 5) We could advertise 9.0.1 as a beta (as everyone seems to consider .1 releases) 6) 9.2.1 final could be released either as 9.3 (next major in the 9 series if we just forget about 9.0.* and 9.2.*) or as 10.1.0 (first major in the 10 series). 7) No difference anymore between even/odd version numbers (for reference the current scheme is explained in [2]) Any thoughts? Sylvain [1] https://mail.haskell.org/pipermail/haskell-cafe/2021-March/133540.html [2] https://gitlab.haskell.org/ghc/ghc/-/wikis/working-conventions/releases

2 1

Errors in haddock git fetch
by Simon Peyton Jones 28 May '21

28 May '21

I'm getting these errors from `git submodule update`. Should I worry? Simon >From https://gitlab.haskell.org/ghc/haddock * [new branch] az/T19834 -> origin/az/T19834 * [new branch] az/T19834-2 -> origin/az/T19834-2 * [new branch] az/T19845 -> origin/az/T19845 * [new branch] az/T19845-2 -> origin/az/T19845-2 * [new branch] az/T19845-3 -> origin/az/T19845-3 * [new branch] dependabot/npm_and_yarn/haddock-api/resources/html/hosted-git-info-2.8.9 -> origin/dependabot/npm_and_yarn/haddock-api/resources/html/hosted-git-info-2.8.9 * [new branch] dependabot/npm_and_yarn/haddock-api/resources/html/lodash-4.17.21 -> origin/dependabot/npm_and_yarn/haddock-api/resources/html/lodash-4.17.21 * [new branch] dn/dn-driver-refactor-and-split -> origin/dn/dn-driver-refactor-and-split b4e7407b..c7281407 ghc-9.2 -> origin/ghc-9.2 dabdee14..4f9088e4 ghc-head -> origin/ghc-head * [new branch] wip/T18389-task-zero -> origin/wip/T18389-task-zero + 7d27ea7a...3b6a8774 wip/T19720 -> origin/wip/T19720 (forced update) + fe35fed3...40ba457f wip/adinapoli-align-ps-messages -> origin/wip/adinapoli-align-ps-messages (forced update) * [new branch] wip/dn-driver-refactor-and-split -> origin/wip/dn-driver-refactor-and-split error: cannot lock ref 'refs/remotes/origin/wip/hsyl20/dynflags': 'refs/remotes/origin/wip/hsyl20/dynflags/exception' exists; cannot create 'refs/remotes/origin/wip/hsyl20/dynflags' ! [new branch] wip/hsyl20/dynflags -> origin/wip/hsyl20/dynflags (unable to update local ref) * [new branch] wip/hsyl20/uncpp -> origin/wip/hsyl20/uncpp Unable to fetch in submodule path 'utils/haddock'; trying to directly fetch 4f9088e4b04e52ca510b55a78048c9230537e449: Submodule path 'utils/haddock': checked out '4f9088e4b04e52ca510b55a78048c9230537e449' simonpj@MSRC-3645512:~/code/HEAD-7$

3 2

Ampere A1 instances for builder/marge?
by Karel Gardas 27 May '21

27 May '21

Hi, just to pass https://www.servethehome.com/oracle-cloud-giving-away-ampere-arm-a1-instanc… -- oracle seems to be giving some "always free" Ampere/ARM based instances. Looks like they even have some RAM and cores so perhaps they may be usable for marge/builder for arm/linux platform? BTW: Not tested myself, not trust Oracle, but sometimes man can get some good even from ... well... Oracle... Thanks, Karel

1 0

Productivity Tip from mpickering
by Matthew Pickering 26 May '21

26 May '21

Hi, I am now using a new flavour combination which is proving *very* nice as a compromise between fast recompiles and passing tests. My normal build command is now: ./hadrian/build --flavour=default+no_profiled_libs+omit_pragmas --freeze1 -j This has the effect of * base libraries are compiled with optimisation * Profiling libraries are not built * Stage 1 compiler is compiled with -O + -fomit-interface-pragmas, so recompilation behaviour is much better. The end result is a nearly clean testsuite run (I think there are two failures) but much faster iterations when modifying `compiler/*`. Cheers, Matt

3 2

Coding style: Using StandaloneKindSignatures in GHC
by Baldur Blöndal 25 May '21

25 May '21

Discussion to permit use of StandaloneKindSignatures in the GHC coding style guide. I believe it increases the clarity of the code, especially as we move to fancier kinds. It is the only way we have for giving full signatures to type synonyms, type classes, type families and others. An example: type Cat :: Type -> Type type Cat ob = ob -> ob -> Type type Category :: forall ob. Cat ob -> Constraint class Category cat where id :: cat a a .. type Proxy :: forall k. k -> Type data Proxy a = Proxy type Some :: forall k. (k -> Type) -> Type data Some f where Some :: f ex -> Some f -- | The regular function type type (->) :: forall {rep1 :: RuntimeRep} {rep2 :: RuntimeRep}. TYPE1 rep1 -> TYPE rep2 -> Type type (->) = FUN 'Many This is in line with function definitions that are always given a top-level, standalone type signature (1) and not like we currently define type families/synonyms (2) by annotating each argument or not at all. Using -XStandaloneKindSignatures (3) matches (1) -- (1) curry :: ((a, b) -> c) -> (a -> b -> c) curry f x y = f (x, y) -- (2) type Curry (f :: (a, b) -> c) (x :: a) (y :: b) = f '(x, y) :: c -- (3) type Curry :: ((a, b) -> c) -> (a -> b -> c) type Curry f x y = f '(x, y) It covers an edgecase that `KindSignatures` don't. The only way for deriving to reference datatype arguments is if they are quantified by the declaration head -- `newtype Bin a ..`. StandaloneKindSignatures allows us to still provide a full signature. We could write `newtype Bin a :: Type -> Type` without it but not `newtype Bin :: Type -> Type -> Type` type Bin :: Type -> Type -> Type newtype Bin a b = Bin (a -> a -> b) deriving (Functor, Applicative) via (->) a `Compose` (->) a Let me know what you think

10 15