March 2021 - ghc-devs

The future of Phabricator
by Ben Gamari 18 Sep '21

18 Sep '21

TL;DR. For several reasons I think we should consider alternatives to Phabricator. My view is that GitLab seems like the best option. Hello everyone, Over the past year I have been growing increasingly weary of our continued dependence on Phabricator. Without a doubt, its code review interface is the best I have used. However, for a myriad of reasons I am recently questioning whether it is still the best tool for GHC's needs. # The problem There are a number of reasons why I am currently uncertain about Phabricator. For one, at this point we have no options for support in the event that something goes wrong as the company responsible for Phabricator, Phacility, has closed their support channels to non-paying customers. Furthermore, in the past year or two Phacility has been placing their development resources in the parts their customers pay them for, which appear to be much different that the parts that we actively use. For this reason, some parts that we rely on seem oddly half-finished. This concern was recently underscored by some rather unfortunate misfeatures in Harbormaster which resulted in broken CI after the Hadrian merge and now apparent bugs which have made it difficult to migrate to the CircleCI integration we previously prepared. Perhaps most importantly, in our recent development priorities survey our use of Phabricator was the most common complaint by a fair margin, both in case of respondents who have contributed patches and those who have not. On the whole, past contributors and potential future contributors alike have strongly indicated that they want a more Git-like experience. Of course, this wasn't terribly surprising; this is just the most recent case where contributors have made this preference known. Frankly, in a sense it is hard to blame them. The fact that users need to install a PHP tool, Arcanist, to contribute anything but documentation patches has always seemed like unnecessary friction to me and I would be quite happy to be rid of it. Indeed we have had a quite healthy number of GitHub documentation patches since we started accepting them. This makes me thing that there may indeed be potential contributoes that we are leaving on the table. # What to do With Rackspace support ending at the end of year, now may be a good time to consider whether we really want to continue on this road. Phabricator is great at code review but I am less and less certain that it is worth the maintenance uncertainty and potential lost contributors that it costs. Moreover, good alternatives seem closer at-hand than they were when we deployed Phabricator. ## Move to GitHub When people complain about our infrastructure, they often use GitHub as the example of what they would like to see. However, realistically I have a hard time seeing GitHub as a viable option. Its feature set is simply insufficient enough to handle the needs of a larger project like GHC without significant external tooling (as seen in the case of Rust-lang). The concrete reasons have been well-documented in previous discussions but, to summarize, * its review functionality is extremely painful to use with larger patches * rebased patches lead to extreme confusion and lost review comments * it lacks support for pre-receive hooks, which serve as a last line of defense to prevent unintended submodule changes * its inability to deal with external tickets is problematic * there is essentially no possibility that we could eventually migrate GHC's tickets to GitHub's ticket tracker without considerable data loss (much less manage the ticket load that would result), meaning that we would forever be stuck with maintaining Trac. * on a personal note, its search functionality has often left me empty-handed On the whole, these issues seem pretty hard to surmount. ## Move to GitLab In using GitLab for another project over the past months I have been positively surprised by its quality. It handles rebased merge requests far better than GitHub, has functional search, and quite a usable review interface. Furthermore, upstream has been extremely responsive to suggestions for improvement [1]. Even out-of-the-box it seems to be flexible enough to accommodate our needs, supporting integration with external issue trackers, having reasonable release management features, and support for code owners to automate review triaging (replacing much of the functionality of Phabricator's Herald). Finally, other FOSS projects' [3] recent migrations from Phabrictor to GitLab have shown that GitLab-the-company is quite willing to offer help when needed. I took some time this weekend to try setting up a quick GHC instance [2] to play around with. Even after just a few hours of playing around I think the result is surprisingly usable. Out of curiosity I also played around with importing some tickets from Trac (building on Matt Pickering's Trac-to-Maniphest migration tool). With relatively little effort I was even able to get nearly all of our tickets (as of 9 months ago) imported while preserving ticket numbers (although there are naturally a few wrinkles that would need to be worked out). Naturally, I think we should treat the question of ticket tracker migration as an orthogonal one to code review, but it is good to know that this is possible. ## Continue with Phabricator Continuing with Phabricator is of course an option. Its review functionality is great and it has served us reasonably well. However, compared to GitLab and even GitHub of today its features seem less distinguished than they once did. Moreover, the prospect of having to maintain a largely stagnant product with no support strikes me as a slightly dangerous game to play. Working around the issues we have recently encountered has already cost a non-negligible amount of time. # The bottom line If it wasn't clear already, I think that we should strongly consider a move to GitLab. At this point it seems clear that it isn't going to vanish, has a steady pace of development, is featureful, and available. However, these are just my thoughts. What do you think? Cheers, - Ben [1] 11.4 will ship with a file tree view in the code review interface, which I reported (https://gitlab.com/gitlab-org/gitlab-ce/issues/46474) as being is one of the Phabricator features I missed the most during review [2] https://gitlab.home.smart-cactus.org/ghc/ghc/issues/14641 [3] The GNOME and freedesktop.org projects have recently migrated, the former from a hodge-podge of self-hosted services and the latter from Phabricator

17 40

New implementation for `ImpredicativeTypes`
by Alejandro Serrano Mena 08 Sep '21

08 Sep '21

Hi all, As I mentioned some time ago, we have been busy working on a new implementation of `ImpredicativeTypes` for GHC. I am very thankful to everybody who back then sent us examples of impredicativity which would be nice to support, as far as we know this branch supports all of them! :) If you want to try it, at https://gitlab.haskell.org/trupill/ghc/commit/a3f95a0fe0f647702fd7225fa719a… you can find the result of the pipeline, which includes builds for several platforms (click on the "Artifacts" button, the one which looks like a cloud, to get them). The code is being developed at https://gitlab.haskell.org/trupill/ghc. Any code should run *unchanged* except for some eta-expansion required for some specific usage patterns of higher-rank types. Please don't hesitate to ask any questions or clarifications about it. A merge request for tracking this can be found at https://gitlab.haskell.org/ghc/ghc/merge_requests/1659 Kind regards, Alejandro

8 23

Potential improvement in compacting GC
by Ömer Sinan Ağacan 02 Sep '21

02 Sep '21

Hello, I recently implemented the algorithm used by GHC's compacting GC in another project. The algorithm (after marking) makes two passes over the heap /generation. In GHC, these passes are implemented in [1] and in the next function. In my implementation I tried 3 ways of implementing these passes, one of which is the same as GHC's code, and benchmarked each version. I found that the fastest implementation is not what's used in GHC, but it could easily be used. I should mention that my code targets Wasm, and I benchmarked Wasm instructions executed. Not CPU cycles, CPU instructions, or anything like that. It's very likely that the results will be different when benchmarking code running on actual hardware. Secondly, in my case the heap is mostly dead (residency is low). In GHC, compaction for the oldest generation is enabled when residency passes a threshold, so the assumption is the heap is mostly live. I'm guessing this should also make some difference. Anyway, the first implementation I tried was similar to GHC's scan, but I did scan += object_size(scan); instead of bumping scan by one, as GHC does in [2]. This was the slowest version. Second implementation did the same as GHC (bumped scan by one). This was faster, but still slower than the next version. What I found to be the best is scanning the bitmap, not the heap. The bitmap iterator reads one word at a time. In each iteration it checks if the bitmap word is 0. In GHC, in the best case this can skip 16 words on heap on 32-bit systems, and 32 words on 64-bit. Reminder: we need two bits per object in the bitmap, see [3]. (this is not the case in my implementation so the payoff is better) When the bitmap word is not 0 I use "count trailing zeros" (or "count leading zeros" depending on the bitmap implementation) to get the number of words to skip. This is a single instruction on Wasm and x86 (TZCNT or LZCNT, available via __builtin_ctzl and __builtin_clzl in gcc). So instead of skipping one word at a time, this can potentially skip 16 words (or 32 on 64-bit architectures). When that's not possible, it can still skip multiple words by using ctz/clz. Ömer [1]: https://github.com/ghc/ghc/blob/bd877edd9499a351db947cd51ed583872b2facdf/rt… [2]: https://github.com/ghc/ghc/blob/bd877edd9499a351db947cd51ed583872b2facdf/rt… [3]: https://github.com/ghc/ghc/blob/bd877edd9499a351db947cd51ed583872b2facdf/rt…

5 6

Why TcLclEnv and DsGblEnv need to store the same IORef for errors?
by Alfredo Di Napoli 01 Apr '21

01 Apr '21

Hello folks, as some of you might know me and Richard are reworking how GHC constructs, emits and deals with errors and warnings (See https://gitlab.haskell.org/ghc/ghc/-/wikis/Errors-as-(structured)-values and #18516). To summarise very briefly the spirit, we will have (eventually) proper domain-specific types instead of SDocs. The idea is to have very precise and "focused" types for the different phases of the compilation pipeline, and a "catch-all" monomorphic `GhcMessage` type used for the final pretty-printing and exception-throwing: data GhcMessage where GhcPsMessage :: PsMessage -> GhcMessage GhcTcRnMessage :: TcRnMessage -> GhcMessage GhcDsMessage :: DsMessage -> GhcMessage GhcDriverMessage :: DriverMessage -> GhcMessage GhcUnknownMessage :: forall a. (Diagnostic a, Typeable a) => a -> GhcMessage While starting to refactor GHC to use these types, I have stepped into something bizarre: the `DsGblEnv` and `TcLclEnv` envs both share the same `IORef` to store the diagnostics (i.e. warnings and errors) accumulated during compilation. More specifically, a function like `GHC.HsToCore.Monad.mkDsEnvsFromTcGbl` simply receives as input the `IORef` coming straight from the `TcLclEnv`, and stores it into the `DsGblEnv`. This is unfortunate, because it would force me to change the type of this `IORef` to be `IORef (Messages GhcMessage)` to accommodate both diagnostic types, but this would bubble up into top-level functions like `initTc`, which would now return a `Messages GhcMessage`. This is once again unfortunate, because is "premature": ideally it might still be nice to return `Messages TcRnMessage`, so that GHC API users could get a very precise diagnostic type rather than the bag `GhcMessage` is. It also violates an implicit contract: we are saying that `initTc` might return (potentially) *any* GHC diagnostic message (including, for example, driver errors/warnings), which I think is misleading. Having said all of that, it's also possible that returning `Messages GhcMessage` is totally fine here and we don't need to be able to do this fine-grained distinction for the GHC API functions. Regardless, I would like to ask the audience: * Why `TcLclEnv` and `DsGblEnv` need to share the same IORef? * Is this for efficiency reasons? * Is this because we need the two monads to independently accumulate errors into the same IORef? Thanks! Alfredo

4 10

Type inference of singular matches on GADTs
by Alexis King 31 Mar '21

31 Mar '21

Hi all, Today I was writing some code that uses a GADT to represent heterogeneous lists: data HList as where HNil :: HList '[] HCons :: a -> HList as -> HList (a ': as) This type is used to provide a generic way to manipulate n-ary functions. Naturally, I have some functions that accept these n-ary functions as arguments, which have types like this: foo :: Blah as => (HList as -> Widget) -> Whatsit The idea is that Blah does some type-level induction on as and supplies the function with some appropriate values. Correspondingly, my use sites look something like this: bar = foo (\HNil -> ...) Much to my dismay, I quickly discovered that GHC finds these expressions quite unfashionable, and it invariably insults them: • Ambiguous type variable ‘as0’ arising from a use of ‘foo’ prevents the constraint ‘(Blah as0)’ from being solved. The miscommunication is simple enough. I expected that when given an expression like \HNil -> ... GHC would see a single pattern of type HList '[] and consequently infer a type like HList '[] -> ... Alas, it was not to be. It seems GHC is reluctant to commit to the choice of '[] for as, lest perhaps I add another case to my function in the future. Indeed, if I were to do that, the choice of '[] would be premature, as as ~ '[] would only be available within one branch. However, I do not in fact have any such intention, which makes me quietly wish GHC would get over its anxiety and learn to be a bit more of a risk-taker. I ended up taking a look at the OutsideIn(X) paper, hoping to find some commentary on this situation, but in spite of the nice examples toward the start about the trickiness of GADTs, I found no discussion of this specific scenario: a function with exactly one branch and an utterly unambiguous pattern. Most examples come at the problem from precisely the opposite direction, trying to tease out a principle type from a collection of branches. The case of a function (or perhaps more accurately, a case expression) with only a single branch does not seem to be given any special attention. Of course, fewer special cases is always nice. I have great sympathy for generality. Still, I can’t help but feel a little unsatisfied here. Theoretically, there is no reason GHC cannot treat \(a `HCons` b `HCons` c `HCons` HNil) -> ... and \a b c -> ... almost identically, with a well-defined principle type and pleasant type inference properties, but there is no way for me to communicate this to the typechecker! So, my questions: 1. Have people considered this problem before? Is it discussed anywhere already? 2. Is my desire here reasonable, or is there some deep philosophical argument for why my program should be rejected? 3. If it /is/ reasonable, are there any obvious situations where a change targeted at what I’m describing (vague as that is) would affect programs negatively, not positively? I realize this gets rather at the heart of the typechecker, so I don’t intend to imply a change of this sort should be made frivolously. Indeed, I’m not even particularly attached to the idea that a change must be made! But I do want to understand the tradeoffs better, so any insight would be much appreciated. Thanks, Alexis

6 15

Multiple versions of happy
by Simon Peyton Jones 30 Mar '21

30 Mar '21

What's the approved mechanism to install multiple versions of happy/alex etc? Eg I tried to build ghc-9.0 and got this: checking for makeinfo... no checking for python3... /usr/bin/python3 checking for ghc-pkg matching /opt/ghc/bin/ghc... /opt/ghc/bin/ghc-pkg checking for happy... /home/simonpj/.cabal/bin/happy checking for version of happy... 1.20.0 configure: error: Happy version 1.19 is required to compile GHC. I so I have to 1. Install happy-1.19 without overwriting the installed happy-1.20 2. Tell configure to use happy-1.19 What's the best way to do those two things? Thanks Simon

3 3

Options for targeting Windows XP?
by Clinton Mead 30 Mar '21

30 Mar '21

I'm currently trying to bring my company around to using a bit of Haskell. One issue is that a number of our clients are based in South East Asia and need software that runs on Windows XP. Unfortunately it seems the last version of GHC that produces executables that run on Windows XP is GHC 7.10. Whilst this table <https://gitlab.haskell.org/ghc/ghc/-/wikis/platforms/windows> suggests the issue may only running GHC 8.0+ on Windows XP, I've confirmed that GHC 8.0 executables (even "Hello World") will not run on Windows XP, presumably because a non-XP WinAPI call in the runtime. My first thought would be to restrict myself to GHC 7.10 features (i.e. 2015). This would be a slight annoyance but GHC 7.10 still presents a reasonable language. But my concern would be that increasingly I'll run into issues with libraries that use extensions post GHC 7.10, particularly libraries with large dependency lists. So there's a few options I've considered at this point: 1. Use GHCJS to compile to Javascript, and then dig out a version of NodeJS that runs on Windows XP. GHCJS seems to at least have a compiler based on GHC 8.6. 2. Patch GHC with an additional command line argument to produce XP/Vista compatible executables, perhaps by looking at the changes between 7.10 -> 8.0, and re-introducing the XP approach as an option. The issue with 1 is that is that as well as being limited by how up to date GHCJS is, this will increase install size, memory usage and decrease performance on Windows XP machines, which are often in our environments quite old and resource and memory constrained. Approach 2 is something I'd be willing to put some work into if it was practical, but my thought is that XP support was removed for a reason, presumably by using newer WinAPI functions simplified things significantly. By re-adding in XP support I'd be complicating GHC once again, and GHC will effectively have to maintain two approaches. In addition, in the long term, whenever a new WinAPI call is added one would now have to check whether it's available in Windows XP, and if it's not produce a Windows XP equivalent. That might seem like just an extra burden of support for already busy GHC developers. But on the other hand, if the GHC devs would be happy to merge a patch and keep up XP support this would be the cleanest option. But then I had a thought. If GHC Core isn't supposed to change much between versions is it? Which made me come up with these approaches: 3. Hack up a script to compile programs using GHC 9 to Core, then feed that Core output into GHC 7.10. OR 4. Produce a chimera style GHC by importing the GHC 9.0 API and the GHC 7.10 API, and making a version of GHC that does Haskell -> Core in GHC 9.0 and the rest of the code generation in GHC 7.10. One issue with 4 will be that presumably that because I'm importing GHC 9.0 API and the 7.10 API separately, all their data types will technically be separate, so I'll need to basically deep copy the GHC 9.0 core datatype (and perhaps others) to GHC 7.10 datatypes. But presuming their largely similar this should be fairly mechanical. So are any of these approaches (well, particularly 2 and 4) reasonable? Or am I going to run into big problems with either of them? Is there another approach I haven't thought of? Thanks, Clinton

8 14

GHC 9.2 has branched
by Ben Gamari 29 Mar '21

29 Mar '21

Hello everyone, At this point the ghc-9.2 branch has officially branched off from master. If there was anything you were holding back from `master`, feel free to now send it off to Marge. I'll be working on doing some release prep and pushing out an alpha in the next few days. This alpha will very likely lack the new NCG (which is still being worked on by Moritz) but otherwise should be functionally complete. Cheers, - Ben

1 0

Pattern matching desugaring regression? Re: [Haskell-cafe] Why does my module take so long to compile?
by Carter Schonwald 28 Mar '21

28 Mar '21

Ccing ghc devs since that’s a better forum perhaps Crazy theory: this is a regression due the the partial changes to pattern matching coverage checking in 8.10 that finished / landed in ghc 9 Why: Desugaring is when pattern/ case statement translation happens I think? And the only obvious “big thing” is that you have some huge , albeit sane for a compiler, pattern matching I’d first check if the new ghc 9 release doesn’t have that regression in build time that you experienced. And if it does file a ticket. I may be totally wrong, but that seems like a decent likelihood ! On Mon, Feb 15, 2021 at 12:20 PM Troels Henriksen <athas(a)sigkill.dk> wrote: > Carter Schonwald <carter.schonwald(a)gmail.com> writes: > > > How big are your data types in the file In question? Do you ghc generics > or > > deriving or template Haskell? Could you share a link to the file in > > question ? > > The file does not define any large data types itself, but it operates on > some fairly large data types (an AST, it's a compiler). But so do many > other modules that work just fine. It uses no generics, nontrivial > deriving, or Template Haskell. It's this file: > > https://github.com/diku-dk/futhark/blob/master/src/Futhark/Pass/ExtractKern… > > I also found a handful of other modules in my project that are > significantly slower to compile in GHC 8.10, and seemingly also because > of the desugarer, but none of them have any obvious smoking guns like > generics or TH. > > The only commonality I can find is that the affected modules contain > functions with a relatively large typeclass context. I use > ConstraintKinds to make them more concise, and I guess unfolded there > may be 30-40 equality/class constraints in total. Like this: > > type DistLore lore = > ( Bindable lore, > HasSegOp lore, > BinderOps lore, > LetDec lore ~ Type, > ExpDec lore ~ (), > BodyDec lore ~ () > ) > > where the 'Bindable' constraint in particular then implies a further > handful of "~" constraints: > > class > ( ASTLore lore, > FParamInfo lore ~ DeclType, > LParamInfo lore ~ Type, > RetType lore ~ DeclExtType, > BranchType lore ~ ExtType, > SetType (LetDec lore) > ) => > Bindable lore > where > ... > > FParamInfo/LParamInfo/etc are type families. Are such constraints > particularly slow? > > -- > \ Troels > /\ Henriksen >

4 7

config.sub
by Simon Peyton Jones 26 Mar '21

26 Mar '21

Folks I'm getting a lot of this simonpj@MSRC-3645512:~/code/HEAD-3$ git status On branch wip/T19569 Your branch is up to date with 'origin/wip/T19569'. Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) (commit or discard the untracked or modified content in submodules) modified: libraries/unix (modified content) Untracked files: (use "git add <file>..." to include in what will be committed) libraries/base/config.sub libraries/ghc-bignum/config.sub What has changed in unix? Answer: simonpj@MSRC-3645512:~/code/HEAD-3$ cd libraries/unix simonpj@MSRC-3645512:~/code/HEAD-3/libraries/unix$ git status HEAD detached at 21437f2 Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) modified: config.sub Ugh. Why is config.sub modified if it's a repo file? And should I ignore the untracked base/config.sub and ghc-bignum? What am I doing wrong? Thanks Simon

2 1