January 2020 - ghc-devs

The future of Phabricator
by Ben Gamari 18 Sep '21

18 Sep '21

TL;DR. For several reasons I think we should consider alternatives to Phabricator. My view is that GitLab seems like the best option. Hello everyone, Over the past year I have been growing increasingly weary of our continued dependence on Phabricator. Without a doubt, its code review interface is the best I have used. However, for a myriad of reasons I am recently questioning whether it is still the best tool for GHC's needs. # The problem There are a number of reasons why I am currently uncertain about Phabricator. For one, at this point we have no options for support in the event that something goes wrong as the company responsible for Phabricator, Phacility, has closed their support channels to non-paying customers. Furthermore, in the past year or two Phacility has been placing their development resources in the parts their customers pay them for, which appear to be much different that the parts that we actively use. For this reason, some parts that we rely on seem oddly half-finished. This concern was recently underscored by some rather unfortunate misfeatures in Harbormaster which resulted in broken CI after the Hadrian merge and now apparent bugs which have made it difficult to migrate to the CircleCI integration we previously prepared. Perhaps most importantly, in our recent development priorities survey our use of Phabricator was the most common complaint by a fair margin, both in case of respondents who have contributed patches and those who have not. On the whole, past contributors and potential future contributors alike have strongly indicated that they want a more Git-like experience. Of course, this wasn't terribly surprising; this is just the most recent case where contributors have made this preference known. Frankly, in a sense it is hard to blame them. The fact that users need to install a PHP tool, Arcanist, to contribute anything but documentation patches has always seemed like unnecessary friction to me and I would be quite happy to be rid of it. Indeed we have had a quite healthy number of GitHub documentation patches since we started accepting them. This makes me thing that there may indeed be potential contributoes that we are leaving on the table. # What to do With Rackspace support ending at the end of year, now may be a good time to consider whether we really want to continue on this road. Phabricator is great at code review but I am less and less certain that it is worth the maintenance uncertainty and potential lost contributors that it costs. Moreover, good alternatives seem closer at-hand than they were when we deployed Phabricator. ## Move to GitHub When people complain about our infrastructure, they often use GitHub as the example of what they would like to see. However, realistically I have a hard time seeing GitHub as a viable option. Its feature set is simply insufficient enough to handle the needs of a larger project like GHC without significant external tooling (as seen in the case of Rust-lang). The concrete reasons have been well-documented in previous discussions but, to summarize, * its review functionality is extremely painful to use with larger patches * rebased patches lead to extreme confusion and lost review comments * it lacks support for pre-receive hooks, which serve as a last line of defense to prevent unintended submodule changes * its inability to deal with external tickets is problematic * there is essentially no possibility that we could eventually migrate GHC's tickets to GitHub's ticket tracker without considerable data loss (much less manage the ticket load that would result), meaning that we would forever be stuck with maintaining Trac. * on a personal note, its search functionality has often left me empty-handed On the whole, these issues seem pretty hard to surmount. ## Move to GitLab In using GitLab for another project over the past months I have been positively surprised by its quality. It handles rebased merge requests far better than GitHub, has functional search, and quite a usable review interface. Furthermore, upstream has been extremely responsive to suggestions for improvement [1]. Even out-of-the-box it seems to be flexible enough to accommodate our needs, supporting integration with external issue trackers, having reasonable release management features, and support for code owners to automate review triaging (replacing much of the functionality of Phabricator's Herald). Finally, other FOSS projects' [3] recent migrations from Phabrictor to GitLab have shown that GitLab-the-company is quite willing to offer help when needed. I took some time this weekend to try setting up a quick GHC instance [2] to play around with. Even after just a few hours of playing around I think the result is surprisingly usable. Out of curiosity I also played around with importing some tickets from Trac (building on Matt Pickering's Trac-to-Maniphest migration tool). With relatively little effort I was even able to get nearly all of our tickets (as of 9 months ago) imported while preserving ticket numbers (although there are naturally a few wrinkles that would need to be worked out). Naturally, I think we should treat the question of ticket tracker migration as an orthogonal one to code review, but it is good to know that this is possible. ## Continue with Phabricator Continuing with Phabricator is of course an option. Its review functionality is great and it has served us reasonably well. However, compared to GitLab and even GitHub of today its features seem less distinguished than they once did. Moreover, the prospect of having to maintain a largely stagnant product with no support strikes me as a slightly dangerous game to play. Working around the issues we have recently encountered has already cost a non-negligible amount of time. # The bottom line If it wasn't clear already, I think that we should strongly consider a move to GitLab. At this point it seems clear that it isn't going to vanish, has a steady pace of development, is featureful, and available. However, these are just my thoughts. What do you think? Cheers, - Ben [1] 11.4 will ship with a file tree view in the code review interface, which I reported (https://gitlab.com/gitlab-org/gitlab-ce/issues/46474) as being is one of the Phabricator features I missed the most during review [2] https://gitlab.home.smart-cactus.org/ghc/ghc/issues/14641 [3] The GNOME and freedesktop.org projects have recently migrated, the former from a hodge-podge of self-hosted services and the latter from Phabricator

17 40

New implementation for `ImpredicativeTypes`
by Alejandro Serrano Mena 08 Sep '21

08 Sep '21

Hi all, As I mentioned some time ago, we have been busy working on a new implementation of `ImpredicativeTypes` for GHC. I am very thankful to everybody who back then sent us examples of impredicativity which would be nice to support, as far as we know this branch supports all of them! :) If you want to try it, at https://gitlab.haskell.org/trupill/ghc/commit/a3f95a0fe0f647702fd7225fa719a… you can find the result of the pipeline, which includes builds for several platforms (click on the "Artifacts" button, the one which looks like a cloud, to get them). The code is being developed at https://gitlab.haskell.org/trupill/ghc. Any code should run *unchanged* except for some eta-expansion required for some specific usage patterns of higher-rank types. Please don't hesitate to ask any questions or clarifications about it. A merge request for tracking this can be found at https://gitlab.haskell.org/ghc/ghc/merge_requests/1659 Kind regards, Alejandro

8 23

Feasibility of native RTS support for continuations?
by Alexis King 03 Mar '20

03 Mar '20

Hi all, tl;dr: I want to try to implement native support for capturing slices of the RTS stack as a personal experiment; please tell me the obstacles I am likely to run into. Much more context follows. --- I have been working on an implementation of an algebraic effect system that uses unsafe primops to be as performant as possible. However, the unavoidable need to CPS the entire program balloons heap allocation. Consider the following definition: f a b = g a >>= \c -> h (b + c) Assume `g` and `h` are not inlined. If the monad used is IO, this will be compiled efficiently: the result of `g` is returned on the stack, and no closure needs to be allocated for the lambda. However, if the monad supports capturing the continuation, the above definition must be CPS’d. After inlining, we end up with f a b = \k -> let lvl = \c -> h (b + c) k in g a lvl which must allocate a closure on the heap. This is frustrating, as it must happen for every call to a non-inlined monadic operation, even if that operation never captures the continuation. In an algebraic effect system, there are many shortcuts that avoid the need to capture the continuation, and under my implementation, large swaths of code never do so. I’ve managed to exploit that to get some savings, but I can’t escape the need to allocate all these closures. This motivates my question: how difficult would it be to allow capturing a portion of the RTS call stack directly? My requirements are fairly minimal, as continuations go: 1. Capturing a continuation is only legal from within a strict state thread (i.e. IO or strict ST). 2. The continuation is captured up to a prompt, which would be a new kind of RTS stack frame. Prompts are not tagged, so there is only ever exactly one prompt active at any time (which may be the root prompt). 3. Capturing a continuation is unsafe. The behavior of capturing a continuation is undefined if the current prompt was not created by the current state thread (and it is never legal to capture up to the root prompt). 4. Applying a continuation is unsafe. Captured continuations return `Any`, and type safety is the caller’s obligation. 5. Continuations are “functional,” which is to say applying them does not trigger any additional stack unwinding. This minimal support for primitive continuation capturing would be enough to support my efficient, safe delimited control implementation. In my ignorant mind, implementing this ought to be as simple as defining two new primops, reset# :: (State# s -> (# State# s, a #)) -> State# s -> (# State# s, a #) shift# :: ((a -> State# s -> (# State# s, Any #)) -> State# s -> (# State# s, Any #)) -> State# s -> (# State# s, a #) where reset# pushes a new prompt frame and shift# captures a slice of the RTS stack up to that frame and copies it into the heap. Restoring a continuation would copy all the captured frames onto the end of the current stack. Sounds simple enough! I would like to experiment with implementing something like this myself, just to see if it would really work, but somehow I doubt it is actually as simple as it sounds. Minor complications are fine, but what worries me are major obstacles I haven’t found in my limited attempts to learn about the RTS. So far, I’ve read the old “The New GHC/Hugs Runtime System” paper, which still seems mostly accurate from a high level, though I imagine many details have changed since then. I’ve also read the Commentary/RTS/Storage/Stack wiki page, and I’ve peeked at some of the RTS source code. I’ve also skimmed a handful of other papers and wiki pages, but it’s hard to know what I’m looking for. Can anyone point me to better resources or give me some insight into what will be hard? Thanks in advance, Alexis

4 15

[ANNOUNCE] GHC 8.8.2 is now available
by Ben Gamari 25 Feb '20

25 Feb '20

Hello everyone, The GHC team is proud to announce the release of GHC 8.8.2. The source distribution, binary distributions, and documentation are available at https://downloads.haskell.org/~ghc/8.8.2 Release notes are also available [1]. This release fixes a handful of issues affecting 8.8.1: - A bug (#17088) in the compacting garbage collector resulting in segmentations faults under specific circumstances. Note that this may affect user programs even if they did not explicitly request the compacting GC (using the -c RTS flag) since GHC may fallback to compacting collection during times of high memory pressure. - A code generation bug (#17334) resulting in GHC panics has been fixed. - A bug in the `process` library causing builds using `hsc2hs` to fail non-deterministically on Windows has been fixed (Trac #17480) - A typechecker bug (#12088) resulting in programs being unexpectedly rejected has been fixed. - A bug in the implementation of compact normal forms resulting in segmentation faults in some uses (#17044) has been fixed. - A bug causing GHC to incorrectly complain about incompatible LLVM versions when using LLVM 7.0.1 has been fixed (#16990). As always, if anything looks amiss do let us know. Happy compiling! Cheers, - Ben [1] https://downloads.haskell.org/ghc/8.8.2/docs/html/users_guide/8.8.2-notes.h…

2 4

[ANNOUNCE] Glasgow Haskell Compiler 8.10.1-rc1 released
by Ben Gamari 30 Jan '20

30 Jan '20

Hello all, The GHC team is happy to announce the availability of the first release candidate of GHC 8.10.1. Source and binary distributions are available at the usual place: https://downloads.haskell.org/ghc/8.10.1-rc1/ GHC 8.10.1 will bring a number of new features including: * The new UnliftedNewtypes extension allowing newtypes around unlifted types. * The new StandaloneKindSignatures extension allows users to give top-level kind signatures to type, type family, and class declarations. * A new warning, -Wderiving-defaults, to draw attention to ambiguous deriving clauses * A number of improvements in code generation, including changes * A new GHCi command, :instances, for listing the class instances available for a type. * An upgraded Windows toolchain lifting the MAX_PATH limitation * A new, low-latency garbage collector. * Improved support profiling, including support for sending profiler samples to the eventlog, allowing correlation between the profile and other program events This is the first and likely final release candidate. For a variety of reasons, it comes a few weeks later than the original schedule of release late December. However, besides a few core libraries book-keeping issues this candidate is believed to be in good condition for the final release. As such, the final 8.10.1 release will likely come in two weeks. Note that at the moment we still require that macOS Catalina users exempt the binary distribution from the notarization requirement by running `xattr -cr .` on the unpacked tree before running `make install`. In addition, we are still looking for any Alpine Linux to help diagnose the correctness issues in the Alpine binary distribution [1]. If you use Alpine any you can offer here would be greatly appreciated. Please do test this release and let us know if you encounter any other issues. Cheers, - Ben [1] https://gitlab.haskell.org/ghc/ghc/issues/17508 [2] https://gitlab.haskell.org/ghc/ghc/issues/17418

5 4

more submodule questions
by Richard Eisenberg 30 Jan '20

30 Jan '20

Hi devs, I recently found this text at the end of https://gitlab.haskell.org/ghc/ghc/wikis/working-conventions/git/submodules <https://gitlab.haskell.org/ghc/ghc/wikis/working-conventions/git/submodules> : --- The CI pipeline of ghc/ghc> <applewebdata://9E3625B8-2065-48C6-A8FA-16817EA6E642/ghc/ghc> includes a linting step to ensure that all submodules refer only to "persistent" commits of the upstream repositories (e.g. not wip/ branches, which may disappear in the future). Specifically, the linter checks that any submodules refer to commits that are reachable by at least one branch that doesn't begin with the prefix wip/. Consequently, you must ensure that any submodule changes introduced in a ghc/ghc> <applewebdata://9E3625B8-2065-48C6-A8FA-16817EA6E642/ghc/ghc> merge request are merged upstream before the merge request is added to the merge queue. --- I don't understand what this means. - By citing "ghc/ghc>", does this mean that the linter only checks for this on branches of the ghc/ghc repo? If I have a fork (e.g. rae/ghc), are these checks disabled? - Does this linter stop CI from progressing to, say, running the testsuite? If so, then how can we run the testsuite via CI if we have any submodule changes? We want to run the testsuite while the work is still in progress. - By "you must ensure ... before the merge request is added to the merge queue": this makes me wonder whether the linter is just a warning or an error. That is, if I must ensure it, then it suggests that CI is not ensuring it. Sorry to be dense here! Thanks, Richard

2 3

stage2 build fails
by Simon Peyton Jones 28 Jan '20

28 Jan '20

I'm getting this with "sh validate -legacy" compiler/main/DynFlags.hs:1344:15: error: [-Woverlapping-patterns, -Werror=overlapping-patterns] Pattern match is redundant In an equation for 'settings': settings s | otherwise = ... | 1344 | | otherwise = panic $ "Invalid cfg parameters." ++ exampleString | ^^^^^^^^^ This is when compiling the stage-2 compiler. There's an ifdef in DynFlags thus #if __GLASGOW_HASKELL__ <= 810 | otherwise = panic $ "Invalid cfg parameters." ++ exampleString #endif but somehow it's not triggering for the stage2 compiler. Any ideas? It's blocking a full build. This #ifdef was added in 8038cbd96f4, when GHC became better at reporting redundant code. Simon

4 12

Intree gmp builds using Hadrian use system gmp headers
by Carter Schonwald 27 Jan '20

27 Jan '20

Hey all, For those using Hadrian for platforms where intree gmp config is important, some sleuthing I’ve done over the past half seems to indicate that currently it’s not correct via Hadrian, but only via make. https://gitlab.haskell.org/ghc/ghc/issues/17756 I’m able to reproduce this on multiple OS X machines (though it took me a while to read apart that is Hadrian specific.) Punchline : at the moment, Hadrian intree gmp builds succeed because gmp headers in default include search paths of gcc/clang get picked up. In a properly isolated environment / one without the offending header leakage, I’m able at least on OS X to get Hadrian with intree gmp to consistently fail.

1 1

gaining access to the `ghc` group
by Richard Eisenberg 27 Jan '20

27 Jan '20

Hi devs, I'm onboarding a new contributor (Gert-Jan Bottu), whose patch (!2465) makes commensurate changes in Haddock. In order to use CI, then, he needs to be able to push a wip/ branch to our fork of Haddock. In order to do that, he needs to be in the `ghc` group. (I'm assuming -- but have not checked -- that just forking Haddock on the gitlab.haskell.org <http://gitlab.haskell.org/> instance is not enough. The wiki page on submodules (https://gitlab.haskell.org/ghc/ghc/wikis/working-conventions/git/submodules <https://gitlab.haskell.org/ghc/ghc/wikis/working-conventions/git/submodules>) suggests it is not.) That same wiki page says the one needs merely ask to join the `ghc` group. Good. So I follow the "ask" link to get to https://gitlab.haskell.org/ghc/ghc/wikis/mailing-lists-and-irc#mailing-list… <https://gitlab.haskell.org/ghc/ghc/wikis/mailing-lists-and-irc#mailing-list…> . That page helpfully describes where we can be reached, but it's not all that helpful for someone who wants to join the `ghc` group. According to that page, newcomers have to post publicly in either ghc-devs (I have never seen such a request there) or in #ghc. If I'm new in town, I wouldn't feel all that happy doing so, likely wanting to wait until I actually had a patch accepted before requesting rights.... but of course I need access to CI in order to get a patch accepted. Instead, would it be possible to have some sort of ghc-admin list, perhaps? While (I think) I know the individuals to contact for a request like this, an official mailing list would make this more transparent and, in my opinion, easier to onboard new contributors. I understand if folks don't want yet another mailing list, but then is here some other approach here that doesn't require posting in public? Thanks! Richard

2 2

Anyone able to build current master on OS X with GCC?
by Carter Schonwald 25 Jan '20

25 Jan '20

I’m finding myself only able to successfully run a full validate build if I set the build configuration to use clang rather than gcc. Otherwise I get a cpp error that makes it look like some #defines are missing in the Gmp binding codes. Anyone able to do a gcc flavored Mac build that survives validate recently ?

1 0