May 2020 - ghc-devs - Haskell.org

The future of Phabricator
by Ben Gamari 18 Sep '21

18 Sep '21

TL;DR. For several reasons I think we should consider alternatives to Phabricator. My view is that GitLab seems like the best option. Hello everyone, Over the past year I have been growing increasingly weary of our continued dependence on Phabricator. Without a doubt, its code review interface is the best I have used. However, for a myriad of reasons I am recently questioning whether it is still the best tool for GHC's needs. # The problem There are a number of reasons why I am currently uncertain about Phabricator. For one, at this point we have no options for support in the event that something goes wrong as the company responsible for Phabricator, Phacility, has closed their support channels to non-paying customers. Furthermore, in the past year or two Phacility has been placing their development resources in the parts their customers pay them for, which appear to be much different that the parts that we actively use. For this reason, some parts that we rely on seem oddly half-finished. This concern was recently underscored by some rather unfortunate misfeatures in Harbormaster which resulted in broken CI after the Hadrian merge and now apparent bugs which have made it difficult to migrate to the CircleCI integration we previously prepared. Perhaps most importantly, in our recent development priorities survey our use of Phabricator was the most common complaint by a fair margin, both in case of respondents who have contributed patches and those who have not. On the whole, past contributors and potential future contributors alike have strongly indicated that they want a more Git-like experience. Of course, this wasn't terribly surprising; this is just the most recent case where contributors have made this preference known. Frankly, in a sense it is hard to blame them. The fact that users need to install a PHP tool, Arcanist, to contribute anything but documentation patches has always seemed like unnecessary friction to me and I would be quite happy to be rid of it. Indeed we have had a quite healthy number of GitHub documentation patches since we started accepting them. This makes me thing that there may indeed be potential contributoes that we are leaving on the table. # What to do With Rackspace support ending at the end of year, now may be a good time to consider whether we really want to continue on this road. Phabricator is great at code review but I am less and less certain that it is worth the maintenance uncertainty and potential lost contributors that it costs. Moreover, good alternatives seem closer at-hand than they were when we deployed Phabricator. ## Move to GitHub When people complain about our infrastructure, they often use GitHub as the example of what they would like to see. However, realistically I have a hard time seeing GitHub as a viable option. Its feature set is simply insufficient enough to handle the needs of a larger project like GHC without significant external tooling (as seen in the case of Rust-lang). The concrete reasons have been well-documented in previous discussions but, to summarize, * its review functionality is extremely painful to use with larger patches * rebased patches lead to extreme confusion and lost review comments * it lacks support for pre-receive hooks, which serve as a last line of defense to prevent unintended submodule changes * its inability to deal with external tickets is problematic * there is essentially no possibility that we could eventually migrate GHC's tickets to GitHub's ticket tracker without considerable data loss (much less manage the ticket load that would result), meaning that we would forever be stuck with maintaining Trac. * on a personal note, its search functionality has often left me empty-handed On the whole, these issues seem pretty hard to surmount. ## Move to GitLab In using GitLab for another project over the past months I have been positively surprised by its quality. It handles rebased merge requests far better than GitHub, has functional search, and quite a usable review interface. Furthermore, upstream has been extremely responsive to suggestions for improvement [1]. Even out-of-the-box it seems to be flexible enough to accommodate our needs, supporting integration with external issue trackers, having reasonable release management features, and support for code owners to automate review triaging (replacing much of the functionality of Phabricator's Herald). Finally, other FOSS projects' [3] recent migrations from Phabrictor to GitLab have shown that GitLab-the-company is quite willing to offer help when needed. I took some time this weekend to try setting up a quick GHC instance [2] to play around with. Even after just a few hours of playing around I think the result is surprisingly usable. Out of curiosity I also played around with importing some tickets from Trac (building on Matt Pickering's Trac-to-Maniphest migration tool). With relatively little effort I was even able to get nearly all of our tickets (as of 9 months ago) imported while preserving ticket numbers (although there are naturally a few wrinkles that would need to be worked out). Naturally, I think we should treat the question of ticket tracker migration as an orthogonal one to code review, but it is good to know that this is possible. ## Continue with Phabricator Continuing with Phabricator is of course an option. Its review functionality is great and it has served us reasonably well. However, compared to GitLab and even GitHub of today its features seem less distinguished than they once did. Moreover, the prospect of having to maintain a largely stagnant product with no support strikes me as a slightly dangerous game to play. Working around the issues we have recently encountered has already cost a non-negligible amount of time. # The bottom line If it wasn't clear already, I think that we should strongly consider a move to GitLab. At this point it seems clear that it isn't going to vanish, has a steady pace of development, is featureful, and available. However, these are just my thoughts. What do you think? Cheers, - Ben [1] 11.4 will ship with a file tree view in the code review interface, which I reported (https://gitlab.com/gitlab-org/gitlab-ce/issues/46474) as being is one of the Phabricator features I missed the most during review [2] https://gitlab.home.smart-cactus.org/ghc/ghc/issues/14641 [3] The GNOME and freedesktop.org projects have recently migrated, the former from a hodge-podge of self-hosted services and the latter from Phabricator

17 40

New implementation for `ImpredicativeTypes`
by Alejandro Serrano Mena 08 Sep '21

08 Sep '21

Hi all, As I mentioned some time ago, we have been busy working on a new implementation of `ImpredicativeTypes` for GHC. I am very thankful to everybody who back then sent us examples of impredicativity which would be nice to support, as far as we know this branch supports all of them! :) If you want to try it, at https://gitlab.haskell.org/trupill/ghc/commit/a3f95a0fe0f647702fd7225fa719a… you can find the result of the pipeline, which includes builds for several platforms (click on the "Artifacts" button, the one which looks like a cloud, to get them). The code is being developed at https://gitlab.haskell.org/trupill/ghc. Any code should run *unchanged* except for some eta-expansion required for some specific usage patterns of higher-rank types. Please don't hesitate to ask any questions or clarifications about it. A merge request for tracking this can be found at https://gitlab.haskell.org/ghc/ghc/merge_requests/1659 Kind regards, Alejandro

8 23

[hadrian] Use LLVM for test and validate target
by Stefan Schulze Frielinghaus 30 May '20

30 May '20

Hi all, Is it possible to force Hadrian to use the LLVM code generator for the test and validate targets? For example, while using the following commands it seems to me that only the native code generator (if available) is used: $ ./hadrian/build -c -j --flavour=quick-llvm test $ ./hadrian/build -c -j --flavour=validate Cheers, Stefan

2 1

STG/CMM questions
by Daneel Yaitskov 29 May '20

29 May '20

Hi, There are 2 questions: 1. How to decipher myThreadId implementation in STG? What is the principle behind STG version? Is it matching first field from TSO by type? "id" is not mentioned anyhow and there is no offset to id field. The whole TSO struct is just mentioned. emitAssign and CmmLocal are used everywhere, so they don't express logic particular to myThreadId function. was: return tso->id; now: currentTSOExpr = CmmReg currentTSOReg MyThreadIdOp -> \[] -> opAllDone $ \[res] -> do emitAssign (CmmLocal res) currentTSOExpr 2. Cmm macro for new TSO field is not found. I added to deriveConstants/Main.hs: ,closureField C "StgTSO" "trace_id" TSO.h: typedef struct StgTSO_ { ... StgClosure* trace_id; PrimsOps.cmm: gcptr p; p = StgTSO_trace_id(CurrentTSO); Linking error happens after make clean && ./boot && ./configure && make -j8 so I guess macros is not defined. Looks like such macros are not written to any file and it is hard to grep them. collect2: error: ld returned 1 exit status `cc' failed in phase `Linker'. (Exit code: 1) utils/iserv/ghc.mk:101: recipe for target 'utils/iserv/stage2/build/tmp/ghc-iserv' failed make[1]: *** [utils/iserv/stage2/build/tmp/ghc-iserv] Error 1 /home/dan/demo/haskell/compiler/ghc/rts/dist/build/libHSrts_thr_p.a(PrimOps.thr_p_o):function stg_getAdamTraceIdzh: error: undefined reference to 'StgTSO_trace_id' /home/dan/demo/haskell/compiler/ghc/rts/dist/build/libHSrts_thr_p.a(PrimOps.thr_p_o):function stg_getAdamTraceIdzh: error: undefined reference to 'n' collect2: error: ld returned 1 exit status `cc' failed in phase `Linker'. (Exit code: 1) utils/iserv/ghc.mk:105: recipe for target 'utils/iserv/stage2_p/build/tmp/ghc-iserv-prof' failed make[1]: *** [utils/iserv/stage2_p/build/tmp/ghc-iserv-prof] Error 1 Makefile:123: recipe for target 'all' failed -- Best regards, Daniil Iaitskov

2 1

Re: keepAlive# primop
by Ben Gamari 27 May '20

27 May '20

Ccing ghc-devs@ since this discussion is something of general interest to the community. Sylvain Henry <sylvain(a)haskus.fr> writes: > Simon, Ben, > > I've been reading and thinking about `readRW#` issues which are very > related to issues we have with `keepAlive#` primop. > > To recap, the problem is that we want some transformations (that Simon > has listed in [1]) to consider: > > ``` > case runRW# f of ... > > case keepAlive# k a of ... > ``` > > as if they were really: > > ``` > case f realWorld# of ... > > case a of ... > ``` > > BUT without breaking the semantics of runRW# and keepAlive#. > > I have been thinking about a solution that I have described on the wiki: > https://gitlab.haskell.org/ghc/ghc/-/wikis/proposal/with-combinator#option-… > > The idea is to keep a set of variable names in each Core case-expression > that are kept alive during the evaluation of the scrutinee. > > I think it would work very nicely with your `newState#` primop described > in [2], both for `runST` and for `unsafeDupablePerformIO` (details on > the wiki). > > It requires a little more upfront work to adapt the code involving > case-expressions. But it will force us to review all transformations to > check if they are sound when keep-alive sets are not empty, which we > would have to do anyway if we implemented another option. We could start > by disabling transformations involving non-empty keep-alive sets and > iterate to enable the sound ones. > > I would like your opinions on the approach. I may have totally missed > something. Thanks for writing this down! Indeed it is an interesting idea. However, as expressed on IRC, I wonder whether this problem rises to the level where it warrants an adaptation to our Core representation. It feels a bit like the tail is wagging the dog here, especially given how the "tail" here merely exists to support FFI. That being said, this is one of the few options which remain on the table that doesn't require changes to user code. Moreover, the applicability to runRW# is quite intriguing. Another (admittedly, more ad-hoc) option that would avoid modifying Core would be to teach the simplifier about the class of "continuation-passing" primops (e.g. `keepAlive#` and `runRW#`), allowing it to push case analyses into the continuation argument. That is, case keepAlive# x expr of pat -> rhs ~> keepAlive# x (case expr of pat -> rhs) Of course, doing this is a bit tricky since one must rewrite the application of keepAlive# to ensure that the resulting application is well-typed. Admittedly, this doesn't help the runRW# case (although this could presumably be accommodated by touch#'ing the final state token in the runRW# desugaring emitted by CorePrep). On the whole, I'm not a fan of this ad-hoc option. It increases the complexity of the simplifier all to support a single operation. By comparison, the Core extension looks somewhat appealing. Cheers, - Ben [1] https://gitlab.haskell.org/ghc/ghc/-/wikis/proposal/with-combinator#option-… P.S. A minor note: the keepAlive# "pseudo-instruction" mentioned on the Wiki [1] is precisely the touch# operation we have today.

3 2

in-tree API Annotations
by Alan & Kim Zimmerman 26 May '20

26 May '20

I am trying to get my head around the best way to put the API annotations into the GHC ParsedSource directly, so ghc-exactprint can move in to GHC and make the development/testing process easier. Prior to API Annotations, there were very few locations in the AST, as they were added as anchor points for the annotations, being separate from the AST and indexed to it by the SrcSpan. I am intending to strip out the superfluous ones, in particular all the (Located RdrName), replacing them with something like data ApiAnnName a = N ApiAnn a instead. So they would still be wrapped, but the annotations would be only ones that can occur for a RdrName, such a backticks, parens, and so on. Does anything actually need these locations now they are there? I am thinking specifically of the .hie files, as the `toHie` instances seem to make use of locations extensively. Regards Alan

1 1

8.12 plans
by Ben Gamari 24 May '20

24 May '20

Hi everyone, The time is again upon us to start thinking about release planning for the next major release: GHC 8.12. In keeping with our 6-month release schedule, I propose the following schedule: * Mid-June 2020: Aim to have all major features in the tree * Late-June 2020: Cut the ghc-8.12 branch * June - August 2020: 3 alpha releases * 1 September 2020: beta release * 25 September 2020: Final 8.12.1 release So, if you have any major features which you would like to merge for 8.12, now is the time to start planning how to wrap them up in the next month or so. As always, do let me know if you think this may be problematic and we can discuss options. Cheers, - Ben

6 7

profiling a constructor?
by Richard Eisenberg 24 May '20

24 May '20

Hi devs, Is there a way to count the number of times a particular constructor is allocated? I might want to know, say, the total number of cons cells allocated over the course of a program, or (in my actual case) the total number of FunTy cells allocated over the course of a program. I can try to do this with an SCC. But it's very clunky: * The constructor may be used in a number of places; each such place would need the SCC. * SCCs can interfere with optimizations. In my use case, this would negate the usefulness of the exercise entirely, as I think some of the structures I wish to observe should never come into being, due to optimizations (e.g. case-of-known-constructor after inlining). Surely, the RTS must be able to count the number of times a constructor is used. But is there any way to access such a feature? If others agree that this would sometimes be helpful, perhaps we can build the feature. Now is not the first time I've wanted this. Thanks! Richard

5 6

How to use the compact regions API to write a compact to a file?
by Matthew Pickering 22 May '20

22 May '20

Dear devs, I have been under the impression for a while that you can write a compact region to a file and the functions in GHC.Compact.Serialized also seem to suggest this is possible. https://hackage.haskell.org/package/ghc-compact-0.1.0.0/docs/GHC-Compact-Se… It seems like there are some functions missing from the API. There is "importCompactByteStrings" but no function in the API which produces ByteStrings. So in order to use this function you have to convert a SerializedCompact into a ByteString, but in what specific way? Has anyone got any code where they have tried to do this? Cheers, Matt

2 2

build ghc with make (ghc-boot-th broken deps)
by Daneel Yaitskov 21 May '20

21 May '20

Hi List, I pulled ghc repo (568d7279a) and trying to build with following env: Ghc from stack (8.8.3). I made links to ghc and ghc-pkg in /usr/bin ./boot is passing well, but ./configure breaks on ghc-boot step. I have no idea how could I fix this issue. I manually modified dep versions in to what I have for ghc-8.8.3 and issue with ghc-boot-th disappeared and appeared parsec libraries/bootstrapping.conf/binary-0.8.7.0.conf depends: array-0.5.4.0 base-4.13.0.0 bytestring-0.10.10.0 containers-0.6.2.1 but there was depends: array-0.5.3.0 base-4.12.0.0 bytestring-0.10.8.2 containers-0.6.0.1 Where is actual problem? Old cabal? 10:45$ cabal --version cabal-install version 1.24.0.2 compiled using version 1.24.2.0 of the Cabal library Configuring ghc-boot-th-8.11.0.20200520... "/usr/bin/ghc-pkg" update -v0 --force --package-db=libraries/bootstrapping.conf libraries/ghc-boot-th/dist-boot/inplace-pkg-config "inplace/bin/ghc-cabal" configure libraries/ghc-boot dist-boot --with-ghc="/usr/bin/ghc" --with-ghc-pkg="/usr/bin/ghc-pkg" --package-db=/home/dan/demo/haskell/compiler/ghc/libraries/bootstrapping.conf --disable-library-for-ghci --enable-library-vanilla --enable-library-for-ghci --disable-library-profiling --disable-shared --with-hscolour="/home/dan/.cabal/bin/HsColour" --configure-option=CFLAGS="-Wall -Werror=unused-but-set-variable -Wno-error=inline -iquote /home/dan/demo/haskell/compiler/ghc/libraries/ghc-boot" --configure-option=LDFLAGS=" " --configure-option=CPPFLAGS=" " --gcc-options="-Wall -Werror=unused-but-set-variable -Wno-error=inline -iquote /home/dan/demo/haskell/compiler/ghc/libraries/ghc-boot " --constraint "binary == 0.8.7.0" --constraint "transformers == 0.5.6.2" --constraint "mtl == 2.2.2" --constraint "hpc == 0.6.1.0" --constraint "ghc-boot-th == 8.11.0.20200520" --constraint "ghc-boot == 8.11.0.20200520" --constraint "template-haskell == 2.17.0.0" --constraint "text == 1.2.4.0" --constraint "parsec == 3.1.14.0" --constraint "Cabal == 3.3.0.0" --constraint "ghc-heap == 8.11.0.20200520" --constraint "exceptions == 0.10.4" --constraint "ghci == 8.11.0.20200520" --constraint "terminfo == 0.4.1.4" --with-gcc="gcc" --with-ld="ld.gold" --with-ar="ar" --with-alex="/home/dan/.cabal/bin/alex" --with-happy="/home/dan/.cabal/bin/happy" Configuring ghc-boot-8.11.0.20200520... Error: The following packages are broken because other packages they depend on are missing. These broken packages must be rebuilt before they can be used. installed package binary-0.8.7.0 is broken due to missing package array-0.5.3.0, base-4.12.0.0, bytestring-0.10.8.2, containers-0.6.0.1 libraries/ghc-boot/ghc.mk:3: recipe for target 'libraries/ghc-boot/dist-boot/package-data.mk' failed make[1]: *** [libraries/ghc-boot/dist-boot/package-data.mk] Error 1 Makefile:123: recipe for target 'all' failed make: *** [all] Error 2 -- Best regards, Daniil Iaitskov

2 3