October 2019 - ghc-devs

The future of Phabricator
by Ben Gamari 18 Sep '21

18 Sep '21

TL;DR. For several reasons I think we should consider alternatives to Phabricator. My view is that GitLab seems like the best option. Hello everyone, Over the past year I have been growing increasingly weary of our continued dependence on Phabricator. Without a doubt, its code review interface is the best I have used. However, for a myriad of reasons I am recently questioning whether it is still the best tool for GHC's needs. # The problem There are a number of reasons why I am currently uncertain about Phabricator. For one, at this point we have no options for support in the event that something goes wrong as the company responsible for Phabricator, Phacility, has closed their support channels to non-paying customers. Furthermore, in the past year or two Phacility has been placing their development resources in the parts their customers pay them for, which appear to be much different that the parts that we actively use. For this reason, some parts that we rely on seem oddly half-finished. This concern was recently underscored by some rather unfortunate misfeatures in Harbormaster which resulted in broken CI after the Hadrian merge and now apparent bugs which have made it difficult to migrate to the CircleCI integration we previously prepared. Perhaps most importantly, in our recent development priorities survey our use of Phabricator was the most common complaint by a fair margin, both in case of respondents who have contributed patches and those who have not. On the whole, past contributors and potential future contributors alike have strongly indicated that they want a more Git-like experience. Of course, this wasn't terribly surprising; this is just the most recent case where contributors have made this preference known. Frankly, in a sense it is hard to blame them. The fact that users need to install a PHP tool, Arcanist, to contribute anything but documentation patches has always seemed like unnecessary friction to me and I would be quite happy to be rid of it. Indeed we have had a quite healthy number of GitHub documentation patches since we started accepting them. This makes me thing that there may indeed be potential contributoes that we are leaving on the table. # What to do With Rackspace support ending at the end of year, now may be a good time to consider whether we really want to continue on this road. Phabricator is great at code review but I am less and less certain that it is worth the maintenance uncertainty and potential lost contributors that it costs. Moreover, good alternatives seem closer at-hand than they were when we deployed Phabricator. ## Move to GitHub When people complain about our infrastructure, they often use GitHub as the example of what they would like to see. However, realistically I have a hard time seeing GitHub as a viable option. Its feature set is simply insufficient enough to handle the needs of a larger project like GHC without significant external tooling (as seen in the case of Rust-lang). The concrete reasons have been well-documented in previous discussions but, to summarize, * its review functionality is extremely painful to use with larger patches * rebased patches lead to extreme confusion and lost review comments * it lacks support for pre-receive hooks, which serve as a last line of defense to prevent unintended submodule changes * its inability to deal with external tickets is problematic * there is essentially no possibility that we could eventually migrate GHC's tickets to GitHub's ticket tracker without considerable data loss (much less manage the ticket load that would result), meaning that we would forever be stuck with maintaining Trac. * on a personal note, its search functionality has often left me empty-handed On the whole, these issues seem pretty hard to surmount. ## Move to GitLab In using GitLab for another project over the past months I have been positively surprised by its quality. It handles rebased merge requests far better than GitHub, has functional search, and quite a usable review interface. Furthermore, upstream has been extremely responsive to suggestions for improvement [1]. Even out-of-the-box it seems to be flexible enough to accommodate our needs, supporting integration with external issue trackers, having reasonable release management features, and support for code owners to automate review triaging (replacing much of the functionality of Phabricator's Herald). Finally, other FOSS projects' [3] recent migrations from Phabrictor to GitLab have shown that GitLab-the-company is quite willing to offer help when needed. I took some time this weekend to try setting up a quick GHC instance [2] to play around with. Even after just a few hours of playing around I think the result is surprisingly usable. Out of curiosity I also played around with importing some tickets from Trac (building on Matt Pickering's Trac-to-Maniphest migration tool). With relatively little effort I was even able to get nearly all of our tickets (as of 9 months ago) imported while preserving ticket numbers (although there are naturally a few wrinkles that would need to be worked out). Naturally, I think we should treat the question of ticket tracker migration as an orthogonal one to code review, but it is good to know that this is possible. ## Continue with Phabricator Continuing with Phabricator is of course an option. Its review functionality is great and it has served us reasonably well. However, compared to GitLab and even GitHub of today its features seem less distinguished than they once did. Moreover, the prospect of having to maintain a largely stagnant product with no support strikes me as a slightly dangerous game to play. Working around the issues we have recently encountered has already cost a non-negligible amount of time. # The bottom line If it wasn't clear already, I think that we should strongly consider a move to GitLab. At this point it seems clear that it isn't going to vanish, has a steady pace of development, is featureful, and available. However, these are just my thoughts. What do you think? Cheers, - Ben [1] 11.4 will ship with a file tree view in the code review interface, which I reported (https://gitlab.com/gitlab-org/gitlab-ce/issues/46474) as being is one of the Phabricator features I missed the most during review [2] https://gitlab.home.smart-cactus.org/ghc/ghc/issues/14641 [3] The GNOME and freedesktop.org projects have recently migrated, the former from a hodge-podge of self-hosted services and the latter from Phabricator

17 40

New implementation for `ImpredicativeTypes`
by Alejandro Serrano Mena 08 Sep '21

08 Sep '21

Hi all, As I mentioned some time ago, we have been busy working on a new implementation of `ImpredicativeTypes` for GHC. I am very thankful to everybody who back then sent us examples of impredicativity which would be nice to support, as far as we know this branch supports all of them! :) If you want to try it, at https://gitlab.haskell.org/trupill/ghc/commit/a3f95a0fe0f647702fd7225fa719a… you can find the result of the pipeline, which includes builds for several platforms (click on the "Artifacts" button, the one which looks like a cloud, to get them). The code is being developed at https://gitlab.haskell.org/trupill/ghc. Any code should run *unchanged* except for some eta-expansion required for some specific usage patterns of higher-rank types. Please don't hesitate to ask any questions or clarifications about it. A merge request for tracking this can be found at https://gitlab.haskell.org/ghc/ghc/merge_requests/1659 Kind regards, Alejandro

8 23

Residency profiles
by Simon Peyton Jones 23 Jan '20

23 Jan '20

Simon, Ben, Omer As you'll see in comments 55-72 of https://ghc.haskell.org/trac/ghc/ticket/9476, Sebastian has been a bit flummoxed by the task of measure residency profiles; that is, how much data is truly live during execution. A major GC measures that, but we are vulnerable to exactly when it happens (even with -G1) and that can lead to irreproducible results. I think what we want is a way to trigger GC at very regular intervals, after (say) each 10kbytes or 100kbytes or 1Mbyte of allocation. That might be expensive, but we'd get reproducible results. I don't think that is possible right now - see the ticket - but it would be easy enough to do wouldn't it? Just give only 10k or 100k or 1M to the allocator when setting it running again. Would you consider this? Or are we just missing something obvious? Needless to say, we want to do all this with full optimisation on, no cost-centre profiling. Thanks Simon

4 8

Invitation to GHC Week
by Matthew Pickering 19 Dec '19

19 Dec '19

Dear all, I am happy to announce GHC Week, a developers meeting focusing on GHC, organized by myself and AndreasK. We are renting a big holiday cottage in rural Belgium during March so that participants can hack on GHC uninterrupted for a whole week. The place has a large number of beds and a nice open living area for collaboration and relaxation. Dates: 13th - 20th March 2020 Location: Annevoie, Belgium If you are interested in attending GHC Week then please consult the linked webpage which contains information about how to attend. There are only a limited number of spaces and we already have a number of confirmed attendees. Hopefully no one will be disappointed but if all 700 subscribers to ghc-devs want to attend there will not be enough space! We would particularly like to encourage undergraduate students who are interested in working on GHC to attend GHC Week. Thanks to our sponsors, Haskell.org and Well-Typed we are in a position to not only make the event itself very affordable, but also to cover travel and accommodation costs for those in need. This is an exceptional opportunity to learn about GHC from experienced developers. So if you can, make sure you don’t miss it! For more details please consult the following page: https://andreaspk.github.io/ghc-week Cheers, Matt & Andreas

1 1

[ANNOUNCE] GHC 8.8.1-alpha2 is now available
by Ben Gamari 12 Dec '19

12 Dec '19

Hello everyone, The GHC team is pleased to announce the second and likely last alpha release of GHC 8.8.1. The source distribution, binary distributions, and documentation are available at https://downloads.haskell.org/~ghc/8.8.1-alpha2 A draft of the release notes is also available [1]. This release is the culmination of over 3000 commits by over one hundred contributors and has several new features and numerous bug fixes relative to GHC 8.6: * Profiling now works correctly on 64-bit Windows (although still may be problematic on 32-bit Windows due to platform limitations; see #15934) * A new code layout algorithm for amd64's native code generator * The introduction of a late lambda-lifting pass which may reduce allocations significantly for some programs. * Further work on Trees That Grow, enabling improved code re-use of the Haskell AST in tooling * More locations where users can write `forall` (GHC Proposal #0007) * Further work on the Hadrian build system This release brings a number of fixes since alpha 1: * A number of linker fixes (#16779, #16784) * The process, binary, Cabal, time, terminfo libraries have all been bumped to their final release versions * A regression rendering TemplateHaskell unusable in cross-compiled configurations has been fixed (#16331) * A regression causing compiler panics on compilation of some programs has been fixed (#16449) * -Wmissing-home-modules now handles hs-boot files correctly (#16551) * A regression causing some programs to fail at runtime has been fixed (#16066) Due to on-going work on our release and testing infrastructure this cycle is proceeding at a pace significantly slower than expected. However, we anticipate that this investment will allow us to release a more reliable, easier-to-install compiler on the planned six-month release cadence in the future. As always, if anything looks amiss do let us know. Happy compiling! Cheers, - Ben [1] https://downloads.haskell.org/ghc/8.8.1-alpha2/docs/html/users_guide/8.8.1-…

8 18

users-guide page for master branch
by Takenobu Tani 03 Dec '19

03 Dec '19

Hi Ben, devs, This is not in hurry. It looks like the users-guide page for master branch [1] has not been updated recently. [1]: https://ghc.gitlab.haskell.org/ghc/doc/users_guide Regards, Takenobu

2 3

Quick Q: do all FFI (non-primop) calls involve State# and RealWorld?
by Christopher Done 09 Nov '19

09 Nov '19

Hi all, I tried compiling this file: {-# LANGUAGE NoImplicitPrelude #-}-- | Demonstrate various use of the FFI.module Foreign whereimport Foreign.Cforeign import ccall "math.h sin" sin :: CDouble -> CDoubleit :: CDoubleit = sin 1 And I’ve noticed that the annotated type given for this foreign op in Core, is (# State# RealWorld, CDouble #), whereas I would have expected e.g. CDouble. Meanwhile, the foreign op call is passed a RealWorld argument. Additionally, code that consumes the result of this foreign call expects a (# CDouble #) as a return value. So there are some assumptions I put in my interpreter to test this FFI call out: 1. Despite claiming to return the real world in a tuple, it actually should just return an unboxed tuple of the value. 2. It should ignore the RealWorld argument entirely. I assume, if I were to lift this function into returning IO, that I should indeed return the RealWorld argument given. So the lesson is: All FFI functions accept a RealWorld, and may return a 2-tuple of State# RealWorld *if* it’s impure, else it’ll return a 1-tuple of the value. Correct? Can someone confirm that my observations are right? Also, if so, is there somewhere I can read more about this? Cheers Chris

2 3

Proposed changes to merge request workflow
by Ben Gamari 08 Nov '19

08 Nov '19

tl;dr. I would like feedback on a few proposed changes [1] to our merge request workflow. Hello everyone, Over the past six months I have been monitoring the operation of our merge request workflow, which arose rather organically in the wake of the initial move to GitLab. While it works reasonably well, there is clearly room for improvement: * we have no formal way to track the status of in-flight merge requests (e.g. for authors to mark an MR as ready for review or reviewers to mark work as ready for merge) * merge requests still at times languish without review * the backport protocol is somewhat error prone and requires a great deal of attention to ensure that patches don't slip through the cracks * there is no technical mechanism to prevent that under-reviewed patches from being merged (either intentionally or otherwise) to `master` To address this I propose [1] a few changes to our workflow: 1. Define explicit phases of the merge request lifecycle, systematically identified with labels. This will help to make it clear who is responsible for a merge request at every stage of its lifecycle. 2. Make it clear that it is the contributor's responsibility to identify reviewers for their merge requests. 3. Institute a final pre-merge sanity check to ensure that patches are adequately reviewed, documented, tested, and have had their ticket and MR metadata updated. Note that this is merely a proposal; I am actively seeking input from the developer community. Do let me know what you think. Cheers, - Ben [1] https://gitlab.haskell.org/ghc/ghc/wikis/proposals/merge-request-workflow

9 13

DriverPipeline/HscMain DynFlags mystery
by Ömer Sinan Ağacan 01 Nov '19

01 Nov '19

Hi, We recently did some refactoring in HscMain and DriverPipeline to generate interfaces after final Cmms are generated (previously interfaces would be generated after the tidying pass). It's mostly done but there's one thing that I couldn't figure out after two full days of debugging (I also asked a few people about it on IRC), so I wanted to ask here in case someone here knows. Previously the interface value (ModIface) would be generated and written to disk in `HscMain.finish`. The DynFlags we use to generate the ModIface and to write it to disk would be the one passed to `HscMain.hscIncrementalCompile`. In the new implementation part of the interface is still generated in `HscMain.hscIncrementalCompile` (see mkPartialIface), using the same DynFlags as before. But more stuff is added after the final Cmms are generated (see mkFullIface calls in DriverPipeline) using DynFlags in `compileOne'` or `runPhase` (called by `runPipeline`). It turns out these DynFlags are different enough from the one passed to `HscMain.hscIncrementalCompile` that some tests fail (I remember a backpack test, but there may be more). ("Full" interfaces are written to disk right after generation) See [1] for the hack I added as a workaround. Basically I keep the DynFlags passed to hscIncrementalCompile so that I can generate the final interfaces correctly. The question is what's changing in DynFlags that makes things go wrong. I tried looking at the fields used by mkFullIface and hscMaybeWriteIface, but as far as I can see none of the fields used by these functions are different from the DynFlags passed to hscIncrementalCompile. If anyone knows what's going on any help would be appreciated. Thanks, Ömer [1]: https://gitlab.haskell.org/ghc/ghc/blob/master/compiler/main/HscTypes.hs#L2…

2 2

Linking problem
by Simon Peyton Jones 30 Oct '19

30 Oct '19

Devs On the branch wip/inline-compulsory, I'm getting a strange link error: simonpj@MSRC-3645512:~/tmp$ ~/code/HEAD-2/inplace/bin/ghc-stage1 Hello.hs [1 of 1] Compiling Main ( Hello.hs, Hello.o ) Linking Hello ... /home/simonpj/code/HEAD-2/rts/dist/build/libHSrts.a(Updates.o):stg_bh_upd_frame_info: error: undefined reference to 'stg_marked_upd_frame_ret' /home/simonpj/code/HEAD-2/rts/dist/build/libHSrts.a(StgMiscClosures.o):stg_CAF_BLACKHOLE_info: error: undefined reference to 'stg_BLACKHOLE_entry' /home/simonpj/code/HEAD-2/rts/dist/build/libHSrts.a(StgMiscClosures.o):__stg_EAGER_BLACKHOLE_info: error: undefined reference to 'stg_BLACKHOLE_entry' /home/simonpj/code/HEAD-2/rts/dist/build/libHSrts.a(Apply.o):stg_AP_STACK_info: error: undefined reference to 'stg_WHITEHOLE_entry' /home/simonpj/code/HEAD-2/rts/dist/build/libHSrts.a(Exception.o):function stg_raisezh: error: undefined reference to 'stg_ap_n_ret' /home/simonpj/code/HEAD-2/rts/dist/build/libHSrts.a(Exception.o):function stg_raisezh: error: undefined reference to 'stg_ap_pv_ret' /home/simonpj/code/HEAD-2/rts/dist/build/libHSrts.a(AutoApply.o):stg_ap_v_info: error: undefined reference to 'stg_BCO_entry' /home/simonpj/code/HEAD-2/rts/dist/build/libHSrts.a(AutoApply.o):stg_ap_f_info: error: undefined reference to 'stg_BCO_entry' /home/simonpj/code/HEAD-2/rts/dist/build/libHSrts.a(AutoApply.o):stg_ap_d_info: error: undefined reference to 'stg_BCO_entry' /home/simonpj/code/HEAD-2/rts/dist/build/libHSrts.a(AutoApply.o):stg_ap_l_info: error: undefined reference to 'stg_BCO_entry' ... etc... But I have done nothing in the RTS! If I try 'nm' I get bash$ nm rts/dist/build/libHSrts.a | grep stg_ap_p_fast U stg_ap_p_fast U stg_ap_p_fast 0000000000003f58 T stg_ap_p_fast So somehow the same symbol is defined and unresolved. Really strange. Can you help? Thanks Simon

2 2