January 2019 - ghc-devs

The future of Phabricator
by Ben Gamari 18 Sep '21

18 Sep '21

TL;DR. For several reasons I think we should consider alternatives to Phabricator. My view is that GitLab seems like the best option. Hello everyone, Over the past year I have been growing increasingly weary of our continued dependence on Phabricator. Without a doubt, its code review interface is the best I have used. However, for a myriad of reasons I am recently questioning whether it is still the best tool for GHC's needs. # The problem There are a number of reasons why I am currently uncertain about Phabricator. For one, at this point we have no options for support in the event that something goes wrong as the company responsible for Phabricator, Phacility, has closed their support channels to non-paying customers. Furthermore, in the past year or two Phacility has been placing their development resources in the parts their customers pay them for, which appear to be much different that the parts that we actively use. For this reason, some parts that we rely on seem oddly half-finished. This concern was recently underscored by some rather unfortunate misfeatures in Harbormaster which resulted in broken CI after the Hadrian merge and now apparent bugs which have made it difficult to migrate to the CircleCI integration we previously prepared. Perhaps most importantly, in our recent development priorities survey our use of Phabricator was the most common complaint by a fair margin, both in case of respondents who have contributed patches and those who have not. On the whole, past contributors and potential future contributors alike have strongly indicated that they want a more Git-like experience. Of course, this wasn't terribly surprising; this is just the most recent case where contributors have made this preference known. Frankly, in a sense it is hard to blame them. The fact that users need to install a PHP tool, Arcanist, to contribute anything but documentation patches has always seemed like unnecessary friction to me and I would be quite happy to be rid of it. Indeed we have had a quite healthy number of GitHub documentation patches since we started accepting them. This makes me thing that there may indeed be potential contributoes that we are leaving on the table. # What to do With Rackspace support ending at the end of year, now may be a good time to consider whether we really want to continue on this road. Phabricator is great at code review but I am less and less certain that it is worth the maintenance uncertainty and potential lost contributors that it costs. Moreover, good alternatives seem closer at-hand than they were when we deployed Phabricator. ## Move to GitHub When people complain about our infrastructure, they often use GitHub as the example of what they would like to see. However, realistically I have a hard time seeing GitHub as a viable option. Its feature set is simply insufficient enough to handle the needs of a larger project like GHC without significant external tooling (as seen in the case of Rust-lang). The concrete reasons have been well-documented in previous discussions but, to summarize, * its review functionality is extremely painful to use with larger patches * rebased patches lead to extreme confusion and lost review comments * it lacks support for pre-receive hooks, which serve as a last line of defense to prevent unintended submodule changes * its inability to deal with external tickets is problematic * there is essentially no possibility that we could eventually migrate GHC's tickets to GitHub's ticket tracker without considerable data loss (much less manage the ticket load that would result), meaning that we would forever be stuck with maintaining Trac. * on a personal note, its search functionality has often left me empty-handed On the whole, these issues seem pretty hard to surmount. ## Move to GitLab In using GitLab for another project over the past months I have been positively surprised by its quality. It handles rebased merge requests far better than GitHub, has functional search, and quite a usable review interface. Furthermore, upstream has been extremely responsive to suggestions for improvement [1]. Even out-of-the-box it seems to be flexible enough to accommodate our needs, supporting integration with external issue trackers, having reasonable release management features, and support for code owners to automate review triaging (replacing much of the functionality of Phabricator's Herald). Finally, other FOSS projects' [3] recent migrations from Phabrictor to GitLab have shown that GitLab-the-company is quite willing to offer help when needed. I took some time this weekend to try setting up a quick GHC instance [2] to play around with. Even after just a few hours of playing around I think the result is surprisingly usable. Out of curiosity I also played around with importing some tickets from Trac (building on Matt Pickering's Trac-to-Maniphest migration tool). With relatively little effort I was even able to get nearly all of our tickets (as of 9 months ago) imported while preserving ticket numbers (although there are naturally a few wrinkles that would need to be worked out). Naturally, I think we should treat the question of ticket tracker migration as an orthogonal one to code review, but it is good to know that this is possible. ## Continue with Phabricator Continuing with Phabricator is of course an option. Its review functionality is great and it has served us reasonably well. However, compared to GitLab and even GitHub of today its features seem less distinguished than they once did. Moreover, the prospect of having to maintain a largely stagnant product with no support strikes me as a slightly dangerous game to play. Working around the issues we have recently encountered has already cost a non-negligible amount of time. # The bottom line If it wasn't clear already, I think that we should strongly consider a move to GitLab. At this point it seems clear that it isn't going to vanish, has a steady pace of development, is featureful, and available. However, these are just my thoughts. What do you think? Cheers, - Ben [1] 11.4 will ship with a file tree view in the code review interface, which I reported (https://gitlab.com/gitlab-org/gitlab-ce/issues/46474) as being is one of the Phabricator features I missed the most during review [2] https://gitlab.home.smart-cactus.org/ghc/ghc/issues/14641 [3] The GNOME and freedesktop.org projects have recently migrated, the former from a hodge-podge of self-hosted services and the latter from Phabricator

17 40

Residency profiles
by Simon Peyton Jones 23 Jan '20

23 Jan '20

Simon, Ben, Omer As you'll see in comments 55-72 of https://ghc.haskell.org/trac/ghc/ticket/9476, Sebastian has been a bit flummoxed by the task of measure residency profiles; that is, how much data is truly live during execution. A major GC measures that, but we are vulnerable to exactly when it happens (even with -G1) and that can lead to irreproducible results. I think what we want is a way to trigger GC at very regular intervals, after (say) each 10kbytes or 100kbytes or 1Mbyte of allocation. That might be expensive, but we'd get reproducible results. I don't think that is possible right now - see the ticket - but it would be easy enough to do wouldn't it? Just give only 10k or 100k or 1M to the allocator when setting it running again. Would you consider this? Or are we just missing something obvious? Needless to say, we want to do all this with full optimisation on, no cost-centre profiling. Thanks Simon

4 8

GHC's module hierarchy
by Simon Peyton Jones 02 Oct '19

02 Oct '19

Dear ghc-devs hsyl20 proposes a radical overhaul of the module structure of GHC itself. He or she suggested it six months ago in https://ghc.haskell.org/trac/ghc/ticket/13009 and has now offered a monster patch https://phabricator.haskell.org/D3647 It's clearly the result of a lot of work, but I was the only one who responded on the original ticket, and it'll affect all of your lives in a very immediate way. So, would you like to * consider the idea * look at the actual re-mapping of modules hsyl20 proposals * express an opinion about whether to go ahead Probably the ticket, rather than Phab, is the best place to comment on the general idea. I'd like to thank hsyl20. GHC's rather flat module structure has grown incrementally over years. But still, there are pros and cons. Simon

2 1

Any ways to test a GHC build against large set of packages (including test suites)?
by Ömer Sinan Ağacan 25 Jul '19

25 Jul '19

Hi, I'd like to test some GHC builds + some compile and runtime flag combinations against a large set of packages by building them and running test suites. For this I need - A set of packages that are known to work with latest GHC - A way to build them and run their test suites (if I could specify compile and runtime flags that'd be even better) I think stackage can serve as (1) but I don't know how to do (2). Can anyone point me to the right direction? I vaguely remember some nix-based solution for this that was being discussed on the IRC channel, but can't recall any details. Thanks, Ömer

9 15

GitLab forks and submodules
by Moritz Angermann 17 Feb '19

17 Feb '19

Hi *, so what do we do with submodules? If you point someone to a fork of ghc, say: gitlab.haskell.org/foo/ghc and they try to check it out, they will run into issues because foo didn't clone all the submodules. So how is one supposed to clone a forked ghc repository? Cheers, Moritz

8 15

Thoughts on the Contributing page
by Ben Gamari 31 Jan '19

31 Jan '19

For those following along at home: David has been looking at revising our contributor documentation. He has started consolidating a variety of relevant content on the Contributing page of the Wiki [1]. Below are my thoughts; feel free to chime in with your own. David, The contributing page is looking quite good. However, I do wonder whether we could reduce the link "fan-out" a bit more: it still rather feels like a collection of links with no clear "beginning". The "Newcomers to GHC" section is a great start but I see two potential issues: * we don't clearly articulate the precise steps that a newcomer will need to take * the first thing we mention are four links which, while useful, constitute a significant volume of reading for a newcomer. I fear we may lose people at this point. Regarding the second point, I think that WorkingConventions/FixingBugs is a good model in that it clearly lists a series of concrete steps that the contributor should take. Admittedly, I think some of the detail should be dropped or moved (e.g. the mention of setting git's user.name variable is likely unnecessary in 2019). I think that something similar to this list should be the first thing one sees when they reach the contributing page. Ideally the "typical" case of a new contributor wanting to submit their first patch would be able to gather everything they need to get started in one or two screenfuls of text. After that prose can come additional links to more in-depth documentation. Other various issues I noticed: * We link to wiki:WorkingConventions from a variety of places (including wiki:Contributing) but it is now just a link to wiki:Contributing. * wiki:WorkingConventions/FixingBugs still has references to Phabricator. In general we should start culling such references. * wiki:WorkingConventions/FixingBugs also has references to Trac documentation. We should try to replace these with the relevant GitLab documentation when we migrate. * wiki:WorkingConventions/FixingBugs should be updated to reflect that GitLab CI is now the source of truth for validation. * I don't know exactly how this should look but I think we need to do a better job of concisely stating what we expect of contributions. That is: * commit messages should be readable and discuss what the patch does. "Fixes #NNNN" is not an adequate commit message. * changes should be commented where necessary (my usual rule of thumb is "write the comment you would have liked to read when you started writing your patch"). We should mention the Note convention. * commit messages and comments should refer to ticket numbers where appropriate (e.g. using usual #NNNN syntax; people often get this wrong). * commits should be either squashed or logically distinct, individually buildable changes. * changes to most of `base` need to go through the CLC (this may be optional as defining what set of `base` this applies to is a bit tricky; in the interest of keeping things concise we may be better off handling this personally on a case-by-case basis) * code changes should conform to GHC's (somewhat fluid, for better or worse) code style (https://ghc.haskell.org/trac/ghc/wiki/Commentary/CodingStyle) * thought should be given to testing Having a document which succinctly summarised these expectations as we could easily refer to it during code review. Even better, we could excerpt it as a checklist in our merge request description template (I have put up an initial attempt at this as !149). Anyways, there is more to be said here but this email is getting a bit long so let's leave it for future discussions. Cheers, - Ben [1] https://ghc.haskell.org/trac/ghc/wiki/Contributing

5 13

RE: Thoughts on the Contributing page
by Andrey Mokhov 30 Jan '19

30 Jan '19

Andreas, Alp: > > - On windows build.bat defaults to stack which I think has never > > worked on my box. > Andrey? (cc'd him) Actually `build.bat` calls the Cabal-based build script. This was a relatively recent change, and we plan to stick to it in the long term, while still providing an alternative `build.stack.bat`. > There are also a few quality of life issues like ctrl+c not canceling > the build on windows. > Which I hope will be resolved at some point but not sure if these > should be showstoppers. Indeed. I hope it's not difficult to fix this, but I'm not sure where to start. Any suggestions are very welcome. Cheers, Andrey

2 1

Re: Thoughts on the Contributing page
by Andreas Klebinger 30 Jan '19

30 Jan '19

On hadrian: Documentation will eventually catch up as more people use hadrian but imo things that need to be supported are: - Some workflows: * make fast * ./validate * make EXTRA_HC_OPTS="..." - On windows build.bat defaults to stack which I think has never worked on my box. - It's still too easy to run into hadrian bugs imo, which will probably work itself out in a few months. There are also a few quality of life issues like ctrl+c not canceling the build on windows. Which I hope will be resolved at some point but not sure if these should be showstoppers. ghc-devs-request(a)haskell.org schrieb: > One more thought I'd like to throw out in the open here: > > The current Newcomers' Guide uses the current Makefile workflow, but > this is on a fast track to deprecation - but then, I doubt Hadrian has > seen enough exposure yet to use for a good beginner-friendly "Just > Works" guide. I'm leaning towards sticking with make for now, also > because existing material is already written this way; and then once > Hadrian is truly ready for prime time, we can rewrite the relevant > parts. > > Thoughts?

2 1

Why are nested brackets disallowed?
by Matthew Pickering 30 Jan '19

30 Jan '19

There is a check in `RnSplice` which errors on the following program with nested brackets. ``` prog = [| [| True |] |] T.hs:4:11: error: • Template Haskell brackets cannot be nested (without intervening splices) • In the Template Haskell quotation [| True |] In the Template Haskell quotation [| [| True |] |] | 4 | prog = [| [| True |] |] | ^^^^^^^^^^ ``` As far as I can see the check was added in 2013 in this commit, https://github.com/ghc/ghc/commit/d0d47ba76f8f0501cf3c4966bc83966ab38cac27#… But there is no note, no tests and no comment about why it was added. I removed the check and added a `BracketE` constructor to the template-haskell AST and the code compiles fine. I can also construct a program which needs to be spliced twice and this also works fine. ``` func Add = [| (+) |] func Mul = [| (*) |] f1 "+" = [| Add |] f1 "*" = [| Mul |] comb s = [| func $(f1 s) |] ``` ``` res = $($(comb "*")) ``` So it seems the restriction is quite arbitrary but I was wondering if I was missing some limitation which meant this check was added. I would not be surprised if something more complicated goes wrong with splicing. Cheers, Matt

4 8

hcs2hs failure
by Simon Peyton Jones 30 Jan '19

30 Jan '19

Is anyone else seeing this in a GHC build utils/hsc2hs/HSCParser.hs:170:14: error: [-Wunused-matches, -Werror=unused-matches] Defined but not used: 'd' | 170 | '\'':d:'\'':_ -> do any3Chars_; text -- Case 2 | ^ Indeed, d looks unused. But this line of code has been there since 28 Dec, so I can't understand why it has not bitten before. With -Werror my build fails. I could switch to -Wwarn, but I think -Werror is the default. Simon

2 1