December 2018 - ghc-devs

The future of Phabricator
by Ben Gamari 18 Sep '21

18 Sep '21

TL;DR. For several reasons I think we should consider alternatives to Phabricator. My view is that GitLab seems like the best option. Hello everyone, Over the past year I have been growing increasingly weary of our continued dependence on Phabricator. Without a doubt, its code review interface is the best I have used. However, for a myriad of reasons I am recently questioning whether it is still the best tool for GHC's needs. # The problem There are a number of reasons why I am currently uncertain about Phabricator. For one, at this point we have no options for support in the event that something goes wrong as the company responsible for Phabricator, Phacility, has closed their support channels to non-paying customers. Furthermore, in the past year or two Phacility has been placing their development resources in the parts their customers pay them for, which appear to be much different that the parts that we actively use. For this reason, some parts that we rely on seem oddly half-finished. This concern was recently underscored by some rather unfortunate misfeatures in Harbormaster which resulted in broken CI after the Hadrian merge and now apparent bugs which have made it difficult to migrate to the CircleCI integration we previously prepared. Perhaps most importantly, in our recent development priorities survey our use of Phabricator was the most common complaint by a fair margin, both in case of respondents who have contributed patches and those who have not. On the whole, past contributors and potential future contributors alike have strongly indicated that they want a more Git-like experience. Of course, this wasn't terribly surprising; this is just the most recent case where contributors have made this preference known. Frankly, in a sense it is hard to blame them. The fact that users need to install a PHP tool, Arcanist, to contribute anything but documentation patches has always seemed like unnecessary friction to me and I would be quite happy to be rid of it. Indeed we have had a quite healthy number of GitHub documentation patches since we started accepting them. This makes me thing that there may indeed be potential contributoes that we are leaving on the table. # What to do With Rackspace support ending at the end of year, now may be a good time to consider whether we really want to continue on this road. Phabricator is great at code review but I am less and less certain that it is worth the maintenance uncertainty and potential lost contributors that it costs. Moreover, good alternatives seem closer at-hand than they were when we deployed Phabricator. ## Move to GitHub When people complain about our infrastructure, they often use GitHub as the example of what they would like to see. However, realistically I have a hard time seeing GitHub as a viable option. Its feature set is simply insufficient enough to handle the needs of a larger project like GHC without significant external tooling (as seen in the case of Rust-lang). The concrete reasons have been well-documented in previous discussions but, to summarize, * its review functionality is extremely painful to use with larger patches * rebased patches lead to extreme confusion and lost review comments * it lacks support for pre-receive hooks, which serve as a last line of defense to prevent unintended submodule changes * its inability to deal with external tickets is problematic * there is essentially no possibility that we could eventually migrate GHC's tickets to GitHub's ticket tracker without considerable data loss (much less manage the ticket load that would result), meaning that we would forever be stuck with maintaining Trac. * on a personal note, its search functionality has often left me empty-handed On the whole, these issues seem pretty hard to surmount. ## Move to GitLab In using GitLab for another project over the past months I have been positively surprised by its quality. It handles rebased merge requests far better than GitHub, has functional search, and quite a usable review interface. Furthermore, upstream has been extremely responsive to suggestions for improvement [1]. Even out-of-the-box it seems to be flexible enough to accommodate our needs, supporting integration with external issue trackers, having reasonable release management features, and support for code owners to automate review triaging (replacing much of the functionality of Phabricator's Herald). Finally, other FOSS projects' [3] recent migrations from Phabrictor to GitLab have shown that GitLab-the-company is quite willing to offer help when needed. I took some time this weekend to try setting up a quick GHC instance [2] to play around with. Even after just a few hours of playing around I think the result is surprisingly usable. Out of curiosity I also played around with importing some tickets from Trac (building on Matt Pickering's Trac-to-Maniphest migration tool). With relatively little effort I was even able to get nearly all of our tickets (as of 9 months ago) imported while preserving ticket numbers (although there are naturally a few wrinkles that would need to be worked out). Naturally, I think we should treat the question of ticket tracker migration as an orthogonal one to code review, but it is good to know that this is possible. ## Continue with Phabricator Continuing with Phabricator is of course an option. Its review functionality is great and it has served us reasonably well. However, compared to GitLab and even GitHub of today its features seem less distinguished than they once did. Moreover, the prospect of having to maintain a largely stagnant product with no support strikes me as a slightly dangerous game to play. Working around the issues we have recently encountered has already cost a non-negligible amount of time. # The bottom line If it wasn't clear already, I think that we should strongly consider a move to GitLab. At this point it seems clear that it isn't going to vanish, has a steady pace of development, is featureful, and available. However, these are just my thoughts. What do you think? Cheers, - Ben [1] 11.4 will ship with a file tree view in the code review interface, which I reported (https://gitlab.com/gitlab-org/gitlab-ce/issues/46474) as being is one of the Phabricator features I missed the most during review [2] https://gitlab.home.smart-cactus.org/ghc/ghc/issues/14641 [3] The GNOME and freedesktop.org projects have recently migrated, the former from a hodge-podge of self-hosted services and the latter from Phabricator

17 40

Residency profiles
by Simon Peyton Jones 23 Jan '20

23 Jan '20

Simon, Ben, Omer As you'll see in comments 55-72 of https://ghc.haskell.org/trac/ghc/ticket/9476, Sebastian has been a bit flummoxed by the task of measure residency profiles; that is, how much data is truly live during execution. A major GC measures that, but we are vulnerable to exactly when it happens (even with -G1) and that can lead to irreproducible results. I think what we want is a way to trigger GC at very regular intervals, after (say) each 10kbytes or 100kbytes or 1Mbyte of allocation. That might be expensive, but we'd get reproducible results. I don't think that is possible right now - see the ticket - but it would be easy enough to do wouldn't it? Just give only 10k or 100k or 1M to the allocator when setting it running again. Would you consider this? Or are we just missing something obvious? Needless to say, we want to do all this with full optimisation on, no cost-centre profiling. Thanks Simon

4 8

GHC's module hierarchy
by Simon Peyton Jones 02 Oct '19

02 Oct '19

Dear ghc-devs hsyl20 proposes a radical overhaul of the module structure of GHC itself. He or she suggested it six months ago in https://ghc.haskell.org/trac/ghc/ticket/13009 and has now offered a monster patch https://phabricator.haskell.org/D3647 It's clearly the result of a lot of work, but I was the only one who responded on the original ticket, and it'll affect all of your lives in a very immediate way. So, would you like to * consider the idea * look at the actual re-mapping of modules hsyl20 proposals * express an opinion about whether to go ahead Probably the ticket, rather than Phab, is the best place to comment on the general idea. I'd like to thank hsyl20. GHC's rather flat module structure has grown incrementally over years. But still, there are pros and cons. Simon

2 1

Any ways to test a GHC build against large set of packages (including test suites)?
by Ömer Sinan Ağacan 25 Jul '19

25 Jul '19

Hi, I'd like to test some GHC builds + some compile and runtime flag combinations against a large set of packages by building them and running test suites. For this I need - A set of packages that are known to work with latest GHC - A way to build them and run their test suites (if I could specify compile and runtime flags that'd be even better) I think stackage can serve as (1) but I don't know how to do (2). Can anyone point me to the right direction? I vaguely remember some nix-based solution for this that was being discussed on the IRC channel, but can't recall any details. Thanks, Ömer

9 15

Using same stdout/stderr files for multiple tests?
by Ömer Sinan Ağacan 27 Jan '19

27 Jan '19

I have a test that I want to run with different compile and runtime parameters. I managed to reuse the source file across different tests by adding a extra_files(['source.hs']) to the tests, but I don't know how to do the same for stdout/stderr files. Any ideas? In more details, I have test.hs test.stdout and two tests test('test', [extra_run_opts('...')], compile_and_run, []) test('test_debug', [extra_run_opts('...'), extra_hc_opts('-debug'), extra_files(['test.hs'])], compile_and_run, []) The first test works fine, but the second test fails because I don't know how to tell it to use test.stdout as the stdout file and it looks for test_debug.stdout. Thanks Ömer

3 4

Welcome to GitLab!
by Ben Gamari 07 Jan '19

07 Jan '19

TL;DR. https://gitlab.haskell.org/ghc/ghc.git is now the official upstream GHC repository. Various introductory notes are discussed. Let me know if you have any trouble. Also, please do verify the correctness of the email address associated with your Trac account in the next few weeks. It will be used to map users when we transition Trac tickets to GitLab. Hello everyone, I am happy to announce that CI on GHC's GitLab instance [1] is now stable. At this point https://gitlab.haskell.org/ghc/ghc.git is to be considered the official upstream repository of GHC. The rest of this email is meant to serve as a brief introduction and status update. It can also be viewed on the GitLab Wiki [2]. [1] https://gitlab.haskell.org/ [2] https://gitlab.haskell.org/ghc/ghc/wikis/welcome # Getting started To get started on GitLab you will first want to either create a new account [1] or login with your GitHub credentials [2]. Once you have an account you should add an SSH key [3] so that you can push to your repositories. If you currently have commit rights to GHC notify me (Ben Gamari) of your user name so I can grant you similar rights in GitLab. [1] https://gitlab.haskell.org/users/sign_in [2] https://gitlab.haskell.org/users/auth/github [3] https://gitlab.haskell.org/profile/keys # Updating your development environment You can updated existing working directory (assuming the usual upstream remote name of `origin`) for the new upstream repository location by running the following: git remote set-url origin https://gitlab.haskell.org/ghc/ghc.git git remote set-url --push origin git@gitlab.haskell.org:ghc/ghc This is all that should be necessary; a quick `git pull origin master` should verify that everything is working as expected. # Continuous integration Continuous integration is now provided by GitLab's native continuous integration infrastructure. We currently test a variety of configurations, including many that neither Phabricator nor CircleCI/Appveyor previously tested (see [1] for an example run): * With the make build system: * x86_64/Linux on Fedora 27, Debian 8, and Debian 9 * i386/Linux on Debian 9 * aarch64/Linux on Debian 9 (currently broken due to a variety of issues) * x86_64/Windows * x86_64/Darwin * x86_64/Linux on Debian 9 in a few special configurations: * unregisterised (still a bit fragile due to #16085) * integer-simple * building GHC with -fllvm * With Hadrian: * x86_64/Linux on Debian 9 * x86_64/Windows (currently broken due to #15950) We also run a slightly larger set of jobs on a nightly basis. Note that binary distributions are saved from most builds and are available for download for a few weeks (we may put in place a longer retention policy for some builds in the future). There are admittedly a few kinks that we are still working out, particularly in the case of Windows (specifically the long build times seen on Windows). If you suspect you are seeing spurious build failures do let us know. To make the best use of our limited computational resources our CI builds occur in three stages: * lint: the style and correctness checkers which would previously be run by `arc lint` and `git push` * build: Debian 9 Linux x86_64 built with make and Hadrian * full-build: the remaining configurations If a build fails at an earlier phase no further phases will be run. [1] https://gitlab.haskell.org/ghc/ghc/pipelines/568 # Structuring your merge request With the transition to GitLab GHC is moving to a model similar to that used by GitHub. If you have a Differential on Phabricator we will finish review there. However, please post new patches as merge requests on GitLab. Note that Phabricator and GitLab have quite different models for handling patches. Under Phabricator a Differential is a single patch with no further structure; larger changes can be composed of multiple dependent Differentials. Under GitLab's model a merge request is a git branch consisting of one or more patches. Larger changes can be handled in one of two ways: a. a set of dependent merge requests, each of which to be squashed when merged. b. a single branch with each atomic change made in a single, buildable commit Due to the difficulty of maintaining dependent merge requests, I would recommend that contributors making larger changes use method (b). # Submitting your merge request for review Depending upon whether you have push rights to the GHC repository there are two ways to submit a merge request: * if you have push access you can push a branch directly to git@gitlab.haskell.org:ghc/ghc.git and open merge request. In this case please do follow the usual branch naming conventions: * prefix all branch names with `wip/` * if you are fixing a particular ticket consider using the name `wip/TNNNN` * if not you can create a fork using the "Fork" button on the project page [1] and push your branch there In either case after you have pushed your branch open a merge request against ghc/ghc [2]. [1] https://gitlab.haskell.org/ghc/ghc/forks/new [2] https://gitlab.haskell.org/ghc/ghc/merge_requests/new # Reviewing and merging merge requests As always, all contributors are encouraged to help review proposed changes. If you are unfamiliar with GitLab's review interface please see GitLab's user documentation [1]. Here are a few quick highlights for those who are familiar with GitHub but haven't yet used GitLab: * As with GitHub, GitLab supports both inline and out-of-line comments. * Comments that are actionable (known as "discussions") can be marked as resolved and collapsed. * Comments can be left on both changed and unchanged lines * Revisions of a merge request can be viewed and compared using the two drop-down menus at the top of the Changes tab * Merge requests can require approvals from particular users before considered as mergable * Merge requests can be placed in "merge when CI passes" state, which will cause merge requests to be merged as soon as they are green From this point onward all changes to GHC will be merged via GitLab's merge requests facility and must pass CI before being merged. To ensure that GHC's git history remains linear ghc/ghc will use GitLab's "fast-forward without a merge commit" merge strategy. Consequently you will be asked to rebase merge requests which are not fast-forward merges before merging (a convenient "Rebase" button will appear if the rebase can be carried out without conflicts. [1] https://gitlab.com/help/user/discussions/index.md#discussions # Status of the Trac migration Tobias will be continuing work on the Trac ticket migration after a bit of a holiday break. Hopefully by mid-January we will be able to move forward on this part of the migration; I will share more details about this as they develop. In the meantime, users of Trac should check and possibly update the email address associated with their account [1]. This address will be used to correlate Trac users with their GitLab equivalents so the correctness of this address will be important in preserving attribution information during the Trac import. [1] https://ghc.haskell.org/trac/ghc/prefs # Next steps We are actively working on cleaning up a few remaining issues with CI: * build times are still very long on Windows, despite the fact that we are only building the `quick` build flavour on that platform; consequently GitLab CI Windows builds do sometimes timeout when we are faced with long build queues. * we at times run low on disk space on our Windows builder runners, resulting in occasional spurious build failures * Appveyor builds (which are supposed to supplement the native GitLab builds) rarely seem to finish GitLab upstream has been incredibly supportive of our transition effort and has expressed interest in assisting us with issues that we encounter. Our current requests can be found on our migration effort's tracking ticket [1]. If you find any additional bugs or workflows that could be improved please do let me know and I can raise the matter with GitLab. [1] https://gitlab.com/gitlab-org/gitlab-ce/issues/55039 # Acknowledgments I would like to acknowledge several parties for their contributions to this effort: * Packet.net and Google X for their generous donation of hosting for continuous integration and web hosting * GitLab and their Open Source program for many productive discussions, their generous support, and the GitLab Ultimate license used by gitlab.haskell.org. * Davean Scies for his help procuring the hosting services that power our continuous integration. * Tweag.io for their offer of help and advice * Matthew Pickering, Alp Mestangullari, Tobias Dammers for their work in setting up the new instance, sorting out the details of the migration, and debugging problems when they arose Finally, thanks to GHC's contributors for their patience during this transition; it has been a long process which has stolen a significant amount of attention from other matters. My apologies we have been a bit less responsive than usual in code review and ticket triage over the past month or two. Regardless, I am hopeful that this wait will be worthwhile. # Final thoughts This is not only a milestone for the GitLab migration but also for GHC itself. For the first time GHC has fully-automated testing, proposed patch CI, and release generation across the full range of Tier 1 configurations it supports, with passing builds in all cases. We are very excited to begin this next chapter of GHC's development and are looking forward to your feedback on how we can further improve our new infrastructure. Onward and upwards! Cheers, - Ben

9 18

How to use Data.Compact.Serialize
by Christopher Done 07 Jan '19

07 Jan '19

Hi, On the docs for Data.Compact.Serialize it says: http://hackage.haskell.org/package/compact-0.1.0.1/docs/Data-Compact-Serial… > Our binary representation contains direct pointers to the info tables > of objects in the region. This means that the info tables of the > receiving process must be laid out in exactly the same way as from the > original process; in practice, this means using static linking, using > the exact same binary and turning off ASLR. It seems to me that in order to use this module in any practical way (i.e. write to file from one process, and then a later run of the process reads it), you need to know a special way to build your binary which isn't fully described here. What flags, for example, should be passed to GHC to make this viable? * Turning off ASLR on Linux is done by writing 0 to /proc/sys/kernel/randomize_va_space, which applies to all programs. That's not the most isolated way to deploy an app, but I discovered that you can set this per process here: https://askubuntu.com/a/507954 * To compile GHC programs statically, use -optl-static -optl-pthread. So in total the example would be: $ stack build compact compact-0.1.0.1: download compact-0.1.0.1: configure compact-0.1.0.1: build compact-0.1.0.1: copy/register Example file from docs: $ cat main.hs {-# LANGUAGE TypeApplications #-} import System.Environment import Data.Compact import Data.Compact.Serialize main = do arg:_ <- getArgs case arg of "write" -> do orig_c <- compact ("I want to serialize this", True) writeCompact @(String, Bool) "somefile" orig_c "read" -> do res <- unsafeReadCompact @(String, Bool) "somefile" case res of Left err -> fail err Right c -> print (getCompact c) Compiling statically: $ stack ghc -- -optl-static -optl-pthread main.hs [1 of 1] Compiling Main ( main.hs, main.o ) Linking main ... [...hundred more warnings like this ...] Check that it's static: $ ldd main not a dynamic executable Write the file: $ setarch `uname -m` -R ./main write Read the file: $ setarch `uname -m` -R ./main read ("I want to serialize this",True) Can a GHC dev confirm that this is the proper way to do this? If so, I'll contribute this little example as documentation to the compact package. Cheers

3 3

Default options for -threaded
by lonetiger＠gmail.com 04 Jan '19

04 Jan '19

Hi All, A user on https://ghc.haskell.org/trac/ghc/ticket/11054 has asked why -N -qa isn’t the default for -threaded. I’m afraid I don’t know a good reason why it’s not. Can anyone help shed some light on this? If there is no good reason, Would anyone object to making it the default? Thanks, Tamar

7 10

Type family equation violates injectivity?
by Dominick Samperi 04 Jan '19

04 Jan '19

When I use v8.6.3 of GHC under Ubuntu to install the inline-r package I get the error "Type family equation violates injectivity annotation," and a type variable on the LHS cannot be inferred from the RHS, due to the lack of injectivity (I suppose). On the other hand, v8.0.2 of GHC (shipped with Haskell Platform under Ubuntu) does not have this problem (it has other problems). Has something changed in the latest version of the compiler that might cause this? Possible work-around? FYI, the line that triggers the error is: type instance G.Mutable (W t ty s) = Mutable.W t ty The variable that cannot be inferred is 's'. Thanks, Dominick

4 12

Phab and trac are gonna stay online in read only mode right?
by Carter Schonwald 30 Dec '18

30 Dec '18

I’m still fuzzy on the long term hosting plans , but I worry that If we don’t keep them online as sort of read only sites with maybe a “look at this new gitlab ” top bar, that a lot of links to stuff will start dying Is there a post wiki / ticket migration plan for this ?

1 0