(re-sending with replay-all this time) Hi Simon, thank you for your concern. Regarding the compiler compilation: I often have to re-run ./configure and ./boot between development phases (usually spaced by one or two weeks), which prevents me from benefiting from `--freeze1` from the get-go. The flavour I use is `Quick`, as it is advertised on https://ghc.dev. The technique I use now is to restrict the `-j` parameter to 1 or 2. I can go above for night builds, but if I want to keep my system usable, this is how it goes. This takes 47m07s according to Hadrian. Moreover, modern computers have a BIOS setting that shuts down the computer when the temperature is too high (between 70°C and 100°C I believe). I ended up upping this limit in my BIOS some time ago, but I wish I didn't need to. (and yes I do frequently remove the dust ;) Regarding documentation contribution, The time it takes to run `hadrian/build -j --flavour=Quick` *plus* `hadrian/build -j2 --freeze1 --flavour=Quick docs --docs=no-sphinx-pdfs ` in order to render the haddocks is way too high for occasional contributors. I started a process while writing this email, in order to get fresh and accurate numbers, and here is what I got in return: juil. 19 19:37:21 elatha systemd[1759]: app-kitty-dedaa3d5a1ee43dd8b2552b7afd53cd5.scope: systemd-oomd killed 36 process(es) in this unit. juil. 19 19:37:21 elatha systemd-oomd[682429]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-kitty-dedaa3d5a1ee43dd8b2552b7afd53cd5.scope due to memory used (16192806912) / total (16358338560) and swap used (8102125568) / total (8589930496) being more than 90.00% Now regarding HLS, I remember that a point of marketing for it is that it supports GHC. Considering the very skilled people who hold the wheel of HLS development, my first (nor second) reflex isn't to doubt them. So yes I believe I must disable HLS when working on GHC, but then how do we spread this kind of information / work-around while at the same time saying "but HLS is still reliable, pinky swear, it's just that it doesn't scale". I do not doubt that HLS does some heavy stuff and I'm truly grateful for all the times where it detects unused imports, unused extensions, and even inserts import lines for me! But I find it hard to promote HLS whilst also putting an asterisk that says "Provided that you can afford it". This is pretty much all I can observe. I'd be more than happy to have better observability into the build process, though! I'm not too desperate for the future, because chatting with Bryan gave me hope for progress in that area. Le 19/07/2022 à 18:29, Simon Peyton Jones a écrit :

That's bad Hecate. We need GHC to be fun to work with, not a pain.

Can you be (much) more specific?  The more concrete the problem, the more likely we can address it.

e.g. What if you don't use HLS?  Or maybe Hadrian is building much more than you need? It would be super helpful to have more information.  There may be things we can't reasonably address (e.g. make a small, light, non-optimising compiler instead, throwing away most of the code base) but I bet that sheer size isn't the only factor.

Thanks!

Simon

On Tue, 19 Jul 2022 at 17:21, Hécate wrote:

Hello ghc-devs,

I hadn't made significant contributions to the GHC code base in a while, until a few days ago, where I discovered that my computer wasn't able to sustain running the test suite, nor handle HLS well.

Whether it is my OS automatically killing the process due to oom-killer or just the fact that I don't have a war machine, I find it too bad and I'm frankly discouraged. This is not the first time such feedback emerges, as the documentation task force for the base library was unable to properly onboard some people from third-world countries who do not have access to hardware we'd consider "standard" in western Europe or some parts of North America. Or at least "standard" until even my standard stuff didn't cut it anymore.

So yeah, I'll stay around but I'm afraid I'm going to have to focus on projects for which the feedback loop is not on the scale of hours , as this is a hobby project.

Hope this will open some eyes.

Cheers, Hécate

-- Hécate ✨ 🐦: @TechnoEmpress IRC: Hecate WWW: https://glitchbra.in RUN: BSD

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

-- Hécate ✨ 🐦: @TechnoEmpress IRC: Hecate WWW:https://glitchbra.in RUN: BSD

Hi Moritz, thank you very much for this bullet list. No doubt my own development workflow could be improved. :) Le 19/07/2022 à 18:34, Moritz Angermann a écrit :

Hi Hecate,

I don't think this is entirely fair in either direction. So sharing my personal experience might shed some light.  I've often worked on GHC on fairly weak machines.  However the ability to use HLS on GHC or even the ability to load GHC into GHCi are fairly recent additions.

I don't run the full test-suite either much.

The general development experience has more been closer to this: - pick an issue I want to work on - checkout the relevant branch (or master) - kick off a ghc build (hadrian) - start looking for the relevant code in GHC to address this. - build a tiny reproducer (if possible, or run the relevant test from the test-suite if available) -- once the initial ghc is build. - hack on the codebase; rebuild (subsequent rebuilds are fairly fast) - retry the reproducer, iterate until done.

Most of my development has been without much codelevel help and at most a syntax highlighter. This is decidedly different from the experience you can have working on haskell libraries with the availability of ghcid, hls, ... would it be nice if ghc development would be that nice as well? I'd assume so, I've just never even tried.

Cheers,  Moritz

On Tue, 19 Jul 2022 at 18:21, Hécate wrote:

Hello ghc-devs,

I hadn't made significant contributions to the GHC code base in a while, until a few days ago, where I discovered that my computer wasn't able to sustain running the test suite, nor handle HLS well.

Whether it is my OS automatically killing the process due to oom-killer or just the fact that I don't have a war machine, I find it too bad and I'm frankly discouraged. This is not the first time such feedback emerges, as the documentation task force for the base library was unable to properly onboard some people from third-world countries who do not have access to hardware we'd consider "standard" in western Europe or some parts of North America. Or at least "standard" until even my standard stuff didn't cut it anymore.

So yeah, I'll stay around but I'm afraid I'm going to have to focus on projects for which the feedback loop is not on the scale of hours , as this is a hobby project.

Hope this will open some eyes.

Cheers, Hécate

-- Hécate ✨ 🐦: @TechnoEmpress IRC: Hecate WWW: https://glitchbra.in RUN: BSD

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

-- Hécate ✨ 🐦: @TechnoEmpress IRC: Hecate WWW:https://glitchbra.in RUN: BSD

Hey everyone! I'm not a frequent contributor but I observed similar challenges as Hécate. I notice couple points. ### HLS and other editor integrations I've never tried HLS for GHC development but it absolutely chokes on Cabal for me (a $2K laptop), so I'm not surprised it's having troubles with GHC. I've never tried to dig into it, but I heard before that disabling plugins is a good start. Ghcid (after the introduction of ghc-in-ghci) was interesting but still on a slow side. I once tried to generate ETAGS and use them from Emacs (with plain haskell-mode): this was quite nice. As Moritz, I didn't use much above syntax coloring, but ETAGS allowed jumping to definitions, which is important. Maintaining tags wasn't fun, on the other hand. In all fairness, I think that's an issue with HLS more than with GHC. ### Build Times I have been using a dedicated server for this, but this still was painful at times (even git clone takes non-negligible amount of time, and I never got used to git worktree because of a hoop you have to jump over, which I already forgot but I know it can be looked up in Andreas Herrmann's presentation on developing GHC). I'm surprised no one seems to try to challenge the status quo. Hadrian is a Shake application. How is Cloud Shake doing? In the era of Nix and Bazel you start assuming niceties like remote caching. It'd be great to improve on this front as it just feels very wrong rebuilding master again and again on every contributor's computer. Especially after so much effort put into GHC modularity, which, I believe, should make it easier to cache. It's sad that GHC still needs ./boot && ./configure: this can preclude any remote caching technology that I can imagine. At one point it seemed like configure could go into Hadrian, but it didn't really happen. In the bright future I see Hadrian doing the whole job and using remote cache on GitLab CI. Not sure how big of a task that would be. -- Cheers, Artem On Tue, 19 Jul 2022 at 15:11, Ben Gamari wrote:

Hécate writes:

...

Hello ghc-devs,

I hadn't made significant contributions to the GHC code base in a while, until a few days ago, where I discovered that my computer wasn't able to sustain running the test suite, nor handle HLS well.

Whether it is my OS automatically killing the process due to oom-killer or just the fact that I don't have a war machine, I find it too bad and I'm frankly discouraged.

Do you know which process was being killed? There is one testsuite tests that I know of which does have quite a considerable memory footprint (T16992) due to its nature; otherwise I would expect a reasonably recent machine to pass the testsuite without much trouble. It's particularly concerning if this is a new regression; is this the first time you have observed this particular failure?

...

This is not the first time such feedback emerges, as the documentation task force for the base library was unable to properly onboard some people from third-world countries who do not have access to hardware we'd consider "standard" in western Europe or some parts of North America. Or at least "standard" until even my standard stuff didn't cut it anymore.

So yeah, I'll stay around but I'm afraid I'm going to have to focus on projects for which the feedback loop is not on the scale of hours , as this is a hobby project.

Hope this will open some eyes.

Hi Hécate,

I would reiterate that the more specific feedback you can offer, the better.

To share my some of my own experience: I have access to a variety of hardware, some of which is quite powerful. However, I find that I end up doing much of my development on my laptop which, while certainly not a slouch (being a Ryzen 4750U), is also not a monster. In particular, while a fresh build takes nearly twice as long on my laptop than some of the other hardware I have, I nevertheless find ways to make it worthwhile (due to the ease of iteration compared to ssh). If you routinely have multi-hour iteration times then something isn't right.

In particular, I think there are a few tricks which make life far easier:

* Be careful about doing things that would incur significant amounts of rebuilding. This includes:

* After modifying, e.g., `compiler/ghc.cabal.in` (e.g. to add a new module to GHC), modify `compiler/ghc.cabal` manually instead of rerunning `configure`.

* Be careful about pulling/rebase. I generally pick a base commit to build off of and rebase sparingly: Having to stop what I'm doing to wait for full rebuild is an easy way to lose momentum.

* Avoid switching branches; I generally have a GHC tree per on-going project.

* Take advantage of Hadrian's `--freeze1` flag

* Use `hadrian/ghci` to typecheck changes

* Use the stage1 compiler instead of stage2 to smoke-test changes when possible. (specifically, using the script generated by Hadrian's `_build/ghc-stage1` target)

* Use the right build flavour for the task at hand: If I don't need a performant compiler and am confident that I can get by without thorough testsuite validation, I use `quick`. Otherwise, plan ahead for what you need (e.g. `default+assertions+debug_info` or `validate`)

* Run the fraction of the testsuite that is relevant to your change. Hadrian's `--test-way` and `--only` flags are your friends.

* Take advantage of CI. At the moment we have a fair amount of CI capacity. If you think that your change is close to working, you can open an MR and start a build locally. If it fails, iterate on just the failing testcases locally.

* Task-level parallelism. Admittedly, this is harder when you are working as a hobby, but I often have two or three projects on-going at a time. While one tree is building I try to make progress on another.

I don't use HLS so I may be insulated from some of the pain in this regard. However, I do know that Matt is a regular user and he disables most plugins.

I would also say that, sadly, GHC is comparable to other similarly-size compilers in its build time: A build of LLVM (not even clang) takes ~50 minutes on my 8-core desktop; impressively, rustc takes ~7 minutes although it is a considerably smaller compiler (being just a front-end). By contrast, GHC takes around 20 minutes. I know that this doesn't make the cost any easier to bear and I would love to bring this number down, but ultimately there are only so many hours in the day.

I think one underexplored approach to addressing the build-time problem is to look not at the full-build time but rather look for common tasks where we could *avoid* doing a full build (e.g. updating documentation, typechecking `base`, running a "good enough" subset of the testsuite) and find ways to make those workflows more efficient.

Cheers,

- Ben

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Artem Pelenitsyn writes:

Hey everyone! I'm not a frequent contributor but I observed similar challenges as Hécate. I notice couple points.

### HLS and other editor integrations

I've never tried HLS for GHC development but it absolutely chokes on Cabal for me (a $2K laptop), so I'm not surprised it's having troubles with GHC. I've never tried to dig into it, but I heard before that disabling plugins is a good start.

Ghcid (after the introduction of ghc-in-ghci) was interesting but still on a slow side.

I once tried to generate ETAGS and use them from Emacs (with plain haskell-mode): this was quite nice. As Moritz, I didn't use much above syntax coloring, but ETAGS allowed jumping to definitions, which is important. Maintaining tags wasn't fun, on the other hand.

In all fairness, I think that's an issue with HLS more than with GHC.

### Build Times

I have been using a dedicated server for this, but this still was painful at times (even git clone takes non-negligible amount of time, and I never got used to git worktree because of a hoop you have to jump over, which I already forgot but I know it can be looked up in Andreas Herrmann's presentation on developing GHC). I'm surprised no one seems to try to challenge the status quo.

IMHO, `git worktree` is indispensible. Not only does it make cloning cheaper but it makes it trivial to share commits between work trees, which is incredibly helpful when cleaning up branch history, backporting, and other common tasks. I just wish it also worked transparently for submodules.

Hadrian is a Shake application. How is Cloud Shake doing? In the era of Nix and Bazel you start assuming niceties like remote caching. It'd be great to improve on this front as it just feels very wrong rebuilding master again and again on every contributor's computer. Especially after so much effort put into GHC modularity, which, I believe, should make it easier to cache.

Sadly using Cloud Shake in Hadrian ran into some rather fundamental difficulties: * GHC has native dependencies (namely, the native toolchain, ncurses, gmp, and possible libdw and libnuma). If everyone were to use, e.g., ghc.nix this could be largely mitigated, but this isn't the world in which we live. * GHC is a bootstrapped compiler. Consequently, most changes will invalidate well over half of the build cache (that is, everything built by stage 1). This significantly limits the benefit that one could gain from Cloud Shake, especially since in the typical development workflow the stage 1 build (which is where most of the caching benefit would be seen) is a rather small cost (IIRC it takes around 5 minutes to make it to the stage 2 build on my machine). One might think that we could simply "freeze" the stage 1 compiler, but in general this is not safe. For instance, it would break subtly on any change to known keys, the interface file format, the ABI, or primop definitions.

It's sad that GHC still needs ./boot && ./configure: this can preclude any remote caching technology that I can imagine. At one point it seemed like configure could go into Hadrian, but it didn't really happen.

I don't see us moving away from `configure` (or something like it) as long as GHC has native dependencies. Having a clear separation between "configuration" and "building" is very much necessary to maintain sanity. Cheers, - Ben

Artem Pelenitsyn writes:

Thanks Ben, very interesting, especially the cloud Shake stuff.

...

If everyone were to use, e.g., ghc.nix this could be largely mitigated, but this isn't the world in which we live.

I don't know why you'd want everyone to use it necessarily to improve things. If there was a clear statement that you may elect to use ghc.nix and get X% speedup, that would be a good start. And then people can decide, based on their setup and their aversion to tools like Nix, etc. in general.

The real issue is that currently you don't benefit much from ghc.nix because the main performance sink is the GHC tree itself. The way out is to use a cloud build system, most likely Cloud Shake, which, as you describe, has a couple of issues in this case:

1) Native dependencies. This should be possible to solve via Nix, but unfortunately, this is not quite there yet because Shake doesn't know about Nix (afaik). I think to actually get there, you'd need some sort of integration between Nix and Shake akin to what Tweag built for Nix and Bazel (cf. rules_nixpkgs [1]). Their moto is: Nix for "external" or "system" components, and Bazel for "internal" or "local" ones.

I disagree. Caching is quite feasible while maintaining a clear division between configuration and the build system. Today, the only channel of communication between `configure` and Hadrian is `hadrian/cfg/system.config`. If `ghc.nix` is doing its job correctly then two invocations of `./configure` in nix-shell on two different machines should end up with the same `hadrian/cfg/system.config`. Further, if two trees have the same contents in that file, then they can share build artifacts [1]. However, to reiterate, the real problem here is the one below:

2) Bootstrapping aspect. Maybe this is a challenge for rebuilds after modification, but I think people on this thread were quoting the "time to first build" more. I don't see how avoiding to build master locally after a fresh (worktree) checkout by downloading build results from somewhere, connects to bootstrapping. I think it doesn't.

If you merely want to build `master` then indeed caching would work fine. However, in that case you could have also just downloaded a binary distribution from GitLab. The problem is that usually the reason that you want to build `master` is that you then want to *modify* it. In general, a modification of `master` will require rebuilding some subset of the stage1 ghc, which will then require a full build of stage2 (which includes GHC, as well as all of the boot libraries, `base`, etc.). The latter would see zero cache hits since one of the inputs, the stage 1 GHC, has changed. Unfortunately, the latter is also well over half of the build effort.

As for rebuilds. People are already using --freeze1 (you suggested it earlier in this very thread!),

Yes, but they are doing so explicitly after having already built their branch to produce a consistent stage1 compiler. If you checkout `master`, build stage 1, switch to some arbitrary branch, and attempt to build stage2 with --freeze1, chances are you will end up with a broken compiler. In the best case this will manifest as a build failure. However, there is a non-negigible possibility that the outcome is far more sinister (e.g. segmentation faults).

so I don't see how saying "freezing stage 1 is dangerous even if faster" connects to practise of GHC development. Of course, you may not find a remote cache with relevant artefacts after local updates, but that's not the point. The point is to not have to build `master`, not `feaure-branch-t12345`. Rebuilds should be rather pain-free in comparison.

For safe caching we must be certain that the build graph is accurate and complete. However, we know with certainty that it currently is not and fixing this requires real implementation effort (David Eichmann spent a few months on this problem in 2019; see #16926 and related tickets). Consequently, we must weigh the benefit of caching against the development cost. Currently, my sense is that the benefit would be some subset of the stage 1 build could be shared some of the time. This strikes me as a rather small benefit compared to the cost. Of course, we would love to have help in addressing #16926. In principle having build caching would be nice; however, at the moment we just don't believe that putting precious GHC team resources towards that goal is the most effective way to serve users. If someone were to come along and start chipping away at the #16926, we would be happy to advise and assist. Cheers, - Ben [1] Strictly speaking, I don't believe this is quite true today since the absolute path of the working tree almost certainly leaks into the build artifacts. However, in principle this could be fixed.

Rodrigo Mesquita writes:

Dear Ben,

The list of tips you put together is quite nice.

I suggest we add it to hadrian’s wiki page under a “Tips for making your life easier” section (as is, it is already useful! at least I learned something new).

Hi Rodrigo, I'm happy to hear that it was useful! Indeed I always struggle to know where to put this information; our developer documentation is quite scattered and it's hard to know where new contributors will enter it. Nevertheless, the page that you mention seems like an obvious place for these tips. Incidentally, I have long felt that migrating the Wiki documentation into a comprehensive developer's guide (see #18853) in the repository (much like the Rustc Book [1]) would be a useful direction of travel. It would be both prompt us review (and, in some cases, retire) existing and give us the opportunity to structure the documentation in a more coherent manner. Moreover, the fact that it would be part of the repository would both give us the benefit of code review and make it easier to ensure that the documentation remains consistent with the code. This is obviously a longer-term goal, but one which it would be great to have help in moving towards. Cheers, - Ben [1] https://doc.rust-lang.org/rustc/contributing.html

This is a great thread. I would love it to conclude with some concrete actions, rather than just petering out. I think one underexplored approach to addressing the build-time problem

is to look not at the full-build time but rather look for common tasks where we could *avoid* doing a full build (e.g. updating documentation, typechecking `base`, running a "good enough" subset of the testsuite) and find ways to make those workflows more efficient.

This is a great example of a concrete step. - Identify common tasks - Write done the procedure for doing that task I suspect that much waiting time is just because we are building too much. e.g. if all you want to do is typeset Haddock docs, you probably don't need the RTS build in threaded-debug way (or even at all). As to the testsuite, 99% of them run fast. If there are some slow culprits that are causing genuine pain, let's identify them. Simon On Tue, 19 Jul 2022 at 20:11, Ben Gamari wrote:

Hécate writes:

...

Hello ghc-devs,

I hadn't made significant contributions to the GHC code base in a while, until a few days ago, where I discovered that my computer wasn't able to sustain running the test suite, nor handle HLS well.

Whether it is my OS automatically killing the process due to oom-killer or just the fact that I don't have a war machine, I find it too bad and I'm frankly discouraged.

Do you know which process was being killed? There is one testsuite tests that I know of which does have quite a considerable memory footprint (T16992) due to its nature; otherwise I would expect a reasonably recent machine to pass the testsuite without much trouble. It's particularly concerning if this is a new regression; is this the first time you have observed this particular failure?

...

This is not the first time such feedback emerges, as the documentation task force for the base library was unable to properly onboard some people from third-world countries who do not have access to hardware we'd consider "standard" in western Europe or some parts of North America. Or at least "standard" until even my standard stuff didn't cut it anymore.

So yeah, I'll stay around but I'm afraid I'm going to have to focus on projects for which the feedback loop is not on the scale of hours , as this is a hobby project.

Hope this will open some eyes.

Hi Hécate,

I would reiterate that the more specific feedback you can offer, the better.

To share my some of my own experience: I have access to a variety of hardware, some of which is quite powerful. However, I find that I end up doing much of my development on my laptop which, while certainly not a slouch (being a Ryzen 4750U), is also not a monster. In particular, while a fresh build takes nearly twice as long on my laptop than some of the other hardware I have, I nevertheless find ways to make it worthwhile (due to the ease of iteration compared to ssh). If you routinely have multi-hour iteration times then something isn't right.

In particular, I think there are a few tricks which make life far easier:

* Be careful about doing things that would incur significant amounts of rebuilding. This includes:

* After modifying, e.g., `compiler/ghc.cabal.in` (e.g. to add a new module to GHC), modify `compiler/ghc.cabal` manually instead of rerunning `configure`.

* Be careful about pulling/rebase. I generally pick a base commit to build off of and rebase sparingly: Having to stop what I'm doing to wait for full rebuild is an easy way to lose momentum.

* Avoid switching branches; I generally have a GHC tree per on-going project.

* Take advantage of Hadrian's `--freeze1` flag

* Use `hadrian/ghci` to typecheck changes

* Use the stage1 compiler instead of stage2 to smoke-test changes when possible. (specifically, using the script generated by Hadrian's `_build/ghc-stage1` target)

* Use the right build flavour for the task at hand: If I don't need a performant compiler and am confident that I can get by without thorough testsuite validation, I use `quick`. Otherwise, plan ahead for what you need (e.g. `default+assertions+debug_info` or `validate`)

* Run the fraction of the testsuite that is relevant to your change. Hadrian's `--test-way` and `--only` flags are your friends.

* Take advantage of CI. At the moment we have a fair amount of CI capacity. If you think that your change is close to working, you can open an MR and start a build locally. If it fails, iterate on just the failing testcases locally.

* Task-level parallelism. Admittedly, this is harder when you are working as a hobby, but I often have two or three projects on-going at a time. While one tree is building I try to make progress on another.

I don't use HLS so I may be insulated from some of the pain in this regard. However, I do know that Matt is a regular user and he disables most plugins.

I would also say that, sadly, GHC is comparable to other similarly-size compilers in its build time: A build of LLVM (not even clang) takes ~50 minutes on my 8-core desktop; impressively, rustc takes ~7 minutes although it is a considerably smaller compiler (being just a front-end). By contrast, GHC takes around 20 minutes. I know that this doesn't make the cost any easier to bear and I would love to bring this number down, but ultimately there are only so many hours in the day.

I think one underexplored approach to addressing the build-time problem is to look not at the full-build time but rather look for common tasks where we could *avoid* doing a full build (e.g. updating documentation, typechecking `base`, running a "good enough" subset of the testsuite) and find ways to make those workflows more efficient.

Cheers,

- Ben

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Oh my, I just discovered that Cheng Shao has recently put something together for working on GHC with Gitpod: https://github.com/TerrorJack/gitpod-ghc However, it looks like it isn't using the Debian images from the GHC CI. Sorry for the noise. Cheers, Steve