June 2020 - ghc-devs - Haskell.org

Merge window logistics and relaxation of submodule policy
by Ben Gamari 22 Jun '20

22 Jun '20

Hi everyone, At this point we are in the home stretch of the GHC 8.12 preparation phase. The merge queue has progressed a bit more slowly than I would have liked over the last few weeks but nevertheless I think we are converging. Regardless, if you have a patch which you believe should go in to 8.12 and doesn't already bear the %8.12 GitLab milestone, please do let me know. # Submodule policy One matter that is currently holding us up is a few submodules which haven't yet had necessary patches merged upstream. As there hasn't yet been much motion on the part of maintainer, I propose that we temporarily relax our policy that GHC submodules refer to upstream commits. That is, we allow GHC to temporarily track a non-upstream branch, ensuring that the branch is merged upstream before the final GHC release. Are there any objections to this policy relaxation? It will require a bit more care on our part, but I think it is a sensible compromise way to give upstream maintainers the time they need while ensuring that GHC isn't subject to unpredictable schedules. Cheers, - Ben

2 1

GitLab status
by Ben Gamari 22 Jun '20

22 Jun '20

Hi everyone, As you may have noticed, yesterday was marked by rather consistent CI failures. I believe I have now fixed the root cause but do let me know if you see any further issues. Below I have included a brief post-mortem describing the incident and what we plan to do to prevent it recurring in the future. Cheers, - Ben # Post-mortem Early Friday morning our storage provider experienced a hiccup which rendered the volume which backed our GitLab repositories unavailable for a few minutes. The interruption was long enough that the filesystem remounted as read-only. This caused a bit of filesystem damage affecting a handful of objects in the ghc/perf-notes repository. This resulted in CI failures when attempts to `git push` to this repository failed. To address this I started by ensuring we had an up-to-date backup of our data and began sorting through the various observed errors. Once I had established that the storage volume had been interrupted I went looking for additional corruption. A integrity check of all GitLab repositories revealed no further corruption. This isn't terribly surprising given that the perf notes repository sees the most commit traffic of all repositories hosted on GitLab. While it would likely be possible to recover the corrupted perf-notes objects from clones on the CI builders, I deemed this to be not worth the effort given that these commits hold replaceable performance metric data and the last good commit appears to have been produced mere hours prior to the corrupted HEAD commit. Consequently, I rather reverted the ghc-notes repository to the last-known-good commit (8154013bfdce86fedf2863cb96ccbb723f1144f8). # Planned changes for future mitigation While this incident didn't result in any significant data loss, it was nevertheless a significant headache and resulted in CI failing for the better part of a day. Moreover, this isn't the first time that the network block storage backing GitLab has let us down. At this point, I have lost confidence that network block storage can be trusted. For this reason, in the future we will eliminate this failure mode by adjusting our deployment to avoid relying on network block storage for any deployment-critical data. We hope that we will be able to carry out this change in the coming weeks.

1 2

[ghc-lib] internal error after removal of integer-simple
by Shayne Fletcher 20 Jun '20

20 Jun '20

With the recent MR that removes integer-simple in favor of ghc-bignum, I find that I get a runtime failure when I try to use ghc-lib to generate core: ``` # Running: stack --no-terminal exec -- mini-compile examples/mini-compile/test/MiniCompileTest.hs examples/mini-compile/test/MiniCompileTest.hs:66:5: error: * GHC internal error: `One' is not in scope during type checking, but it passed the renamer tcl_env of environment: [628 :-> ATcTyCon TrName :: *, 62b :-> APromotionErr RecDataConPE, 62e :-> APromotionErr RecDataConPE] * In the definition of data constructor `TrNameS' In the data declaration for `TrName' | 66 | = TrNameS Addr# -- Static | ^^^^^^^^^^^^^ mini-compile: GHC internal error: `One' is not in scope during type checking, but it passed the renamer tcl_env of environment: [628 :-> ATcTyCon TrName :: *, 62b :-> APromotionErr RecDataConPE, 62e :-> APromotionErr RecDataConPE] ``` Anyone have any pointers on what is going wrong and what I should be looking at? -- *Shayne Fletcher* Language Engineer */* +1 917 699 7663 *Digital Asset* <https://digitalasset.com/>, creators of *DAML <https://daml.com/>* -- This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html <http://www.digitalasset.com/emaildisclaimer.html>. If you are not the intended recipient, please delete this message.

4 9

Gitlab is stuck
by Simon Peyton Jones 19 Jun '20

19 Jun '20

Something is badly wrong with gitlab. I'm trying to push a branch. Is some disk full? I am fully stalled. Thanks Simon bash$ git push --force Counting objects: 79, done. Delta compression using up to 20 threads. Compressing objects: 100% (65/65), done. Writing objects: 100% (79/79), 46.21 KiB | 3.55 MiB/s, done. Total 79 (delta 68), reused 16 (delta 14) error: remote unpack failed: unable to create temporary object directory remote: warning: The last gc run reported the following. Please correct the root cause remote: and remove gc.log. remote: Automatic cleanup will not be performed until the file is removed. remote: remote: warning: There are too many unreachable loose objects; run 'git prune' to remove them. remote: To gitlab.haskell.org:ghc/ghc.git ! [remote rejected] wip/T18282 -> wip/T18282 (unpacker error) error: failed to push some refs to 'git@gitlab.haskell.org:ghc/ghc.git'

2 3

Gitlab corrupt
by Simon Peyton Jones 19 Jun '20

19 Jun '20

Ben Something is amiss. "corrupt" sounds bad! Simon bash$ git fetch error: garbage at end of loose object '62be92e5be07b34fc25c77e36223268c4a6eadec' fatal: loose object 62be92e5be07b34fc25c77e36223268c4a6eadec (stored in ./objects/62/be92e5be07b34fc25c77e36223268c4a6eadec) is corrupt fatal: The remote end hung up unexpectedly

2 1

Receiving type information from environment instead of hardcoding.
by Rinat Stryungis 19 Jun '20

19 Jun '20

Hi. I have a question about a possible way of unification of Nat and Natural. I've almost done that, but only in case of using integer-gmp. If I use integer-simple there is a completely different definition of Natural. How I construct now naturalTyCon (to make `naturalTy` to use it instead of `typeNatKind`) : ```naturalTyCon :: TyCon naturalTyCon = pcTyCon naturalTyConName Nothing [] [natSDataCon,natJDataCon] natSDataCon :: DataCon natSDataCon = pcDataCon natSDataConName [] [wordPrimTy] naturalTyCon etc... ``` Now I have to check`DynFlags` in a few places to reimplement `naturalTyCon` in case of using `integer-simple`. Is there a way to avoid hardcoding of `naturalTy`? My colleague said that it would be nice to get `naturalTy` from an environment by something like `lookupTyCon`, but there are many functions whose don't use any environment like functions from `typeNatTyCons` list in `GHC.Builtin.Types.Literals`. Now I am going to use `DynFlags` checking, but it looks like an ugly way... -- Best regards. Rinat Striungis

3 3

Gitlab 503
by Simon Peyton Jones 19 Jun '20

19 Jun '20

I'm getting a lot of "Gitlab is unavailable" 503 messages, after editing a wiki page. Does anyone know what is happening? Simon

1 0

Linear types performance characterisation
by Ben Gamari 17 Jun '20

17 Jun '20

Hi everyone, Last week I discussed the plan for merging the LinearTypes branch into GHC 8.12 with Arnaud, Richard, Andreas, and Simon. Many thanks to all of them for their respective roles in pushing this patch over the finish line. One thing that we wanted to examine prior to merge is compiler performance across a larger collection of packages. For this I used the head.hackage patch-set, comparing the Linear Types branch with its corresponding base commit in `master`. Here I will describe the methodology used for this comparison and briefly summarize the (happily, quite positive) results. # Methodology I collected total bytes allocated (as reported by the runtime system), elapsed runtime (as reported by the runtime system), and instructions (as reported by `perf stat`) of head.hackage builds in two configurations: * the `opt` configuration * the `noopt` configuration, which passed `--disable-optimisation` to cabal-install These configurations were evaluated on two commits: * `master`: 2b792facab46f7cdd09d12e79499f4e0dcd4293f * `linear-bang`: 481cf412d6e619c0e47960f4c70fb21f19d6996d Unfortunately, the `noopt` configuration appears to be be affected by a few cabal-install bugs [1,2] and consequently some packages may *still* be compiled with optimisation, so take these numbers with a grain of salt. The test environment was a reasonably quiet Ryzen 7 1800X with 32 GBytes of RAM. The test was run by first building the two tested commits in Hadrian's default build flavour. The head.hackage CI driver was then invoked as follows: # Don't parallelize for stable performance measurements export CPUS=1 export USE_NIX=1 export EXTRA_HC_OPTS=-ddump-timings export COLLECT_PERF_STATS=1 mkdir -p runs # master export GHC=/home/ben/ghc/ghc-compare-2/_build/stage1/bin/ghc ./run-ci --cabal-option=--disable-optimisation mv ci/run runs/master-noopt ./run-ci mv ci/run runs/master-opt # linear-bang export GHC=/home/ben/ghc/ghc-compare-1/_build/stage1/bin/ghc ./run-ci --cabal-option=--disable-optimisation mv ci/run runs/linear-noopt ./run-ci mv ci/run runs/linear-opt As we are building all packages (nearly 300 in total) serially, the full run takes quite a while (around 8 hours IIRC). The final run of this test used head.hackage commit e7e5c5cfbfd42c41b1e62d42bb18483a83b78701 (on the `rts-stats` branch). # Results I examined several different metrics of compiler performance * the total_wall_seconds RTS metric gives an picture of overall compilation effort * time reported by -ddump-timings, summed by module, gives a slightly finer-grained measurement of per-module compilation time * the RTS's bytes_allocated metric gives overall compiler allocations * the RTS"s max_bytes_used metric gives a sense of AST size (and potentially the existence of leaks) To cut straight to the chase, the measurements show the following: metric -O0 -O1 ------------------- --------- ---------- total_wall_seconds +0.3% +0.6% total_cpu_seconds +0.3% +0.7% max_bytes_used +4.2% +4.8% GC_cpu_seconds +1.5% +2.1% mut_cpu_seconds no change no change sum(per-module-time) +4.2% +4.2% sum(per-module-alloc) +0.8% +0.8% There are a few things to point out here: the overall change in compiler runtime is thankfully quite reasonable. However, max_bytes_used increases rather considerably. This seems to give rise to an appreciable regression in GC time. It would be interesting to know whether this can be improved with optimisation to data representation. The fact that the cumulative per-module metrics didn't change between -O0 and -O1 indicate to me that there is a methodological problem which needs to be addressed in the test infrastructure. I investigated this a bit and have a hypothesis for what might be going on here; nevertheless, in the interest of publishing these measurements I'm ignoring these measurements for the time being. I have attached the Jupyter notebook that gave rise to these numbers. This gives a finer-grained breakdown of the data including histograms showing the variance of each metric. Perhaps this will be helpful in better understanding the effects. I would be happy to share my run data as well although it is a bit large. All-in-all, the Tweag folks have done a great job in squashing the performance numbers noticed a few weeks ago. The current numbers look quite acceptable for GHC 8.12. Congratulations to Arnaud, Krzysztof, and Richard on landing this feature! I'm very much looking forward to see what the community does with it in the coming years. Cheers, - Ben [1] https://github.com/haskell/cabal/issues/5353 [2] https://github.com/haskell/cabal/issues/3883

2 1

Search in GitLab
by Simon Peyton Jones 15 Jun '20

15 Jun '20

Does anyone know how to search better in GitLab. Currently I'm using the standard GitLab search. I'm searching for "<<loop>>" where I intend the quotes meaning exactly that string as usual in a search term. But I get lots of results mentioning loop, without the angle brackets. Moreover I want to sort the results by date or ticket number, and I can't see how to do that. Does Google index our repo? Can I use Google to search it somehow? Thanks Simon

6 8

Looking for a recent error reporting change
by Chris Smith 15 Jun '20

15 Jun '20

Hi. In GHC 8.6.5, if I compile a main module that both fails to define `main` and also contains other errors, the other errors are not reported because GHC aborts after realizing that there is no `main`. In GHC 8.10.1, all errors are reported. I'm looking for where this change was made, in the hopes that it's easy to backport to a local branch of GHC 8.6.5. Does anyone know off-hand?

4 4