March 2021 - ghc-devs

HLint in the GHC CI, an eight-months retrospective
by Hécate 26 Mar '21

26 Mar '21

Hello fellow devs, this email is an activity report on the integration of the HLint[0] tool in the Continuous Integration (CI) pipelines. On Jul. 5, 2020 I opened a discussion ticket[1] on the topic of code linting in the several components of the GHC code-base. It has served as a reference anchor for the Merge Requests (MR) that stemmed from it, and allowed us to refine our expectations and processes. If you are not acquainted with its content, I invite you to read the whole conversation. Subsequently, several Hadrian lint rules have been integrated in the following months, in order to run HLint on targeted components of the GHC repository (the base library, the compiler code-base, etc). Being satisfied with the state of the rules we applied to the code-base, such as removing extraneous pragmata and keywords, it was decided to integrate the base library linting rule in the CI. This was five months ago, in September[2], and I am happy to report that developer friction has been so far minimal. In parallel to this work on the base library, I took care of cleaning-up the compiler, and harmonised the various micro coding styles that have emerged quite organically during the decades of development that are behind us (I never realised how many variations of the same ten lines of pragmata could coexist in the same folders). Upon feedback from stakeholders of this sub-code base, the rules file was altered to better suit their development needs, such as not removing extraneous `do` keywords, as they are useful to introduce a block in which debug statements can be easily inserted. Since today, the linting of the compiler code-base has been integrated in our CI pipelines, without further burdening our CI times. Things seem to run smoothly, and I welcome comments and requests of any kind related to this area of our code quality process. Regarding our future plans, there has been a discussion about integrating such a linting mechanism for our C code-base, in the RTS. Nothing is formally established yet, so I would be grateful if people who have experience and wisdom about it can chime in to contribute to the discussion: https://gitlab.haskell.org/ghc/ghc/-/issues/19437. And I would like to say that I am overall very thankful for the involvement of the people who have been giving us feedback and have been reviewing the resulting MRs. Have a very nice day, Hécate --- [0]: https://github.com/ndmitchell/hlint [1]: https://gitlab.haskell.org/ghc/ghc/-/issues/18424 [2]: https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4147 -- Hécate ✨ 🐦: @TechnoEmpress IRC: Uniaika WWW: https://glitchbra.in RUN: BSD

2 2

On CI
by Moritz Angermann 24 Mar '21

24 Mar '21

Hi there! Just a quick update on our CI situation. Ben, John, Davean and I have been discussion on CI yesterday, and what we can do about it, as well as some minor notes on why we are frustrated with it. This is an open invitation to anyone who in earnest wants to work on CI. Please come forward and help! We'd be glad to have more people involved! First the good news, over the last few weeks we've seen we *can* improve CI performance quite substantially. And the goal is now to have MR go through CI within at most 3hs. There are some ideas on how to make this even faster, especially on wide (high core count) machines; however that will take a bit more time. Now to the more thorny issue: Stat failures. We do not want GHC to regress, and I believe everyone is on board with that mission. Yet we have just witnessed a train of marge trials all fail due to a -2% regression in a few tests. Thus we've been blocking getting stuff into master for at least another day. This is (in my opinion) not acceptable! We just had five days of nothing working because master was broken and subsequently all CI pipelines kept failing. We have thus effectively wasted a week. While we can mitigate the latter part by enforcing marge for all merges to master (and with faster pipeline turnaround times this might be more palatable than with 9-12h turnaround times -- when you need to get something done! ha!), but that won't help us with issues where marge can't find a set of buildable MRs, because she just keeps hitting a combination of MRs that somehow together increase or decrease metrics. We have three knobs to adjust: - Make GHC build faster / make the testsuite run faster. There is some rather interesting work going on about parallelizing (earlier) during builds. We've also seen that we've wasted enormous amounts of time during darwin builds in the kernel, because of a bug in the testdriver. - Use faster hardware. We've seen that just this can cut windows build times from 220min to 80min. - Reduce the amount of builds. We used to build two pipelines for each marge merge, and if either of both (see below) failed, marge's merge would fail as well. So not only did we build twice as much as we needed, we also increased our chances to hit bogous build failures by 2. We need to do something about this, and I'd advocate for just not making stats fail with marge. Build errors of course, but stat failures, no. And then have a separate dashboard (and Ben has some old code lying around for this, which someone would need to pick up and polish, ...), that tracks GHC's Performance for each commit to master, with easy access from the dashboard to the offending commit. We will also need to consider the implications of synthetic micro benchmarks, as opposed to say building Cabal or other packages, that reflect more real-world experience of users using GHC. I will try to provide a data driven report on GHC's CI on a bi-weekly or month (we will have to see what the costs for writing it up, and the usefulness is) going forward. And my sincere hope is that it will help us better understand our CI situation; instead of just having some vague complaints about it. Cheers, Moritz

11 23

GHC 8.10 backports?
by Moritz Angermann 24 Mar '21

24 Mar '21

Hi there! Does anyone have any backports they'd like to see for consideration for 8.10.5? Cheers, Moritz

5 8

Call for Talks: Haskell Implementors' Workshop
by Ningning Xie 22 Mar '21

22 Mar '21

Call for Talks ACM SIGPLAN Haskell Implementors' Workshop https://icfp21.sigplan.org/home/hiw-2021 Virtual, 22 Aug, 2021 Co-located with ICFP 2021 https://icfp21.sigplan.org/ Important dates --------------- Deadline: Wednesday, 30 June, 2021 (AoE) Notification: Wednesday, 14 July, 2021 Workshop: Sunday, 22 August, 2021 The 13th Haskell Implementors' Workshop is to be held alongside ICFP 2021 this year virtually. It is a forum for people involved in the design and development of Haskell implementations, tools, libraries, and supporting infrastructure, to share their work and discuss future directions and collaborations with others. Talks and/or demos are proposed by submitting an abstract, and selected by a small program committee. There will be no published proceedings. The workshop will be informal and interactive, with open spaces in the timetable and room for ad-hoc discussion, demos, and lightning short talks. Scope and target audience ------------------------- It is important to distinguish the Haskell Implementors' Workshop from the Haskell Symposium which is also co-located with ICFP 2021. The Haskell Symposium is for the publication of Haskell-related research. In contrast, the Haskell Implementors' Workshop will have no proceedings -- although we will aim to make talk videos, slides and presented data available with the consent of the speakers. The Implementors' Workshop is an ideal place to describe a Haskell extension, describe works-in-progress, demo a new Haskell-related tool, or even propose future lines of Haskell development. Members of the wider Haskell community encouraged to attend the workshop -- we need your feedback to keep the Haskell ecosystem thriving. Students working with Haskell are specially encouraged to share their work. The scope covers any of the following topics. There may be some topics that people feel we've missed, so by all means submit a proposal even if it doesn't fit exactly into one of these buckets: * Compilation techniques * Language features and extensions * Type system implementation * Concurrency and parallelism: language design and implementation * Performance, optimisation and benchmarking * Virtual machines and run-time systems * Libraries and tools for development or deployment Talks ----- We invite proposals from potential speakers for talks and demonstrations. We are aiming for 20-minute talks with 5 minutes for questions and changeovers. We want to hear from people writing compilers, tools, or libraries, people with cool ideas for directions in which we should take the platform, proposals for new features to be implemented, and half-baked crazy ideas. Please submit a talk title and abstract of no more than 300 words. Submissions should be made via HotCRP. The website is: https://icfp-hiw21.hotcrp.com/ We will also have lightning talks session. These have been very well received in recent years, and we aim to increase the time available to them. Lightning talks be ~7mins and are scheduled on the day of the workshop. Suggested topics for lightning talks are to present a single idea, a work-in-progress project, a problem to intrigue and perplex Haskell implementors, or simply to ask for feedback and collaborators. Logistics --------- Due to the on-going COVID-19 situation, ICFP (and, consequently, HIW) will be held remotely this year. However, the organizers are still working hard to provide for a great workshop experience. While we are sad that this year will lack the robust hallway track that is often the highlight of HIW, we believe that this remote workshop presents a unique opportunity to include more of the Haskell community in our discussion and explore new modes of communicating with our colleagues. We hope that you will join us in making this HIW as vibrant as any other. Program Committee ----------------- * Dominique Devriese (Vrije Universiteit Brussel) * Daan Leijen (Microsoft Research) * Andres Löh (Well-Typed LLP) * Julie Moronuki (Typeclass Consulting) * John Wiegley (DFINITY) * Ningning Xie (the University of Hong Kong) * Edward Z. Yang (Facebook AI Research) Contact ------- * Ningning Xie <xnning at hku.hk>

1 0

GitLab is down: urgent
by Simon Peyton Jones 22 Mar '21

22 Mar '21

GHC's GitLab seems to be down. Ben? (I just get 502's) Simon

10 21

Post-mortem for last-night's GitLab outage
by Ben Gamari 19 Mar '21

19 Mar '21

Hi everyone, It appears that gitlab.haskell.org's GitLab services went down around nine hours ago (around midnight EST). Surprisingly, the outage appears to be entirely unrelated to yesterday's upgrade. Rather, the problem was merely that the docker repository had grown to fill the entirety of the server's data volume. I have fixed this (and prevented future occurrences of the same issue) by moving our Docker images to a new volume. Services should be now once again fully operational. Disk usage is something that we have struggled with in the past, in part due the relatively small local disk capacity of our servers and previous unreliability of our hosting provider's iSCSI block storage infrastructure. The latter has previously prompted us to avoid using iSCSI volumes for operation-critical data, while the former has meant that we had to keep bulk data size like Docker images in careful check, lest we run out of local storage. At this point, it has been over half a year since we have experienced any trouble with iSCSI. For this reason, I have moved the Docker images back to iSCSI. This should eliminate this failure mode in the future. Meanwhile the GitLab database remains on local storage, also minimizing the potential for downtime due to future iSCSI failures. Cheers, - Ben

1 0

GitLab upgrade soon
by Ben Gamari 19 Mar '21

19 Mar '21

Hi all, I will be performing a GitLab upgrade starting in approximately one hour. While I generally try to do this with more notice and out of working hours, in this case there is an exploitable GitLab vulnerability that deserves swift action. Thank you for you patience. Cheers, - Ben

2 2

Pull request to editline
by Oleg Grenrus 18 Mar '21

18 Mar '21

Hi Judah, I'm sending your an email, in case you haven't noticed GitHub notifications. I have a PR https://github.com/judah/haskeline/pull/153 (now opened for two months). It's blocking work on Data.List specialization. Also Ben have pinged you on https://github.com/judah/haskeline/issues/154 to make releases for 9.0.1 and 9.0.2. It would be very great if my patch could be merged and released, that should make editline ready for 9.2 and maybe even 9.4 (I think we are too late with refactor of Data.List for 9.2). - Oleg

2 2

Is referring to GHC-proposals in GHC user manual bad practice or not?
by Oleg Grenrus 17 Mar '21

17 Mar '21

I have a following question: My lexer rules related proposal was recently accepted. The biggest part of getting it in is writing documentation for it. While looking at Divergence from Haskell 98 and Haskell 2010 section of the user manual, in particular Lexical syntax, it already has See "GHC Proposal #229 for the precise rules.". Can I just the same? (I think there was an implicit acceptance of that practice in e.g. https://gitlab.haskell.org/ghc/ghc/-/merge_requests/1664#note_238759) However, I think that referring to proposals text for "essential" bits of information is a bad practice. Because GHC proposals are sometimes amended, one have to look into GitHub history to find out what were there for a particular time point of a GHC release. Very laborous. --- Currently there is 23 references to about a dozen of proposals. An example are passages like In 9.0, the behavior of this extension changed, and now we require that a negative literal must not be preceded by a closing token (see `GHC Proposal #229 <https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0229-w…>`__ for the definition of a closing token). or a future release will be turned off by default and then possibly removed. The reasons for this and the deprecation schedule are described in `GHC proposal #30 <https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0030-r…>`__. And there are better examples, which are references for more information, not essential one, like See the proposal `DuplicateRecordFields without ambiguous field access <https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0366-n…>`_ and the documentation on :extension:`DuplicateRecordFields` for further details. (I'd put the internal user manual link first), or But these automatic eta-expansions may silently change the semantics of the user's program, and deep skolemisation was removed from the language by `GHC Proposal #287 <https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0287-s…>`__. This proposal has many more examples. --- So to boil down my question, can I write Lexical syntax of identifiers and decimal numbers differs slightly from the Haskell report. See GHC Proposal #403 for the precise rules and differences. - Oleg

5 7

Generalising KnowNat/Char/Symbol?
by Sylvain Henry 16 Mar '21

16 Mar '21

Hi, I would like to have a KnownWord constraint to implement a type-safe efficient sum type. For now [1] I have: data V (vs :: [Type]) = Variant !Word Any where Word is a tag used as an index in the vs list and Any a value (unsafeCoerced to the appropriate type). Instead I would like to have something like: data V (vs :: [Type]) = Variant (forall w. KnownWord w => Proxy w -> Index w vs) Currently if I use KnownNat (instead of the proposed KnownWord), the code isn't very good because Natural equality is implemented using `naturalEq` which isn't inlined and we end up with sequences of comparisons instead of single case-expressions with unboxed literal alternatives. I could probably implement KnownWord and the required stuff (axioms and whatnot), but then someone will want KnownInt and so on. So would it instead make sense to generalise the different "Known*" we currently have with: class KnownValue t (v :: t) where valueSing :: SValue t v newtype SValue t (v :: t) = SValue t litVal :: KnownValue t v => proxy v -> t type KnownNat = KnownValue Natural type KnownChar = KnownValue Char type KnownSymbol = KnownValue String type KnownWord = KnownValue Word Thoughts? Sylvain [1] https://hackage.haskell.org/package/haskus-utils-variant-3.1/docs/Haskus-Ut…

3 2