April 2021 - ghc-devs

The future of Phabricator
by Ben Gamari 18 Sep '21

18 Sep '21

TL;DR. For several reasons I think we should consider alternatives to Phabricator. My view is that GitLab seems like the best option. Hello everyone, Over the past year I have been growing increasingly weary of our continued dependence on Phabricator. Without a doubt, its code review interface is the best I have used. However, for a myriad of reasons I am recently questioning whether it is still the best tool for GHC's needs. # The problem There are a number of reasons why I am currently uncertain about Phabricator. For one, at this point we have no options for support in the event that something goes wrong as the company responsible for Phabricator, Phacility, has closed their support channels to non-paying customers. Furthermore, in the past year or two Phacility has been placing their development resources in the parts their customers pay them for, which appear to be much different that the parts that we actively use. For this reason, some parts that we rely on seem oddly half-finished. This concern was recently underscored by some rather unfortunate misfeatures in Harbormaster which resulted in broken CI after the Hadrian merge and now apparent bugs which have made it difficult to migrate to the CircleCI integration we previously prepared. Perhaps most importantly, in our recent development priorities survey our use of Phabricator was the most common complaint by a fair margin, both in case of respondents who have contributed patches and those who have not. On the whole, past contributors and potential future contributors alike have strongly indicated that they want a more Git-like experience. Of course, this wasn't terribly surprising; this is just the most recent case where contributors have made this preference known. Frankly, in a sense it is hard to blame them. The fact that users need to install a PHP tool, Arcanist, to contribute anything but documentation patches has always seemed like unnecessary friction to me and I would be quite happy to be rid of it. Indeed we have had a quite healthy number of GitHub documentation patches since we started accepting them. This makes me thing that there may indeed be potential contributoes that we are leaving on the table. # What to do With Rackspace support ending at the end of year, now may be a good time to consider whether we really want to continue on this road. Phabricator is great at code review but I am less and less certain that it is worth the maintenance uncertainty and potential lost contributors that it costs. Moreover, good alternatives seem closer at-hand than they were when we deployed Phabricator. ## Move to GitHub When people complain about our infrastructure, they often use GitHub as the example of what they would like to see. However, realistically I have a hard time seeing GitHub as a viable option. Its feature set is simply insufficient enough to handle the needs of a larger project like GHC without significant external tooling (as seen in the case of Rust-lang). The concrete reasons have been well-documented in previous discussions but, to summarize, * its review functionality is extremely painful to use with larger patches * rebased patches lead to extreme confusion and lost review comments * it lacks support for pre-receive hooks, which serve as a last line of defense to prevent unintended submodule changes * its inability to deal with external tickets is problematic * there is essentially no possibility that we could eventually migrate GHC's tickets to GitHub's ticket tracker without considerable data loss (much less manage the ticket load that would result), meaning that we would forever be stuck with maintaining Trac. * on a personal note, its search functionality has often left me empty-handed On the whole, these issues seem pretty hard to surmount. ## Move to GitLab In using GitLab for another project over the past months I have been positively surprised by its quality. It handles rebased merge requests far better than GitHub, has functional search, and quite a usable review interface. Furthermore, upstream has been extremely responsive to suggestions for improvement [1]. Even out-of-the-box it seems to be flexible enough to accommodate our needs, supporting integration with external issue trackers, having reasonable release management features, and support for code owners to automate review triaging (replacing much of the functionality of Phabricator's Herald). Finally, other FOSS projects' [3] recent migrations from Phabrictor to GitLab have shown that GitLab-the-company is quite willing to offer help when needed. I took some time this weekend to try setting up a quick GHC instance [2] to play around with. Even after just a few hours of playing around I think the result is surprisingly usable. Out of curiosity I also played around with importing some tickets from Trac (building on Matt Pickering's Trac-to-Maniphest migration tool). With relatively little effort I was even able to get nearly all of our tickets (as of 9 months ago) imported while preserving ticket numbers (although there are naturally a few wrinkles that would need to be worked out). Naturally, I think we should treat the question of ticket tracker migration as an orthogonal one to code review, but it is good to know that this is possible. ## Continue with Phabricator Continuing with Phabricator is of course an option. Its review functionality is great and it has served us reasonably well. However, compared to GitLab and even GitHub of today its features seem less distinguished than they once did. Moreover, the prospect of having to maintain a largely stagnant product with no support strikes me as a slightly dangerous game to play. Working around the issues we have recently encountered has already cost a non-negligible amount of time. # The bottom line If it wasn't clear already, I think that we should strongly consider a move to GitLab. At this point it seems clear that it isn't going to vanish, has a steady pace of development, is featureful, and available. However, these are just my thoughts. What do you think? Cheers, - Ben [1] 11.4 will ship with a file tree view in the code review interface, which I reported (https://gitlab.com/gitlab-org/gitlab-ce/issues/46474) as being is one of the Phabricator features I missed the most during review [2] https://gitlab.home.smart-cactus.org/ghc/ghc/issues/14641 [3] The GNOME and freedesktop.org projects have recently migrated, the former from a hodge-podge of self-hosted services and the latter from Phabricator

17 40

New implementation for `ImpredicativeTypes`
by Alejandro Serrano Mena 08 Sep '21

08 Sep '21

Hi all, As I mentioned some time ago, we have been busy working on a new implementation of `ImpredicativeTypes` for GHC. I am very thankful to everybody who back then sent us examples of impredicativity which would be nice to support, as far as we know this branch supports all of them! :) If you want to try it, at https://gitlab.haskell.org/trupill/ghc/commit/a3f95a0fe0f647702fd7225fa719a… you can find the result of the pipeline, which includes builds for several platforms (click on the "Artifacts" button, the one which looks like a cloud, to get them). The code is being developed at https://gitlab.haskell.org/trupill/ghc. Any code should run *unchanged* except for some eta-expansion required for some specific usage patterns of higher-rank types. Please don't hesitate to ask any questions or clarifications about it. A merge request for tracking this can be found at https://gitlab.haskell.org/ghc/ghc/merge_requests/1659 Kind regards, Alejandro

8 23

Potential improvement in compacting GC
by Ömer Sinan Ağacan 02 Sep '21

02 Sep '21

Hello, I recently implemented the algorithm used by GHC's compacting GC in another project. The algorithm (after marking) makes two passes over the heap /generation. In GHC, these passes are implemented in [1] and in the next function. In my implementation I tried 3 ways of implementing these passes, one of which is the same as GHC's code, and benchmarked each version. I found that the fastest implementation is not what's used in GHC, but it could easily be used. I should mention that my code targets Wasm, and I benchmarked Wasm instructions executed. Not CPU cycles, CPU instructions, or anything like that. It's very likely that the results will be different when benchmarking code running on actual hardware. Secondly, in my case the heap is mostly dead (residency is low). In GHC, compaction for the oldest generation is enabled when residency passes a threshold, so the assumption is the heap is mostly live. I'm guessing this should also make some difference. Anyway, the first implementation I tried was similar to GHC's scan, but I did scan += object_size(scan); instead of bumping scan by one, as GHC does in [2]. This was the slowest version. Second implementation did the same as GHC (bumped scan by one). This was faster, but still slower than the next version. What I found to be the best is scanning the bitmap, not the heap. The bitmap iterator reads one word at a time. In each iteration it checks if the bitmap word is 0. In GHC, in the best case this can skip 16 words on heap on 32-bit systems, and 32 words on 64-bit. Reminder: we need two bits per object in the bitmap, see [3]. (this is not the case in my implementation so the payoff is better) When the bitmap word is not 0 I use "count trailing zeros" (or "count leading zeros" depending on the bitmap implementation) to get the number of words to skip. This is a single instruction on Wasm and x86 (TZCNT or LZCNT, available via __builtin_ctzl and __builtin_clzl in gcc). So instead of skipping one word at a time, this can potentially skip 16 words (or 32 on 64-bit architectures). When that's not possible, it can still skip multiple words by using ctz/clz. Ömer [1]: https://github.com/ghc/ghc/blob/bd877edd9499a351db947cd51ed583872b2facdf/rt… [2]: https://github.com/ghc/ghc/blob/bd877edd9499a351db947cd51ed583872b2facdf/rt… [3]: https://github.com/ghc/ghc/blob/bd877edd9499a351db947cd51ed583872b2facdf/rt…

5 6

linting GHC
by Richard Eisenberg 01 May '21

01 May '21

Hi devs, I see in CI that we're now linting the GHC source code. In order to avoid CI failure, I would like to lint locally, and I somehow know (though I forget where I learned it) that hadrian/build lint:compiler is the way to do this. After building hlint, I am able to do this. But the output is surprising; here is the tail: # hlint (for lint:compiler) compiler/GHC/Tc/Utils/Unify.hs:2:1-36: Warning: Unused LANGUAGE pragma Found: {-# LANGUAGE MultiWayIf #-} Perhaps you should remove it. compiler/stage1/build/GHC/Parser.hs:6:1-29: Warning: Unused LANGUAGE pragma Found: {-# LANGUAGE ViewPatterns #-} Perhaps you should remove it. compiler/stage1/build/GHC/Parser.hs:8:1-27: Warning: Unused LANGUAGE pragma Found: {-# LANGUAGE LambdaCase #-} Perhaps you should remove it. compiler/stage1/build/GHC/Parser.hs:38:1-54: Warning: Use fewer imports Found: import Control.Monad ( unless, liftM, when, (<=<) ) import Control.Monad ( ap ) Perhaps: import Control.Monad ( unless, liftM, when, (<=<), ap ) compiler/stage1/build/GHC/Parser.hs:43:1-18: Warning: Use fewer imports Found: import GHC.Prelude import GHC.Prelude Perhaps: import GHC.Prelude compiler/stage1/build/GHC/Cmm/Parser.hs:17:1-24: Warning: Use fewer imports Found: import GHC.StgToCmm.Prof import GHC.StgToCmm.Prof Perhaps: import GHC.StgToCmm.Prof compiler/stage1/build/GHC/Cmm/Parser.hs:71:1-20: Warning: Use fewer imports Found: import Control.Monad import Control.Monad ( ap ) Perhaps: import Control.Monad compiler/stage2/build/GHC/Parser.hs:6:1-29: Warning: Unused LANGUAGE pragma Found: {-# LANGUAGE ViewPatterns #-} Perhaps you should remove it. compiler/stage2/build/GHC/Parser.hs:8:1-27: Warning: Unused LANGUAGE pragma Found: {-# LANGUAGE LambdaCase #-} Perhaps you should remove it. compiler/stage2/build/GHC/Parser.hs:38:1-54: Warning: Use fewer imports Found: import Control.Monad ( unless, liftM, when, (<=<) ) import Control.Monad ( ap ) Perhaps: import Control.Monad ( unless, liftM, when, (<=<), ap ) compiler/stage2/build/GHC/Parser.hs:43:1-18: Warning: Use fewer imports Found: import GHC.Prelude import GHC.Prelude Perhaps: import GHC.Prelude compiler/stage2/build/GHC/Cmm/Parser.hs:17:1-24: Warning: Use fewer imports Found: import GHC.StgToCmm.Prof import GHC.StgToCmm.Prof Perhaps: import GHC.StgToCmm.Prof compiler/stage2/build/GHC/Cmm/Parser.hs:71:1-20: Warning: Use fewer imports Found: import Control.Monad import Control.Monad ( ap ) Perhaps: import Control.Monad 13 hints Error when running Shake build system: at want, called at src/Main.hs:104:30 in main:Main * Depends on: lint:compiler at cmd_, called at src/Rules/Lint.hs:70:3 in main:Rules.Lint * Raised the exception: Development.Shake.cmd, system command failed Command line: hlint -j --cpp-define x86_64_HOST_ARCH --cpp-include=includes --cpp-include=_build/stage1/lib --cpp-include=compiler --cpp-include=_build/stage1/lib/ghcplatform.h --cpp-include=_build/stage1/compiler/build -h compiler/.hlint.yaml compiler Exit code: 1 Stderr: There are several curiosities here: * I introduced one of these errors, in GHC.Tc.Utils.Unify; the other errors are in files I have not modified. Will these affect CI? * Hadrian crashes at the end. Is that expected? It says "system command failed" with a bunch of gobbledegook. Is this an error within Hadrian? Or is this just Hadrian's way of saying that it encountered an error when linting? If it's the latter, the current message is hard to read. * As you may be able to see, I use a dark-background terminal window. The colors output by the hlint process are nigh impossible to read when describing redundant LANGUAGE pragmas. Are these colors configurable? Thanks! Richard

2 1

Compiling libraries
by Simon Peyton Jones 29 Apr '21

29 Apr '21

Ben and other friends What is the approved way to build libraries with (and for) a compiler that I've just built? I tried this: bash$ cabal v2-install --with-ghc=/home/simonpj/code/ghc-9.2-branch/inplace/bin/ghc-stage2 streamly --allow-newer but it failed with Data/Primitive/Types.hs:273:844: error: * Couldn't match expected type 'Word#' with actual type 'Word8#' * In the fourth argument of 'shimmedSetWord8Array#', namely 'x#' (This was for a compiler built from the ghc-9.2 branch.) I'd like to be able to do the same for HEAD. I need to do this to help with the numerous tickets asking about library regressions. For example, I want to be able repoduce @harendra's example<https://gitlab.haskell.org/ghc/ghc/-/issues/19557#note_348969>, but I can't because I can't build streamly. Thanks! Simon

4 4

[ANNOUNCE] Experimental Windows GHC 8.10.5 binary release. Supports Haskell DLLs.
by kyra 29 Apr '21

29 Apr '21

An experimental Windows GHC 8.10.5 64-bit Visual C/Native Windows SDK binary compatible distribution is released. Features: Supports Haskell DLLs. Both static and dynamic compilers are shipped. Uses LLVM tools exclusively, hence linking is blazing fast compared to the stock distro. Is position-independent (can be loaded to any address, not only the lower 2GBs) and generates position-independent code. Based on and targets Microsoft Visual C runtime and native Windows SDK. The "full" variant of distro contains 438 extra prebuilt packages and a set utilities, including not only the mandatory cabal, stack, Happy and Alex, but also linters, formatters, preprocessors, and even beasts like pandoc, haskell-language-server and even Agda compiler. Distribution site: https://awson.github.io/ghc-nw/. Cheers, K. Briantsev (aka awson)

1 0

simple Haskell help needed on #19746
by Richard Eisenberg 28 Apr '21

28 Apr '21

Hi devs, tl;dr: Is there any (efficient) way to get the String consumed by a `reads`? I'm stuck in thinking about a fix for #19746. Happily, the problem is simple enough that I could assign it in the first few weeks of a Haskell course... and yet I can't find a good solution! So I pose it here for inspiration. The high-level problem: Assign correct source spans to options within a OPTIONS_GHC pragma. Current approach: The payload of an OPTIONS_GHC pragma gets turned into a String and then processed by GHC.Utils.Misc.toArgs :: String -> Either String [String]. The result of toArgs is either an error string (the Left result) or a list of lexed options (the Right result). A little-known fact is that Haskell strings can be put in a OPTIONS_GHC pragma. So I can write both {-# OPTIONS_GHC -funbox-strict-fields #-} and {-# OPTIONS_GHC "-funbox-strict-fieds" #-}. Even stranger, I can write {-# OPTIONS_GHC ["-funbox-strict-fields"] #-}, where GHC will understand a list of strings. While I don't really understand the motivation for this last feature (I posted #19750 about this), the middle option, with the quotes, seems like it might be useful. Desired approach: change toArgs to have this type: RealSrcLoc -> String -> Either String [Located String], where the input RealSrcLoc is the location of the first character of the input String. Then, as toArgs processes the input, it advances the RealSrcLoc (with advanceSrcLoc), allowing us to create correct SrcSpans for each String. Annoying fact: Not all characters advance the source location by one character. Tabs and newlines don't. Perhaps some other characters don't, too. Central stumbling block: toArgs uses `reads` to parse strings. This makes great sense, because `reads` already knows how to convert Haskell String syntax into a proper String. The problem is that we have no idea what characters were consumed by `reads`. And, short of looking at the length of the remainder string in `reads` and comparing it to the length of the input string, there seems to be no way to recreate this lost information. Note that comparing lengths is slow, because we're dealing with Strings here. Once we know what was consumed by `reads`, then we can just repeatedly call advancedSrcLoc, and away we go. Ideas to get unblocked: 1. Just do the slow (quadratic in the number of options) thing, looking at the lengths of strings often. 2. Reimplement reading of strings to return both the result and the characters consumed 3. Incorporate the parsing of OPTIONS_GHC right into the lexer It boggles me that there isn't a better solution here. Do you see one? Thanks, Richard

3 4

External STG Interpreter blog post
by Csaba Hruska 27 Apr '21

27 Apr '21

Hello, I've written a blog post about GHC-WPC's external stg interpreter. https://www.patreon.com/posts/external-stg-49857800 Feedback is welcome. Cheers, Csaba

3 4

magicDict
by Simon Peyton Jones 26 Apr '21

26 Apr '21

Ed, and other ghc-devs We are busy tidying up magicDict, and making it much more type-safe: see * https://gitlab.haskell.org/ghc/ghc/-/issues/16646 * https://gitlab.haskell.org/ghc/ghc/-/merge_requests/5573 As part of that change we're think of changing its currently-rather-obscure name. I rather favour "reflectDict". Any other views? Simon

5 14

[ANNOUNCE] GHC 9.2.1-alpha2 released
by Ben Gamari 22 Apr '21

22 Apr '21

Hi all, The GHC developers are very happy to announce the availability of the second alpha release in the 9.2.1 series. Binary distributions, source distributions, and documentation are available at https://downloads.haskell.org/ghc/9.2.1-alpha2 GHC 9.2 will bring a number of exciting features including: * Many changes in the area of records, including the new `RecordDotSyntax` and `NoFieldSelectors` language extensions, as well as Support for `DuplicateRecordFields` with `PatternSynonyms`. * Introduction of the new `GHC2021` language extension set, giving users convenient access to a larger set of language extensions which have been long considered stable. * Merge of `ghc-exactprint` into the GHC tree, providing infrastructure for source-to-source program rewriting out-of-the-box. * Introduction of a `BoxedRep` `RuntimeRep`, allowing for polymorphism over levity of boxed objects (#17526) * Implementation of the `UnliftedDataTypes` extension, allowing users to define types which do not admit lazy evaluation ([proposal]) * The new [-hi profiling] mechanism which provides significantly improved insight into thunk leaks. * Support for the `ghc-debug` out-of-process heap inspection library [ghc-debug] * Support for profiling of pinned objects with the cost-centre profiler (#7275) * Introduction of Haddock documentation support in TemplateHaskell (#5467) * Proper support for impredicative types in the form of Quick-Look impredicativity. * A native code generator backend for AArch64. This pre-release brings nearly 50 fixes relative to the first alpha, although the long-awaited ARM NCG backend hasn't quite landed yet. As always, do give this a try and open a [ticket] if you see anything amiss. Happy testing, - Ben [proposal]: https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0265-u… [-hi profiling]: https://well-typed.com/blog/2021/01/first-look-at-hi-profiling-mode/ [ghc-debug]: http://ghc.gitlab.haskell.org/ghc-debug/ [ticket]: https://gitlab.haskell.org/ghc/ghc/-/issues/new

1 0