August 2019 - ghc-devs

The future of Phabricator
by Ben Gamari 18 Sep '21

18 Sep '21

TL;DR. For several reasons I think we should consider alternatives to Phabricator. My view is that GitLab seems like the best option. Hello everyone, Over the past year I have been growing increasingly weary of our continued dependence on Phabricator. Without a doubt, its code review interface is the best I have used. However, for a myriad of reasons I am recently questioning whether it is still the best tool for GHC's needs. # The problem There are a number of reasons why I am currently uncertain about Phabricator. For one, at this point we have no options for support in the event that something goes wrong as the company responsible for Phabricator, Phacility, has closed their support channels to non-paying customers. Furthermore, in the past year or two Phacility has been placing their development resources in the parts their customers pay them for, which appear to be much different that the parts that we actively use. For this reason, some parts that we rely on seem oddly half-finished. This concern was recently underscored by some rather unfortunate misfeatures in Harbormaster which resulted in broken CI after the Hadrian merge and now apparent bugs which have made it difficult to migrate to the CircleCI integration we previously prepared. Perhaps most importantly, in our recent development priorities survey our use of Phabricator was the most common complaint by a fair margin, both in case of respondents who have contributed patches and those who have not. On the whole, past contributors and potential future contributors alike have strongly indicated that they want a more Git-like experience. Of course, this wasn't terribly surprising; this is just the most recent case where contributors have made this preference known. Frankly, in a sense it is hard to blame them. The fact that users need to install a PHP tool, Arcanist, to contribute anything but documentation patches has always seemed like unnecessary friction to me and I would be quite happy to be rid of it. Indeed we have had a quite healthy number of GitHub documentation patches since we started accepting them. This makes me thing that there may indeed be potential contributoes that we are leaving on the table. # What to do With Rackspace support ending at the end of year, now may be a good time to consider whether we really want to continue on this road. Phabricator is great at code review but I am less and less certain that it is worth the maintenance uncertainty and potential lost contributors that it costs. Moreover, good alternatives seem closer at-hand than they were when we deployed Phabricator. ## Move to GitHub When people complain about our infrastructure, they often use GitHub as the example of what they would like to see. However, realistically I have a hard time seeing GitHub as a viable option. Its feature set is simply insufficient enough to handle the needs of a larger project like GHC without significant external tooling (as seen in the case of Rust-lang). The concrete reasons have been well-documented in previous discussions but, to summarize, * its review functionality is extremely painful to use with larger patches * rebased patches lead to extreme confusion and lost review comments * it lacks support for pre-receive hooks, which serve as a last line of defense to prevent unintended submodule changes * its inability to deal with external tickets is problematic * there is essentially no possibility that we could eventually migrate GHC's tickets to GitHub's ticket tracker without considerable data loss (much less manage the ticket load that would result), meaning that we would forever be stuck with maintaining Trac. * on a personal note, its search functionality has often left me empty-handed On the whole, these issues seem pretty hard to surmount. ## Move to GitLab In using GitLab for another project over the past months I have been positively surprised by its quality. It handles rebased merge requests far better than GitHub, has functional search, and quite a usable review interface. Furthermore, upstream has been extremely responsive to suggestions for improvement [1]. Even out-of-the-box it seems to be flexible enough to accommodate our needs, supporting integration with external issue trackers, having reasonable release management features, and support for code owners to automate review triaging (replacing much of the functionality of Phabricator's Herald). Finally, other FOSS projects' [3] recent migrations from Phabrictor to GitLab have shown that GitLab-the-company is quite willing to offer help when needed. I took some time this weekend to try setting up a quick GHC instance [2] to play around with. Even after just a few hours of playing around I think the result is surprisingly usable. Out of curiosity I also played around with importing some tickets from Trac (building on Matt Pickering's Trac-to-Maniphest migration tool). With relatively little effort I was even able to get nearly all of our tickets (as of 9 months ago) imported while preserving ticket numbers (although there are naturally a few wrinkles that would need to be worked out). Naturally, I think we should treat the question of ticket tracker migration as an orthogonal one to code review, but it is good to know that this is possible. ## Continue with Phabricator Continuing with Phabricator is of course an option. Its review functionality is great and it has served us reasonably well. However, compared to GitLab and even GitHub of today its features seem less distinguished than they once did. Moreover, the prospect of having to maintain a largely stagnant product with no support strikes me as a slightly dangerous game to play. Working around the issues we have recently encountered has already cost a non-negligible amount of time. # The bottom line If it wasn't clear already, I think that we should strongly consider a move to GitLab. At this point it seems clear that it isn't going to vanish, has a steady pace of development, is featureful, and available. However, these are just my thoughts. What do you think? Cheers, - Ben [1] 11.4 will ship with a file tree view in the code review interface, which I reported (https://gitlab.com/gitlab-org/gitlab-ce/issues/46474) as being is one of the Phabricator features I missed the most during review [2] https://gitlab.home.smart-cactus.org/ghc/ghc/issues/14641 [3] The GNOME and freedesktop.org projects have recently migrated, the former from a hodge-podge of self-hosted services and the latter from Phabricator

17 40

Residency profiles
by Simon Peyton Jones 23 Jan '20

23 Jan '20

Simon, Ben, Omer As you'll see in comments 55-72 of https://ghc.haskell.org/trac/ghc/ticket/9476, Sebastian has been a bit flummoxed by the task of measure residency profiles; that is, how much data is truly live during execution. A major GC measures that, but we are vulnerable to exactly when it happens (even with -G1) and that can lead to irreproducible results. I think what we want is a way to trigger GC at very regular intervals, after (say) each 10kbytes or 100kbytes or 1Mbyte of allocation. That might be expensive, but we'd get reproducible results. I don't think that is possible right now - see the ticket - but it would be easy enough to do wouldn't it? Just give only 10k or 100k or 1M to the allocator when setting it running again. Would you consider this? Or are we just missing something obvious? Needless to say, we want to do all this with full optimisation on, no cost-centre profiling. Thanks Simon

4 8

[ANNOUNCE] GHC 8.8.1-alpha2 is now available
by Ben Gamari 12 Dec '19

12 Dec '19

Hello everyone, The GHC team is pleased to announce the second and likely last alpha release of GHC 8.8.1. The source distribution, binary distributions, and documentation are available at https://downloads.haskell.org/~ghc/8.8.1-alpha2 A draft of the release notes is also available [1]. This release is the culmination of over 3000 commits by over one hundred contributors and has several new features and numerous bug fixes relative to GHC 8.6: * Profiling now works correctly on 64-bit Windows (although still may be problematic on 32-bit Windows due to platform limitations; see #15934) * A new code layout algorithm for amd64's native code generator * The introduction of a late lambda-lifting pass which may reduce allocations significantly for some programs. * Further work on Trees That Grow, enabling improved code re-use of the Haskell AST in tooling * More locations where users can write `forall` (GHC Proposal #0007) * Further work on the Hadrian build system This release brings a number of fixes since alpha 1: * A number of linker fixes (#16779, #16784) * The process, binary, Cabal, time, terminfo libraries have all been bumped to their final release versions * A regression rendering TemplateHaskell unusable in cross-compiled configurations has been fixed (#16331) * A regression causing compiler panics on compilation of some programs has been fixed (#16449) * -Wmissing-home-modules now handles hs-boot files correctly (#16551) * A regression causing some programs to fail at runtime has been fixed (#16066) Due to on-going work on our release and testing infrastructure this cycle is proceeding at a pace significantly slower than expected. However, we anticipate that this investment will allow us to release a more reliable, easier-to-install compiler on the planned six-month release cadence in the future. As always, if anything looks amiss do let us know. Happy compiling! Cheers, - Ben [1] https://downloads.haskell.org/ghc/8.8.1-alpha2/docs/html/users_guide/8.8.1-…

8 18

[ANNOUNCE] GHC 8.8.1 is now available
by Ben Gamari 07 Oct '19

07 Oct '19

Hello everyone, The GHC team is pleased to announce the release candidate for GHC 8.8.1. The source distribution, binary distributions, and documentation are available at https://downloads.haskell.org/ghc/8.8.1 This release is the culmination of over 3000 commits by over one hundred contributors and has several new features and numerous bug fixes relative to GHC 8.6: * Visible kind applications are now supported (GHC Proposal #15) * Profiling now works correctly on 64-bit Windows (although still may be problematic on 32-bit Windows due to platform limitations; see #15934) * A new code layout algorithm for amd64's native code generator significantly improving the runtime performance of some kernels * The introduction of a late lambda-lifting pass which may reduce allocations significantly for some programs. * Further work on Trees That Grow, enabling improved code re-use of the Haskell AST in tooling * Users can write `forall` in more contexts (GHC Proposal #7) * The pattern-match checker is now more precise in the presence of strict fields with uninhabited types. * A comprehensive audit of GHC's memory ordering barriers has been performed, resulting in a number of fixes that should significantly improve the reliability of programs on architectures with weakly-ordered memory models (e.g. PowerPC, many ARM and AArch64 implementations). * A long-standing linker limitation rendering GHCi unusable with projects with cyclic symbol dependencies has been fixed (#13786) * Further work on the Hadrian build system * Countless miscellaneous bug-fixes Unfortunately, due to a build issue (#17108) found late in the release process i386 Windows builds are currently unavailable. These will be provided in the coming weeks. As always, if anything looks amiss do let us know. Happy compiling! Cheers, - Ben [1] https://downloads.haskell.org/ghc/8.8.1/docs/html/users_guide/8.8.1-notes.h…

4 5

How to work out why a data constructor is allocated using gdb?
by Matthew Pickering 04 Oct '19

04 Oct '19

Hi, I am trying to work out questions such as * Why are there thousands of Module data constructors allocated when building something with GHC * What is allocating all a lot of strings when building GHC? In order to do this I can use gdb in order to find some of the Module/String closures but then I'm a bit stuck about what to do. gdb displays a list of all the Module closures for example and then you can usually try to find the retainer for Module by using findPtr. If the retainer is a THUNK closure, it would be easy, as THUNK closures have DWARF information which maps straight to a particular line. However if the retainer is just some other data constructor, for example, the Module is stored in a Map, it's data constructors all the way up and none of them have DWARF info. I need to fall back to domain specific knowledge to work out where such a sequence of constructors might appear in my program. * Is there anything better I can do to map a constructor allocation to a more precise source location? The string closures were causing me some particular issues as `findPtr` was not showing any retainers so it's hard to work out why they are not GCd. * What situations can an object be retained but show no retainer when using findPtr? Cheers, Matt

2 3

GHC's module hierarchy
by Simon Peyton Jones 02 Oct '19

02 Oct '19

Dear ghc-devs hsyl20 proposes a radical overhaul of the module structure of GHC itself. He or she suggested it six months ago in https://ghc.haskell.org/trac/ghc/ticket/13009 and has now offered a monster patch https://phabricator.haskell.org/D3647 It's clearly the result of a lot of work, but I was the only one who responded on the original ticket, and it'll affect all of your lives in a very immediate way. So, would you like to * consider the idea * look at the actual re-mapping of modules hsyl20 proposals * express an opinion about whether to go ahead Probably the ticket, rather than Phab, is the best place to comment on the general idea. I'd like to thank hsyl20. GHC's rather flat module structure has grown incrementally over years. But still, there are pros and cons. Simon

2 1

Min closure payload size?
by Ömer Sinan Ağacan 11 Sep '19

11 Sep '19

Hi, I was trying to understand why some info tables that have no ptrs and nptrs like GCD_CAF end up with 1 nptrs in the generated info table and found this code in Constants.h: /* ----------------------------------------------------------------------------- Minimum closure sizes This is the minimum number of words in the payload of a heap-allocated closure, so that the closure has enough room to be overwritten with a forwarding pointer during garbage collection. -------------------------------------------------------------------------- */ #define MIN_PAYLOAD_SIZE 1 We use this in a few places in the compiler and add at least one word space in the payload. However the comment is actually wrong, forwarding pointers are made by tagging the info ptr field so we don't need a word in the payload for forwarding pointers. I tried updating this as 0 but that caused a lot of test failures (mostly in GHCi). I'm wondering if I'm missing anything or is it just some code assuming min payload size 1 without using this macro. Any ideas? Ömer

3 6

How do I find out which info table a continuation belongs to?
by Ömer Sinan Ağacan 02 Sep '19

02 Sep '19

I'm currently working on a bug and one of the things I often want to know is what's on the stack. The problem is I can't see labels of continuations so the information is really useless. Example: >>> call printStack(((StgTSO*)0x42000e0198)->stackobj) 0x42000c8788: RET_SMALL (0x512d70) 0x42000c8790: RET_SMALL (0x40edf0) stk[5] (0x42000c8798) = 0x7b3938 0x42000c87a0: CATCH_FRAME(0x735a98,0x7d3ff2) 0x42000c87b8: STOP_FRAME(0x7311b8) (I modified the printer to print stack locations when printing stacks) Here I need to know which info table the RET_SMALLs return to. Normally I do this for other kinds of closures: >>> print ((StgClosure*)...)->header.info $15 = (const StgInfoTable *) 0x404dc0 <Main_nats_info> But for continuations that doesn't work: >>> print ((StgClosure*)0x42000c8788)->header.info $11 = (const StgInfoTable *) 0x512d80 >>> info symbol 0x512d80 No symbol matches 0x512d80. Anyone know how to make this work? Can I maybe mark the continuations label in the generated assembly somehow to make those labels available in gdb? Thanks Ömer

2 3

Getting a hole's relevant local binds?
by Sandy Maguire 29 Aug '19

29 Aug '19

Hi all, I'm trying to get my hands on the relevant local binds (as reported by ghc in the presence of a type hole) for editor tooling support. Tracing the code suggests that these things come from the `TcLclEnv`, but afaict, all remnants of `TcLclEnv` are thrown away by the time we get a `TypecheckedModule`. Am I mistaken in this? If not, how receptive would y'all be to a patch that puts the `TcLclEnv`, or something similar inside `XUnboundVar GhcTc`. This way editors would have an easy means of getting their hand on whatever is in scope at the site of a hole, without resorting to parsing error messages. Cheers, Sandy -- I'm currently traveling the world, sleeping on people's couches and doing full-time collaboration on Haskell projects. If this seems interesting to you, please consider signing up as a host! https://isovector.github.io/erdos/

3 7

A few questions about BuiltInSynFamily/CoAxiomRules
by Jan van Brügge 29 Aug '19

29 Aug '19

Hi lovely people, sorry if my recent emails are getting annoying. In the last few days I refactored my code to use `BuiltInSynFamily` and `CoAxiomRule` to replace what was very ad-hoc code. So far so easy. But I have a few questions due to sparse documentation. First, about `BuiltInSynFamily`. It is a record of three functions. From what I can tell by looking at `TcTypeNats`, the two `interact` functions are used to solve the argument parts of builtin families based on known results. `interactTop` seems to simply constraints on their own, `interactInert` seems to simplify based on being given two similar contraints. By big questions is what exactly `matchFam` does. The argument seems to be the arguments to the type family, but the tuple in the result is less clear. The axiom rule is the proof witness, the second argument I guess is the type arguments you actually care about? What is this used for? The last one should be the result type. Attached to that, what are the garantuees about the list of types that you get? I assumed at first they were all flattened, but my other type family is not. I am talking about this piece of code here: ``` matchFamRnil :: [Type] -> Maybe (CoAxiomRule, [Type], Type) matchFamRnil [] = Just (axRnilDef, [], mkRowTy k v []) where binders = mkTemplateKindTyConBinders [liftedTypeKind, liftedTypeKind] [k, v] = map (mkTyVarTy . binderVar) binders matchFamRnil _ = Nothing matchFamRowExt :: [Type] -> Maybe (CoAxiomRule, [Type], Type) matchFamRowExt [_k, _v, x, y, row@(RowTy (MkRowTy k v flds))] = Just (axRowExtDef, [x, y, row], RowTy (MkRowTy k v $ (x, y):flds)) matchFamRowExt [k, v, x, y, _rnil] = Just (axRowExtRNilDef, [k, v], RowTy (MkRowTy k v [(x, y)])) matchFamRowExt _ = Nothing ``` I needed an extra `_rnil` case in `matchFamRowExt` because `RowExt "foo" Int RNil` did not match the first pattern match (the dumped list I got was `[Symbol, Type, "foo", Int, RNil]`). Also, is there a better way to conjure up polykinded variables out of the blue or is this fine? I thought about leaving off that info from the type, but then I would have the same question when trying to implement `typeKind` or `tcTypeKind` (for a non-empty row I can use the kinds of the head of the list, for an empty one I would need polykinded variables) My last question is about CoAxiomRules. The note says that they represent "forall as. (r1 ~ r2, s1 ~ s2) => t1 ~ t2", but it is not clear for me in what relation the constraints on the left are with the right. From the typeNats it looks like `t1` is the type family applied to the `_1` arguments and `t2` is the calculated result using the `_2` arguments. Why are we not getting just a list of types as inputs? Is the `_1` always a unification/type variable and not really a type you can use to calculate stuff? Also, if I extrapolate my observations to type families without arguments, I would assume that I do not have constraints on the left hand side as `t1` would be the family appied to the arguments (none) and `t2` would be the calculated result from the `_2` args (I do not need anything to return an empty row). Is this correct or am I horribly wrong? Thanks for your time listening to my questions, maybe I can open a PR with a bit of documentation with eventual answers. Cheers, Jan

3 4