- ghc-devs - Haskell.org

Feasibility of native RTS support for continuations?
by Alexis King 03 Mar '20

03 Mar '20

Hi all, tl;dr: I want to try to implement native support for capturing slices of the RTS stack as a personal experiment; please tell me the obstacles I am likely to run into. Much more context follows. --- I have been working on an implementation of an algebraic effect system that uses unsafe primops to be as performant as possible. However, the unavoidable need to CPS the entire program balloons heap allocation. Consider the following definition: f a b = g a >>= \c -> h (b + c) Assume `g` and `h` are not inlined. If the monad used is IO, this will be compiled efficiently: the result of `g` is returned on the stack, and no closure needs to be allocated for the lambda. However, if the monad supports capturing the continuation, the above definition must be CPS’d. After inlining, we end up with f a b = \k -> let lvl = \c -> h (b + c) k in g a lvl which must allocate a closure on the heap. This is frustrating, as it must happen for every call to a non-inlined monadic operation, even if that operation never captures the continuation. In an algebraic effect system, there are many shortcuts that avoid the need to capture the continuation, and under my implementation, large swaths of code never do so. I’ve managed to exploit that to get some savings, but I can’t escape the need to allocate all these closures. This motivates my question: how difficult would it be to allow capturing a portion of the RTS call stack directly? My requirements are fairly minimal, as continuations go: 1. Capturing a continuation is only legal from within a strict state thread (i.e. IO or strict ST). 2. The continuation is captured up to a prompt, which would be a new kind of RTS stack frame. Prompts are not tagged, so there is only ever exactly one prompt active at any time (which may be the root prompt). 3. Capturing a continuation is unsafe. The behavior of capturing a continuation is undefined if the current prompt was not created by the current state thread (and it is never legal to capture up to the root prompt). 4. Applying a continuation is unsafe. Captured continuations return `Any`, and type safety is the caller’s obligation. 5. Continuations are “functional,” which is to say applying them does not trigger any additional stack unwinding. This minimal support for primitive continuation capturing would be enough to support my efficient, safe delimited control implementation. In my ignorant mind, implementing this ought to be as simple as defining two new primops, reset# :: (State# s -> (# State# s, a #)) -> State# s -> (# State# s, a #) shift# :: ((a -> State# s -> (# State# s, Any #)) -> State# s -> (# State# s, Any #)) -> State# s -> (# State# s, a #) where reset# pushes a new prompt frame and shift# captures a slice of the RTS stack up to that frame and copies it into the heap. Restoring a continuation would copy all the captured frames onto the end of the current stack. Sounds simple enough! I would like to experiment with implementing something like this myself, just to see if it would really work, but somehow I doubt it is actually as simple as it sounds. Minor complications are fine, but what worries me are major obstacles I haven’t found in my limited attempts to learn about the RTS. So far, I’ve read the old “The New GHC/Hugs Runtime System” paper, which still seems mostly accurate from a high level, though I imagine many details have changed since then. I’ve also read the Commentary/RTS/Storage/Stack wiki page, and I’ve peeked at some of the RTS source code. I’ve also skimmed a handful of other papers and wiki pages, but it’s hard to know what I’m looking for. Can anyone point me to better resources or give me some insight into what will be hard? Thanks in advance, Alexis

4 15

Reason for skipping sanity checking threads and mut_lists before a GC?
by Ömer Sinan Ağacan 03 Mar '20

03 Mar '20

Hi, With `+RTS -DS` we call this function before and after a GC: void checkSanity (bool after_gc, bool major_gc) { checkFullHeap(after_gc && major_gc); checkFreeListSanity(); // always check the stacks in threaded mode, because checkHeap() // does nothing in this case. if (after_gc) { checkMutableLists(); checkGlobalTSOList(true); } } For some reason this skips mut lists and threads before a GC, I don't understand why that is necessary. Does anyone know the reason? Thanks, Ömer

1 0

Re: [ANNOUNCE] GHC 8.8.3 is now available
by Ben Gamari 29 Feb '20

29 Feb '20

Ben Gamari <ben(a)well-typed.com> writes: > --text follows this line-- > <#part sign=pgpmime> > > Hello everyone, > > The GHC team is proud to announce the release of GHC 8.8.3. The source > distribution, binary distributions, and documentation are available at > > https://downloads.haskell.org/~ghc/8.8.3 > > Release notes are also available [1]. > Unfortunately there has been a slight mix-up in the binary distributions associated with this release and I will need to do a re-spin. I have pulled down the binary distributions for the time being. They should be back in a few hours. Many apologies for the confusion! Cheers, - Ben

2 2

hsinspect-0.0.11 released
by Tseen She 28 Feb '20

28 Feb '20

Hello ghc developers, With many thanks to Daniel Grober, I have released 0.0.11 of hsinspect to Hackage, which now supports ghc versions 8.4.x, 8.6.x and 8.8.x. I would like to bring this tool to the attention of this community of ghc developers because it is using the ghc api exclusively to provide semantic information about .hs files, so it may be of interest to you to see a tool using your api in the wild. And selfishly, I learnt so much from Daniel's reviews of two short functions recently (which were subject to a change in ghc behavour in 8.8) that I would very much welcome further reviews by this group. The source code is split across three parts: The plugin (an alternative to hie-bios, which didn't work well for me): https://gitlab.com/tseenshe/hsinspect/-/blob/master/plugin/GhcFlags/Plugin.… The library, one file per feature: https://gitlab.com/tseenshe/hsinspect/-/tree/master/library/HsInspect and the Executable (setup, then just call the relevant library): https://gitlab.com/tseenshe/hsinspect/-/blob/master/exe/Main.hs My usecase is to provide semantic information for my Emacs mode, which is a completely from-scratch haskell-mode that I've been working on in my free time for the past year: https://gitlab.com/tseenshe/haskell-tng.el For anybody curious about the wider project, there are some screenshots of the features in the README, which also fully justifies why I undertook this endeavour: i.e. why I don't believe I could achieve what I wanted in haskell-mode or LSP. I am content for this tool to be used just by myself, my work colleagues, and anybody who is interested in contributing. Best regards, Tseen She

1 0

8.8.x change in lookupModule / findModule semantics?
by Tseen She 28 Feb '20

28 Feb '20

Hi ghc-devs, When upgrading my dev tool to use the 8.8.x api I have found that calls to lookupModule / findModule are failing on uncompilable code, whereas these calls succeeded on 8.4 and 8.6. Unlike another question that I have asked [1] which seems related, I don't intend to do any typechecking here, I only want access to the ModuleInfo. Has this been an intentional change? Is there a way to recover the original behaviour? If I cannot get this to work then it means that a new feature introduced in https://gitlab.haskell.org/ghc/ghc/merge_requests/1541 that I was quite excited about is unusable :-( [1] https://mail.haskell.org/pipermail/ghc-devs/2020-February/018655.html

2 12

Confused about PAP object layout
by Ömer Sinan Ağacan 27 Feb '20

27 Feb '20

Hi Simon, In this code: (slightly simplified) StgPtr scavenge_PAP (StgPAP *pap) { evacuate(&pap->fun); return scavenge_PAP_payload (pap->fun, pap->payload, pap->n_args); } StgPtr scavenge_PAP_payload (StgClosure *fun, StgClosure **payload, StgWord size) { const StgFunInfoTable *fun_info = get_fun_itbl(UNTAG_CONST_CLOSURE(fun)); StgPtr p = (StgPtr)payload; switch (fun_info->f.fun_type) { case ARG_GEN_BIG: scavenge_large_bitmap(p, GET_FUN_LARGE_BITMAP(fun_info), size); p += size; break; ... } return p; } Here the `size` argument in `scavenge_PAP_payload` is the number of arguments applied to the function in `pap->fun`. But when scavenging the function's bitmap we're using this number as the size of the bitmap which doesn't make sense to me, because I think size of the function's bitmap and size of the PAP's payload may be different. Or in other words I may have the same function used in many PAPs with different n_args, but that'd be buggy if this code is correct. I haven't checked every single place where we build a PAP but for example the `NEW_PAP` macro uses the argument's (another PAP) function directly, without making any bitmap-related changes, but bumps n_args by one. If the code above is right, then this new PAP will be scavenged incorrectly. Am I missing anything? Thanks, Ömer

5 13

ghc 8.8.x change in behaviour when loading in-memory modules
by Tseen She 26 Feb '20

26 Feb '20

Dear ghc devs, I am in the process of updating my dev tool, which uses the ghc api, to support ghc-8.8.x. However, I have noticed that one of my regression tests is failing, and I have isolated the problem to a change in behaviour in typecheckModule when using a Target with an in-memory buffer (i.e. targetContents is a Just). Using debug tracing, I can confirm that in the following code, the ms_hspp_buf of a ModSummary has my in-memory version: https://gitlab.com/tseenshe/hsinspect/-/blob/dfcb50a4e3437fbb4d0a3eb786b99c… minf_rdr_env' :: GHC.GhcMonad m => GHC.ModuleName -> m GlobalRdrEnv minf_rdr_env' m = do modSum <- GHC.getModSummary m pmod <- GHC.parseModule modSum tmod <- GHC.typecheckModule pmod let (tc_gbl_env, _) = GHC.tm_internals_ tmod pure $ tcg_rdr_env tc_gbl_env You can see how I setup the session in https://gitlab.com/tseenshe/hsinspect/-/blob/dfcb50a4e3437fbb4d0a3eb786b99c… And whereas in 8.4.4 and 8.6.5 the typecheckModule call is working on the in-memory representation, in 8.8.1 and 8.8.2 it is falling back to the version on disk and therefore failing, because I have made in-memory changes that are necessary. Aside: I am able to observe this because I have a function that trims everything except the module declaration and import section, and can therefore tolerate typos in the rest of the file. Now my code fails if the entire file does not typecheck, which is also incredibly slow, because I only want access to the import section. Best regards, Tseen She

1 0

Time-profiling the compiler pipeline
by Sergej Jaskiewicz 26 Feb '20

26 Feb '20

Hello everyone, Recently I've been exploring ways to improve the LLVM support in GHC. While doing that, I've encountered an issue that I think is worth looking into. Namely, the compilation times with LLVM are considerably longer than with the native code generator. To be more concrete, on my machine I've managed to build the whole toolchain in just half an hour with NCG, but the same build with LLVM-powered GHC took several hours. If we want to improve compilation times, we should be able to measure it first. As far as I understand, the only way for profiling the compiler pipeline today is building GHC with special flags: https://gitlab.haskell.org/ghc/ghc/wikis/debugging/profiling-ghc This has three important downsides: * We need build the whole compiler ourselves. If I'm just a user interested in why my program is so slow to compile and how I should fix it, this is too much for me to ask to do. * There's just too much irrelevant data in the output. For example, it doesn't tell me which part of my Haskell code is taking so long to compile. It would be nice if the report contained all the metadata that I need to go and fix my code. * If we're using LLVM, we're pretty much out of luck, because we're using an existing LLVM binary which is likely not built for profiling. There have been the same problem with the Clang compiler until recently. In Clang 9, which was released in the last year, there's a new flag '-ftime-trace' that produces a beautiful flame chart with all the metadata needed to identify bottlenecks. This flag tells Clang to produce a JSON file that can be visualized in Chrome's chrome://tracing page, or on https://www.speedscope.app. This is described in more details here: https://aras-p.info/blog/2019/01/16/time-trace-timeline-flame-chart-profile… I'm thinking about implementing similar functionality in GHC. One option would be to utilize the plugin infrastructure. Maybe I'm missing something and there are better options? — Sergej.

2 1

Windows build broken
by Andreas Klebinger 26 Feb '20

26 Feb '20

Hi devs, it seems the windows build is broken. (Can't build stage2 locally). Quickest way to reproduce is a validate of master. It also happens on CI: https://gitlab.haskell.org/ghc/ghc/-/jobs/270248 I remember ben mentioning something along the lines of builds with "old" (8.6) boot compilers being broken. Is that the culprint? How do I get back to a functioning environment? Cheers, Andreas

2 1

[ANNOUNCE] GHC 8.8.2 is now available
by Ben Gamari 25 Feb '20

25 Feb '20

Hello everyone, The GHC team is proud to announce the release of GHC 8.8.2. The source distribution, binary distributions, and documentation are available at https://downloads.haskell.org/~ghc/8.8.2 Release notes are also available [1]. This release fixes a handful of issues affecting 8.8.1: - A bug (#17088) in the compacting garbage collector resulting in segmentations faults under specific circumstances. Note that this may affect user programs even if they did not explicitly request the compacting GC (using the -c RTS flag) since GHC may fallback to compacting collection during times of high memory pressure. - A code generation bug (#17334) resulting in GHC panics has been fixed. - A bug in the `process` library causing builds using `hsc2hs` to fail non-deterministically on Windows has been fixed (Trac #17480) - A typechecker bug (#12088) resulting in programs being unexpectedly rejected has been fixed. - A bug in the implementation of compact normal forms resulting in segmentation faults in some uses (#17044) has been fixed. - A bug causing GHC to incorrectly complain about incompatible LLVM versions when using LLVM 7.0.1 has been fixed (#16990). As always, if anything looks amiss do let us know. Happy compiling! Cheers, - Ben [1] https://downloads.haskell.org/ghc/8.8.2/docs/html/users_guide/8.8.2-notes.h…

2 4