- ghc-devs - Haskell.org

Expanding a particular type family in type errors
by Alexis King 05 May '20

05 May '20

Hi all, As part of my ongoing rework of arrow notation <https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3191>, I have introduced two wired-in type families, which assist in the typechecking process. The details are not terribly important (you can read the proposal if you’d like for those), but the relevant detail is that equalities of the shape ArrowStackTup a ~ b are produced by the typechecker in key places. ArrowStackTup is one of my wired-in type families, and it’s really an implementation detail. Unfortunately, I often end up with type errors that leak this detail, with expected/actual types like these ones: Expected type: ArrowStackTup '[Int] -> Bool Actual type: Int -> String This is quite annoying, as it’s exceedingly rare that these type families actually get stuck, so they can almost always be expanded in type errors. As it happens, `ArrowStackTup '[a]` expands to simply `a`, so the type error I would like to report is this one: Expected type: Int -> Bool Actual type: Int -> String Not technically challenging, but I find myself faced with the question of where this special expansion logic ought to go. It seems like it could go in any of several places: It could be hard-wired into pprType, and perhaps selectively disabled with an -fprint-* flag. This is nice in that it’s universal, but it’s almost certainly a step too far: the casts for ArrowStackTup still end up in Core, and expanding the type synonyms there would be quite confusing. The expansion could happen in tidyType, since it’s called before reporting an error. But this seems probably even worse than putting it in pprType, since it’s still used in too many places, and it isn’t supposed to actually change the type. It could be handled as an extra, ad-hoc preprocessing step in reportWanteds. This is much closer to reasonable, though it feels like quite a hack. A separate error Reporter could catch these errors before the other reporters do and perform the expansion there. But I don’t think this actually makes sense, as the above example demonstrates that ArrowStackTup might be buried inside another type and in fact might not actually be the source of the type error at all! It could be done last-minute in mkEqErr. But I don’t know if this is too late, and ArrowStackTup could leak into an error through some other code path. Of those options, the best one I’ve come up with seems to be option 3, an ad-hoc preprocessing step in reportWanteds. Does that seem reasonable? Or is it too much of a kludge? Thanks, Alexis

2 5

ZuriHac 2020 - GHC Track - Call for Contributions
by Andreas Herrmann 05 May '20

05 May '20

Dear GHC developers, As you may have heard this year's ZuriHac will be an online event due to COVID-19 [1]. Nonetheless, we would like to organize a GHC track again this year with the goal to foster contributions to GHC and teach newcomers how to participate in GHC's development. To that end we would like to invite you to hold seminars, workshops, or hack sessions on topics centered around GHC. Please let us know by replying to this email if you would like to contribute to this year's GHC track. ZuriHac will take place from Friday 12 June to Sunday 14 June. We are still working out the technical details. However, we are planning to provide a virtual space for people to collaborate on projects, participate in voice/video chats, and live stream talks. Best, Andreas [1]: https://zfoh.ch/zurihac2020

1 0

Preview of new profiling mode - profile by info table
by Matthew Pickering 03 May '20

03 May '20

Hi all, I have hacked together a new profiling mode code named "Profile by Info Table" which uses the address of the info table as the band identity in the profile. The main point of this is that you can then look up the source location of the info table in the dwarf information when rendering the chart. For example, in this profile I have only included THUNK closures, which you can reliably look up in the dwarf information. You can't include things like constructor closures for example as every constructor has the same info table regardless of where it comes from in the source. In the profiles below if you click on the Closure Descs tab you can see the precise source position the band originated from. As a test, I compiled the Cabal library with -O2 and here is the result. Firstly a profile sorted by size of the bands. https://mpickering.github.io/ghc-hi-cabal-o2.eventlog.html I also added a mode to eventlog2html which calculates the gradient of the line so you can more easily spot things which are leaking but not very big themselves which might be retaining other closures. https://mpickering.github.io/ghc-hi-cabal-o2-gradient.eventlog.html We can see clearly here that there are calls to compute unfoldings which remain unevaluated for a long time and therefore retaining reference to core expressions. (See the 0x177ff78 closure for example). The only issue is that at the moment reading the DWARF information takes about 48gb of memory. Cheers, Matt

1 0

Hadrian build with DWARF information doesn't contain as much debug information as I would expect
by Matthew Pickering 03 May '20

03 May '20

I followed the instructions on the wiki to enable debug symbols in my build of GHC. (https://gitlab.haskell.org/ghc/ghc/-/wikis/building/hadrian#enabling-dwarf-…) So I added these flags to may hadrian.settings file stage1.*.ghc.hs.opts += -g3 stage1.*.cabal.configure.opts += --disable-library-stripping --disable-executable-stripping stage1.ghc-bin.ghc.link.opts += -eventlog The resulting executable has debug information in it for the executable component but not for any of the libraries in including the compiler library. ("../sysdeps/x86_64/start.S",Just 4414944,Just 4414987,139633489318136) ("init.c",Nothing,Nothing,139633489318136) ("../sysdeps/x86_64/crti.S",Nothing,Nothing,139633489318136) ("ghc/Main.hs",Just 4415312,Just 4615455,139633489318136) ("ghc/GHCi/Leak.hs",Just 4615480,Just 4623414,139633489318136) ("ghc/GHCi/UI.hs",Just 4623440,Just 5461990,139633489318136) ("ghc/GHCi/UI/Info.hs",Just 5461992,Just 5571230,139633489318136) ("ghc/GHCi/UI/Monad.hs",Just 5571232,Just 5679695,139633489318136) ("ghc/GHCi/UI/Tags.hs",Just 5679696,Just 5704775,139633489318136) ("ghc/GHCi/Util.hs",Just 24,Just 173,139633489318136) ("../sysdeps/x86_64/crtn.S",Nothing,Nothing,139633489318136) I tried building a project with cabal and the resulting executable had debug information for every file in the dependencies as well as the main project. So how do I convince hadrian to include the correct information? Is it a bug in hadrian? I checked the command line when building the library and `-g3` is passed. Cheers, Matt

3 3

Justifying sched_yield() in the RTS
by Travis Whitaker 01 May '20

01 May '20

Hello GHC devs, Through the course of reading some recent material posted by Linus Torvalds ^1 ^2, I remembered stumbling upon GHC Trac #3553 ^3 some years ago. Looking past the harsh tone Linus used in his notes, I think he makes some pretty reasonable points about the problems that can be caused by using spinlocks in userspace. Specifically: - A group of threads yielding while waiting for a spinlock are, in effect, simply trying to coax the scheduler into scheduling the thread that is holding the lock. This is much easier for the scheduler to do correctly with a futex or similar, since the scheduler has some context around which tasks are blocking/waking each other up. - Apparently the Linux scheduler (and maybe other schedulers) has some smarts for preserving cache locality while choosing which physical execution units run which tasks, and reordering threads in the run list in an ad-hoc way with sched_yield() interferes with this mechanism. - Using sched_yield() (or other random interference with run lists) can cause disproportionate havoc on NUMA systems, where jostling around the lock-holding thread by burning time slices and yielding is especially costly. All that said, there is perhaps one concrete advantage (aside from absolute performance) to the current spinlock implementation: the only functionality it requires from the host OS is a yield-like operation. A yielding spinlock occupies a comfortable local optimum, achieving decent performance across platforms with minimal exposure to cross-OS API differences. Revisiting #3553, it seems the only reason that a futex was not used in place of a spinlock with sched_yield() was that, despite all of the points against it, the spinlock code empirically performed better. However, these tests were performed years ago and the futex implementation in the Linux kernel has changed quite a bit. I think there is a healthy amount of empirical evidence from the GHC user community that there are problems afoot with the way parallel GC does locking, and indeed I wonder if the bad cases Linus described are playing a role. Advice like "use -qg" or "use -qnM with small M" is often parroted (this Twitter thread ^4 contains both pieces of advice). Furthermore, I would wager that an RTS using futexes for locking rather than spinning plays nicer with other CPU intensive tasks on the same machine. The "scheduler storm" caused by a large number of threads waiting for a spinlock could have a negative impact on neighboring processes, e.g. a large number of HECs spinning certainly don't do any favors for a busy neighboring DB process. I'm curious if others have thoughts on reviving the futex investigation. Perhaps the futexes provided by modern Linux are more competitive with the current spinlock implementation, or perhaps better scaling on machines with high core counts is worth some cost. In the case that futexes remain handily defeated by yielding spinlocks, it's a worthy endeavor to explain precisely _why_ this happens. Other programs have seen performance issues crop up when the kernel makes subtle changes to how sched_yield() works. ^5 1: https://www.realworldtech.com/forum/?threadid=189711&curpostid=189723 2: https://www.realworldtech.com/forum/?threadid=189711&curpostid=189752 3: https://gitlab.haskell.org/ghc/ghc/issues/3553 4: https://twitter.com/ndm_haskell/status/1076187051610042368?s=20 5: https://lwn.net/Articles/31462/

3 3

git prune?
by Simon Peyton Jones 01 May '20

01 May '20

I'm getting lots of the messages below. Running 'git prune' makes no difference. Simon simonpj@MSRC-3645512:~/code/HEAD-5/testsuite/tests/th$ git push --set-upstream origin wip/T18121 Counting objects: 13, done. Delta compression using up to 20 threads. Compressing objects: 100% (13/13), done. Writing objects: 100% (13/13), 1.64 KiB | 418.00 KiB/s, done. Total 13 (delta 11), reused 0 (delta 0) remote: remote: To create a merge request for wip/T18121, visit: remote: https://gitlab.haskell.org/ghc/ghc/-/merge_requests/new?merge_request%5Bsou… remote: remote: warning: The last gc run reported the following. Please correct the root cause remote: and remove gc.log. remote: Automatic cleanup will not be performed until the file is removed. remote: remote: warning: There are too many unreachable loose objects; run 'git prune' to remove them. remote:

2 1

Connecting the Wolfram Language to Haskell
by Simon Peyton Jones 30 Apr '20

30 Apr '20

Friends Short summary: can someone familiar with using the GHC API offer advice on getting the Wolfram Language connected to Haskell? Many of you will know of the Wolfram Language<https://www.wolfram.com/language/>, which is IMHO pretty cool and not well enough known. There are lots of interesting things about it, but a huge thing is that it's deeply connected to the vast pile of carefully curated data that Wolfram Research has gathered and organised over several decades. Anyway, they'd like to be able to invoke Haskell code from the Wolfram Language interpreter. But as you'll see below, they are understandably stumbling on the engineering aspects of doing so. Rather than spam Stephen Wolfram himself, I'm cc'ing Todd Gayley who is the team lead. (Hi Todd.) I'd really appreciate it if some of you could help advise him. It would be terrific to have a direct connection between Wolfram and Haskell! Many thanks Simon From: Stephen Wolfram Sent: 01 April 2020 01:49 To: Simon Peyton Jones <simonpj(a)microsoft.com> Subject: Haskell question, etc. Simon --- [...snip...] What brings me to email is a Haskell question: It's probably best that I just forward what my team wants to ask: ========= A powerful feature of the Wolfram Language is that it allows users to evaluate code in other programing languages, returning the results to a Wolfram notebook (https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Freference…) We have implemented this feature for a number of languages, including Python, NodeJS, Julia, and Ruby. We would very much like to include Haskell in this set, but we have little Haskell expertise, and have hit a stumbling block. We would appreciate it if some Haskell experts could help us out with this feature. We have the following basic line of code for evaluating a string of Haskell code: r <- liftIO $ runInterpreter $ do { setImports ["Prelude"]; eval strToEvaluate} The problem is that this is a one-shot evaluation, and we want a long-lived interactive Haskell session, to which a series of inputs can be directed. We have been told that we have to use GHCi for that, but we don't know how to do it. The basic flow of our functionality is as follows: 1) User calls StartExternalSession["LanguageName"] to launch an interpreter for the language. This process remains running and can be used for multiple calls. 2) User calls ExternalEvaluate[session, "some code"] to execute the given code in the external language and return a result converted into native Wolfram Language types (strings, numbers, lists, associations, etc.) ZeroMQ is used as the transport between Wolfram Language and the external language interpreter, and JSON is the data format we use to return results. Beyond the basics of hooking up GHCi for this type of session use, we also would like assistance in introspection of Haskell results so that they can be sent back to WL in the most useful form. This is the general structure of what we do for other languages: if result is an object return object data else if result is a function return function data else return JSON form of the expression We have attached a simple file of Haskell code that one of our engineers has successfully used to get a basic evaluation of Haskell code from the Wolfram Language, but it uses the single-shot evaluation code that was given above, and so is not suitable. We would appreciate any help that you can give us, or developers or resources you can point us at, to assist in integrating Haskell into our ExternalEvaluate system. ======== Any help greatly appreciated! (Not least because I'd personally like to play with this... :) ) --- Stephen

2 1

Why doesn’t the simple optimizer fuse casts?
by Alexis King 28 Apr '20

28 Apr '20

This question is spurred by curiosity more than anything else, but I’ve noticed that the simple optimizer doesn’t fuse nested casts, and I’m wondering if there’s any reason it couldn’t. To make what I’m talking about more concrete, suppose we have an expression like this: (x |> co) |> sym co It seems like it would be trivial for simpleOptExpr to fuse the nested casts to get x |> co; sym co and then the coercion optimizer could get rid of it entirely. Moreover, this seems within the spirit of the simple optimizer, since it’s really just “cleaning up” an expression. Is there any reason the simple optimizer doesn’t do this, or is it just something nobody implemented? (For context, I’ve recently been staring at a lot of -ddump-ds output, and there happen to be a bunch of nested casts in the result that are really just noise. It would be nice if the simple optimizer got rid of them for me.) Alexis

4 5

RE: T13456
by Simon Peyton Jones 28 Apr '20

28 Apr '20

Ben I'm still getting framework failures from the testsuite, as below. But now it's not just me: it's CI! See !2600 which is failing in this way. It'd be good to nail this... it seems wrong to have to ignore framework failures when checking that a build validates. Simon | -----Original Message----- | From: Simon Peyton Jones | Sent: 20 April 2020 21:42 | To: Ben Gamari <ben(a)well-typed.com> | Subject: RE: T13456 | | Thanks! | | | -----Original Message----- | | From: Ben Gamari <ben(a)well-typed.com> | | Sent: 20 April 2020 18:57 | | To: Simon Peyton Jones <simonpj(a)microsoft.com>; ghc-devs <ghc- | | devs(a)haskell.org> | | Subject: Re: T13456 | | | | Simon Peyton Jones via ghc-devs <ghc-devs(a)haskell.org> writes: | | | | > I'm getting this failure (below) from validate fairly consistently. | | > It is often silenced by adding an empty file | | > ghci/should_run/T13456.stderr But it's troubling. Does anyone else | see | | this? How can I debug it? | | > | | Indeed this is odd. I have not seen this in CI or my local builds. It's | | possible that I have seen it in local builds that were failing for | other | | reasons but ignored it. | | | | While I don't know why you are seeing these failures in general, the | fact | | that they are reported as framework failures is arguably a bug. I would | | argue that we should treat a non-existing .stderr file as we would an | | empty file. I've opened !3121 fixing this. Hopefully you will see a | more | | helpful error message with this patch. | | | | Cheers, | | | | - Ben

2 2

Desugaring matches with already-desugared RHSs?
by Alexis King 27 Apr '20

27 Apr '20

Hi all, I’m currently in the process of rewriting most of the arrow desugaring code. One of the most awkward parts of the current implementation is the way case commands are desugared. Given a case command like case e1 of A a b -> cmd1 B c -> cmd2 C d e f -> cmd3 the desugarer actually replaces each command on the RHS with an Either-wrapped tuple to get something like this: arr (\env -> case e1 of A a b -> Left (Left (a, b)) B c -> Left (Right c) C d e f -> Right (d, e, f)) >>> ((cmd1 ||| cmd2) ||| cmd3) This means the RHSs of the case expression are really already desugared, and ideally they would be CoreExprs, but matchWrapper expects the RHSs to be HsExprs. The current implementation accommodates this restriction by building fake HsExprs with no location information, but this means the logic for building the tuples in the RHSs has to be duplicated (since other places do want CoreExprs). I was thinking it would be nice to avoid this hack, but I’m not sure what the best way to do it is. One way would be to create a variant of matchWrapper with a type like matchWrapper' :: HsMatchContext GhcRn -> Maybe (LHsExpr GhcTc) -> MatchGroup GhcTc rhs -> (rhs -> DsM CoreExpr) -- how to desugar the RHSs -> DsM ([Id], CoreExpr) and update dsGRHSs to accept an extra argument as well. Then the arrow desugaring code could just pass `return` to matchWrapper' so it wouldn’t touch its RHSs. But I’m not sure if this approach makes sense — nothing else in the desugarer seems to work this way. Is there a better approach I’m not seeing? Thanks, Alexis

2 1