- ghc-devs - Haskell.org

Changes to UserSettings.hs
by Matthew Pickering 10 Jan '19

10 Jan '19

Hi all, Last night my patch got merged which changed some aspects of the hadrian settings file. If you have a custom settings file (ie hadrian/UserSettings.hs), you need to remove reference to the `stage1Only` variable and add a new variable called `finalStage` which should default to `Stage2`. https://gitlab.haskell.org/ghc/ghc/commit/6486c6e49c53e75f37ed732b38c5be7ae… Cheers, Matt

1 0

Gitlab notes
by Simon Peyton Jones 09 Jan '19

09 Jan '19

Friends I've added a Wiki page to summarise things I've learned about Gitlab. Please do correct any errors I've made! https://ghc.haskell.org/trac/ghc/wiki/WorkingConventions/GitLabSPJ Thanks Simon

5 6

Hadrian
by Simon Peyton Jones 09 Jan '19

09 Jan '19

Alp You've written a helpful wiki page describing how to use Hadrian https://gitlab.haskell.org/ghc/ghc/blob/master/hadrian/doc/make.md Could you document where build-products live? I often look at .hi files. Could you add * How to do the equivalent of "sh validate" * When validating, how to do things like this: GhcStage1HcOpts += -DDEBUG -O GhcStage2HcOpts += -Wwarn -ticky GhcLibHcOpts += -ticky -Wwarn BUILD_PROF_LIBS = NO SplitObjs = YES HADDOCK_DOCS = NO BUILD_SPHINX_HTML = NO BUILD_SPHINX_PDF = NO * After modifying the source code, how to make stage1 compiler (only). In 'make' it is $ cd compiler; make 1 * After modifying the source code, build the stage2 compiler (only). In 'make' it is $ cd compiler; make 2 Perhaps you say build -freeze1? * In the testsuite, how to use the stage1 compiler $ make stage=1 TEST=T2543 * In nofib, how to build one benchmark, with extra flags, perhaps even with the stage1 compiler $ make stage=1 EXTRA_HC_OPTS="-ticky -ddump-simpl -ddump-stg" Thanks! Simon

2 4

Blocking a task indefinitely in the RTS
by Phyx 09 Jan '19

09 Jan '19

Hi All, I'm looking for a way to block a task indefinitely until it is woken up by an external event in both the threaded and non-threaded RTS and returns a value that was stored/passed. MVar works great for the threaded RTS, but for the non-threaded there's a bunch of deadlock detection in the scheduler that would forcibly free the lock and resume the task with an opaque exception. This means that MVar and anything derived from it is not usable. STMs are more expensive but also have the same deadlock code. So again no go. The reason it looks like a deadlock to the RTS is that the "Wake-up" call in the non-threaded rts will come from C code running inside the RTS. The RTS essentially just sees all tasks blocked on it's main capability and (usually rightly so) assumes a deadlock occurred. You have other states like BlockedOnForeign etc but those are not usable as a primitive. Previous iterations of I/O managers have each invented primitives for this such as asyncRead#, but they are not general and can't be re-used, and requires a different solution for threaded and non-threaded. I have started making a new primitive IOPort for this, based on the MVar code, but this is not trivial... (currently I'm getting a segfault *somewhere* in the primitive's cmm code). The reason is that the semantics are decidedly different from what MVars guarantee. I should also mention that this is meant to be internal to base (i.e no exported). So before I continue down this path and some painful debugging..., does anyone know of a way to block a task, unblock it later and pass a value back? It does not need to support anything complicated such as multiple take/put requests etc. Cheers, Tamar

4 12

[Q] Inlining done: evtRead
by Gabor Greif 08 Jan '19

08 Jan '19

Hi Simon, a simplifier question... Roughly a year ago I started learning about imported Ids, their unfoldings etc. I have very small example program that compiles on Linux. ```haskell import GHC.Event main = print evtRead ``` `evtRead` is a newtype-wrapped Int. When you compile above program with HEAD GHC without optimisation, you'll see that `evtRead` gets passed directly to `show`. But with -O1 it's unfolding will be inlined, floated to toplevel, and dumped as static global data into the using Main module. This was not the case in GHC 8.4. Not sure about 8.6 (will check). Anyway here is the inlining notice that the simplifier gave me (-ddump-inlinings -dverbose-core2core) > Inlining done: GHC.Event.Internal.evtRead > Inlined fn: (GHC.Types.I# 1#) > `cast` (Sym (GHC.Event.Internal.N:Event[0]) > :: GHC.Types.Coercible GHC.Types.Int > GHC.Event.Internal.Event) > Cont: Stop[BoringCtxt] GHC.Event.Internal.Event > I believe this is a regression, as copies of global data can pop up in potentially many different modules. What do you think? Which change could have caused this? Cheers, Gabor

3 7

Better DWARF info for Cmm procedures?
by Ömer Sinan Ağacan 07 Jan '19

07 Jan '19

Hi, Currently debugging Cmm is a bit painful because we don't have enough debug information to map assembly to Cmm lines, so I have do the mapping manually. However I realized that when building .cmm files we actually generates some debug information, in form of "ticks": //tick src<rts/Apply.cmm:631:9-37> _c2e::I64 = I64[R1 + 32]; Here the tick says that this assignment is for this Cmm line in Apply.cmm: Words = StgAP_STACK_size(ap); I was wondering what needs to be done to generate DWARF information from those so that gdb can show Cmm line we're executing, and gdb commands like `next`, `break` etc. work. I also realize that we don't consistently generate these ticks for all Cmm lines, for example, in the same Cmm dump there isn't a tick before this line: (_c2j::I64) = call MO_Cmpxchg W64(R1, stg_AP_STACK_info, stg_WHITEHOLE_info); It's actually for Apply.cmm:646. So there are two problems: - Generate ticks for _all_ Cmm lines - Generate DWARF information from those so that gdb can show current Cmm line, and commands like `next` and `break` work. Anyone know how hard would this be to implement? I'm wondering if we could turn this into a SoC project. It's a very well defined task, and given that we have some DWARF support already, perhaps it's not too hard for a SoC. Ömer

2 3

Welcome to GitLab!
by Ben Gamari 07 Jan '19

07 Jan '19

TL;DR. https://gitlab.haskell.org/ghc/ghc.git is now the official upstream GHC repository. Various introductory notes are discussed. Let me know if you have any trouble. Also, please do verify the correctness of the email address associated with your Trac account in the next few weeks. It will be used to map users when we transition Trac tickets to GitLab. Hello everyone, I am happy to announce that CI on GHC's GitLab instance [1] is now stable. At this point https://gitlab.haskell.org/ghc/ghc.git is to be considered the official upstream repository of GHC. The rest of this email is meant to serve as a brief introduction and status update. It can also be viewed on the GitLab Wiki [2]. [1] https://gitlab.haskell.org/ [2] https://gitlab.haskell.org/ghc/ghc/wikis/welcome # Getting started To get started on GitLab you will first want to either create a new account [1] or login with your GitHub credentials [2]. Once you have an account you should add an SSH key [3] so that you can push to your repositories. If you currently have commit rights to GHC notify me (Ben Gamari) of your user name so I can grant you similar rights in GitLab. [1] https://gitlab.haskell.org/users/sign_in [2] https://gitlab.haskell.org/users/auth/github [3] https://gitlab.haskell.org/profile/keys # Updating your development environment You can updated existing working directory (assuming the usual upstream remote name of `origin`) for the new upstream repository location by running the following: git remote set-url origin https://gitlab.haskell.org/ghc/ghc.git git remote set-url --push origin git@gitlab.haskell.org:ghc/ghc This is all that should be necessary; a quick `git pull origin master` should verify that everything is working as expected. # Continuous integration Continuous integration is now provided by GitLab's native continuous integration infrastructure. We currently test a variety of configurations, including many that neither Phabricator nor CircleCI/Appveyor previously tested (see [1] for an example run): * With the make build system: * x86_64/Linux on Fedora 27, Debian 8, and Debian 9 * i386/Linux on Debian 9 * aarch64/Linux on Debian 9 (currently broken due to a variety of issues) * x86_64/Windows * x86_64/Darwin * x86_64/Linux on Debian 9 in a few special configurations: * unregisterised (still a bit fragile due to #16085) * integer-simple * building GHC with -fllvm * With Hadrian: * x86_64/Linux on Debian 9 * x86_64/Windows (currently broken due to #15950) We also run a slightly larger set of jobs on a nightly basis. Note that binary distributions are saved from most builds and are available for download for a few weeks (we may put in place a longer retention policy for some builds in the future). There are admittedly a few kinks that we are still working out, particularly in the case of Windows (specifically the long build times seen on Windows). If you suspect you are seeing spurious build failures do let us know. To make the best use of our limited computational resources our CI builds occur in three stages: * lint: the style and correctness checkers which would previously be run by `arc lint` and `git push` * build: Debian 9 Linux x86_64 built with make and Hadrian * full-build: the remaining configurations If a build fails at an earlier phase no further phases will be run. [1] https://gitlab.haskell.org/ghc/ghc/pipelines/568 # Structuring your merge request With the transition to GitLab GHC is moving to a model similar to that used by GitHub. If you have a Differential on Phabricator we will finish review there. However, please post new patches as merge requests on GitLab. Note that Phabricator and GitLab have quite different models for handling patches. Under Phabricator a Differential is a single patch with no further structure; larger changes can be composed of multiple dependent Differentials. Under GitLab's model a merge request is a git branch consisting of one or more patches. Larger changes can be handled in one of two ways: a. a set of dependent merge requests, each of which to be squashed when merged. b. a single branch with each atomic change made in a single, buildable commit Due to the difficulty of maintaining dependent merge requests, I would recommend that contributors making larger changes use method (b). # Submitting your merge request for review Depending upon whether you have push rights to the GHC repository there are two ways to submit a merge request: * if you have push access you can push a branch directly to git@gitlab.haskell.org:ghc/ghc.git and open merge request. In this case please do follow the usual branch naming conventions: * prefix all branch names with `wip/` * if you are fixing a particular ticket consider using the name `wip/TNNNN` * if not you can create a fork using the "Fork" button on the project page [1] and push your branch there In either case after you have pushed your branch open a merge request against ghc/ghc [2]. [1] https://gitlab.haskell.org/ghc/ghc/forks/new [2] https://gitlab.haskell.org/ghc/ghc/merge_requests/new # Reviewing and merging merge requests As always, all contributors are encouraged to help review proposed changes. If you are unfamiliar with GitLab's review interface please see GitLab's user documentation [1]. Here are a few quick highlights for those who are familiar with GitHub but haven't yet used GitLab: * As with GitHub, GitLab supports both inline and out-of-line comments. * Comments that are actionable (known as "discussions") can be marked as resolved and collapsed. * Comments can be left on both changed and unchanged lines * Revisions of a merge request can be viewed and compared using the two drop-down menus at the top of the Changes tab * Merge requests can require approvals from particular users before considered as mergable * Merge requests can be placed in "merge when CI passes" state, which will cause merge requests to be merged as soon as they are green From this point onward all changes to GHC will be merged via GitLab's merge requests facility and must pass CI before being merged. To ensure that GHC's git history remains linear ghc/ghc will use GitLab's "fast-forward without a merge commit" merge strategy. Consequently you will be asked to rebase merge requests which are not fast-forward merges before merging (a convenient "Rebase" button will appear if the rebase can be carried out without conflicts. [1] https://gitlab.com/help/user/discussions/index.md#discussions # Status of the Trac migration Tobias will be continuing work on the Trac ticket migration after a bit of a holiday break. Hopefully by mid-January we will be able to move forward on this part of the migration; I will share more details about this as they develop. In the meantime, users of Trac should check and possibly update the email address associated with their account [1]. This address will be used to correlate Trac users with their GitLab equivalents so the correctness of this address will be important in preserving attribution information during the Trac import. [1] https://ghc.haskell.org/trac/ghc/prefs # Next steps We are actively working on cleaning up a few remaining issues with CI: * build times are still very long on Windows, despite the fact that we are only building the `quick` build flavour on that platform; consequently GitLab CI Windows builds do sometimes timeout when we are faced with long build queues. * we at times run low on disk space on our Windows builder runners, resulting in occasional spurious build failures * Appveyor builds (which are supposed to supplement the native GitLab builds) rarely seem to finish GitLab upstream has been incredibly supportive of our transition effort and has expressed interest in assisting us with issues that we encounter. Our current requests can be found on our migration effort's tracking ticket [1]. If you find any additional bugs or workflows that could be improved please do let me know and I can raise the matter with GitLab. [1] https://gitlab.com/gitlab-org/gitlab-ce/issues/55039 # Acknowledgments I would like to acknowledge several parties for their contributions to this effort: * Packet.net and Google X for their generous donation of hosting for continuous integration and web hosting * GitLab and their Open Source program for many productive discussions, their generous support, and the GitLab Ultimate license used by gitlab.haskell.org. * Davean Scies for his help procuring the hosting services that power our continuous integration. * Tweag.io for their offer of help and advice * Matthew Pickering, Alp Mestangullari, Tobias Dammers for their work in setting up the new instance, sorting out the details of the migration, and debugging problems when they arose Finally, thanks to GHC's contributors for their patience during this transition; it has been a long process which has stolen a significant amount of attention from other matters. My apologies we have been a bit less responsive than usual in code review and ticket triage over the past month or two. Regardless, I am hopeful that this wait will be worthwhile. # Final thoughts This is not only a milestone for the GitLab migration but also for GHC itself. For the first time GHC has fully-automated testing, proposed patch CI, and release generation across the full range of Tier 1 configurations it supports, with passing builds in all cases. We are very excited to begin this next chapter of GHC's development and are looking forward to your feedback on how we can further improve our new infrastructure. Onward and upwards! Cheers, - Ben

9 18

FYI panic "flatten_args wandered into deeper water than usual"
by Nicolas Frisby 07 Jan '19

07 Jan '19

While developing a typechecker plugin, I experienced this error message. > ghc: panic! (the 'impossible' happened) > (GHC version 8.6.2 for x86_64-unknown-linux): > flatten_args wandered into deeper water than usual > Call stack: > CallStack (from HasCallStack): > callStackDoc, called at compiler/utils/Outputable.hs:1160:37 in ghc:Outputable > pprPanic, called at compiler/typecheck/TcFlatten.hs:1331:24 in ghc:TcFlatten > > Please report this as a GHC bug: http://www.haskell.org/ghc/reportabug I'm sending this email just in case a dev is curious how this panic arises or if another plugin developer sees the same thing -- the error message is not very informative, and the corresponding GHC source code is a bit obtuse. Based on the relevant GHC source code and my recent plugin edits, I was able to guess correctly that this was due to me generating an extra sort argument to a promoted datacon. Specifically, I was incorrectly using > mkTyConApp (promoteDataCon _) [typeKind k,k,...] instead of just > mkTyConApp (promoteDataCon _) [k,...] Thanks! HTH. -Nick

2 1

Building dictionary terms in Core?
by Ömer Sinan Ağacan 07 Jan '19

07 Jan '19

Hi, In #15646 (recent discussion in Gitlab MR 55) we need dictionary arguments in Core (in desugarer) to apply to functions like `fromRational :: Fractional a => Rational -> a`, but we don't know how to build the dictionary term (`Fractional a`) in Core. Can anyone who know help us in the MR? Thanks, Ömer

3 2

How to use Data.Compact.Serialize
by Christopher Done 07 Jan '19

07 Jan '19

Hi, On the docs for Data.Compact.Serialize it says: http://hackage.haskell.org/package/compact-0.1.0.1/docs/Data-Compact-Serial… > Our binary representation contains direct pointers to the info tables > of objects in the region. This means that the info tables of the > receiving process must be laid out in exactly the same way as from the > original process; in practice, this means using static linking, using > the exact same binary and turning off ASLR. It seems to me that in order to use this module in any practical way (i.e. write to file from one process, and then a later run of the process reads it), you need to know a special way to build your binary which isn't fully described here. What flags, for example, should be passed to GHC to make this viable? * Turning off ASLR on Linux is done by writing 0 to /proc/sys/kernel/randomize_va_space, which applies to all programs. That's not the most isolated way to deploy an app, but I discovered that you can set this per process here: https://askubuntu.com/a/507954 * To compile GHC programs statically, use -optl-static -optl-pthread. So in total the example would be: $ stack build compact compact-0.1.0.1: download compact-0.1.0.1: configure compact-0.1.0.1: build compact-0.1.0.1: copy/register Example file from docs: $ cat main.hs {-# LANGUAGE TypeApplications #-} import System.Environment import Data.Compact import Data.Compact.Serialize main = do arg:_ <- getArgs case arg of "write" -> do orig_c <- compact ("I want to serialize this", True) writeCompact @(String, Bool) "somefile" orig_c "read" -> do res <- unsafeReadCompact @(String, Bool) "somefile" case res of Left err -> fail err Right c -> print (getCompact c) Compiling statically: $ stack ghc -- -optl-static -optl-pthread main.hs [1 of 1] Compiling Main ( main.hs, main.o ) Linking main ... [...hundred more warnings like this ...] Check that it's static: $ ldd main not a dynamic executable Write the file: $ setarch `uname -m` -R ./main write Read the file: $ setarch `uname -m` -R ./main read ("I want to serialize this",True) Can a GHC dev confirm that this is the proper way to do this? If so, I'll contribute this little example as documentation to the compact package. Cheers

3 3