June 2013 - ghc-devs - Haskell.org

Unreliability of the build system
by Jan Stolarek 28 Jun '13

28 Jun '13

Fellow developers, I need advice on using GHC build system. Recently I've been spending more time on resolving problems with the build system itself than on developing GHC, and this is not how things are supposed to be. I have found testsuite and validation to be very unreliable - things seem to break randomly and results are not repeatable. I suspect that I might be doing something wrong and I will appreciate any advice or explanations why I'm having these problem. I have three GHC trees: * ghc-working - source tree I use for development * ghc-build - lndir'ed to ghc-working, used for building * ghc-validate - source tree used for validation, ghc-working is set up as one of the remotes Here's the situation from yesterday evening. I updated ghc-working and ghc-validate trees from the origin - according to the fingerprint.py script both trees were identical. I ran validation in ghc-validate and ghc-build trees (on the master branches, no my changes). The only difference between them should be that ghc-build is symlinked. Validation in ghc-validate passed with two failures (others on IRC confirmed they are having same failures), while validation in ghc-build failed with: Haddock coverage: 100% ( 1 / 1) in 'GHC.IntWord64' 78% ( 7 / 9) in 'GHC.Types' 17% ( 1 / 6) in 'GHC.CString' 3% ( 2 / 63) in 'GHC.Tuple' 0% ( 0 / 3) in 'GHC.Debug' 0% ( 0 /342) in 'GHC.PrimopWrappers' 30% (160 /531) in 'GHC.Prim' 100% ( 3 / 3) in 'GHC.Magic' 38% ( 6 / 16) in 'GHC.Classes' haddock: internal error: /dane/uczelnia/projekty/ghc-build/inpl ace/lib/html/Ocean.std-theme/ocean.css: copyFile: does not exis t (No such file or directory) gmake[1]: *** [libraries/ghc-prim/dist-install/doc/html/ghc-pri m/ghc-prim.haddock] Błąd 1 gmake[1]: *** Kasuję plik `libraries/ghc-prim/dist-install/doc/ html/ghc-prim/ghc-prim.haddock' Today I ran ./sync-all pull in both trees, but no new commits were added (according to fingerprint). I run the validation again and the validation in ghc-build fails as previously, while the validation in ghc-validate reports over 200 unexpected failures. What is more, I found that rerunning some of the failing tests (make TEST=whatever) often results with these tests passing. This unreliability is very frustrating. I've been having these problems since I started working on GHC in December. I'm wondering whether this is a common thing or am I doing something not-so-obviously wrong? As usual, I will appreciate any advice from those more experienced than me. Janek

8 39

Stage1Only
by Trevor Elliott 28 Jun '13

28 Jun '13

Hi Everyone, I've been trying to produce just a stage1 compiler by setting Stage1Only in my mk/build.mk. What I've been discovering, is that the build system seems to ignore this (or I haven't set it in the right place) and trys to build a stage2 compiler and its associated tools. What is the best way to stop the build system after stage1 + libraries has finished? Thanks! --trevor

4 14

invoking front-end phases from within a GHC plugin?
by Nicolas Frisby 26 Jun '13

26 Jun '13

I'd like to generate new declarations/bindings from within a GHC plugin. My current concrete goal is just to add a new vanilla data definition, but long-term it could be more open-ended. Crucially, these new declarations will reference variables bound by the original module and its imports. Long-term, it'd be nice if we could invoke the parser/renamer/typechecker in order to do so. Broadly, the required plumbing seems doable. I wrote a custom initTc that initializes the TcGblEnv from the ModGuts that the plugin receives. This is enough to invoke the renamer/typechecker/desugarer. The resulting new ModGuts can then be merged into the original ModGuts, perhaps after some Simplification. Has anyone explored/designed down this path? I've got some questions about a snag I hit. Parsing seems fine (superficially) but the renamer fails. I'm re-using the global Rdr environment from the plugin's input ModGuts, so that free vars from the new syntax will reference other bindings in the original module and its imports. However, that environment is keyed on the uniques of FastStrings, ultimately raising "name not found" errors. This happens because the global variable for the FastString Table is not being shared between the actual compiler and the new image of the GHC library that the plugin is using. One solution seems to be including the FastString Table in the "reinitializeGlobals" mechanism. This would require a GHC patch. Questions: 1) Was the FastString Table intentionally left out of the reinitializeGlobals mechanism? 2) Are there similar obstacles lurking beyond this one that people know about? How incompatible is the invocation of front-end phases wrt the current plugin architecture? My design assumes the HscEnv and ModGuts received by the plugin include enough information to reconstruct a meaningful TcGblEnv representing the original module. This seems to be at least nearly true. I am, however, not very well-versed with how imports are handled (EPS and such), so I don't know if the FastString Table issue is just the tip of the iceberg. Thanks for your thoughts.

2 3

validate results on Mac
by Kazu Yamamoto 25 Jun '13

25 Jun '13

Hi, This is just FYI. Here is the results of validate on Mac yesterday: Unexpected passes: driver T5313 (normal) perf/should_run T149 (normal) Unexpected failures: ../../libraries/process/tests process007 [bad stdout] (normal) codeGen/should_run cgrun071 [bad exit code] (normal) driver T2507 [bad stderr] (normal) ghc-api/T7478 T7478 [bad exit code] (normal) perf/compiler T3064 [stat not good enough] (normal) perf/compiler T4801 [stat too good] (normal) --Kazu

1 0

PROPOSAL: Literate Haskell in markdown noation
by Iavor Diatchki 25 Jun '13

25 Jun '13

Hello, I think that good support for writing literate Haskell in markdown notation would be great! Over the weekend I updated the wiki page with a fairly detailed description of the proposal, which Trevor implemented. What do folks think about the feature, and if we are happy with the design, would there be objections if I merged it in? -Iavor On Wed, Jun 19, 2013 at 4:28 PM, Trevor Elliott <trevor(a)galois.com> wrote: > Hi All, > > I've implemented support for literate markdown in a branch [1]. After > doing a search on the bug tracker to see if anyone had already worked on > this I came across bug #4836 [2], which details some unexpected > interactions between the markdown format, unlit and CPP. Specifically, > unlit attempts to keep CPP in the comment sections of a literate file, > which clashes with the way that headings are defined in markdown. > > The approach that I took is that when GHC is processing markdown files, it > passes the -r and -# flags to unlit, causing it to remove CPP in the > comment parts of the file. I've added support for .markdown and .md as > source file extensions supported by GHC to be able to detect when this is > necessary. CPP can still be used within code blocks, giving a path forward > for the case where the file was generated by something that wishes to leave > #line markers. I've documented the changes on the wiki at [3]. > > Does this approach sound reasonable? If so, should I attach a patch to > #4836, or create a new ticket? > > Thanks! > > --trevor > > > [1] https://github.com/elliottt/**ghc/tree/literate-markdown<https://github.com/elliottt/ghc/tree/literate-markdown> > [2] http://hackage.haskell.org/**trac/ghc/ticket/4836<http://hackage.haskell.org/trac/ghc/ticket/4836> > [3] http://hackage.haskell.org/**trac/ghc/wiki/LiterateMarkdown<http://hackage.haskell.org/trac/ghc/wiki/LiterateMarkdown> > > > _______________________________________________ > ghc-devs mailing list > ghc-devs(a)haskell.org > http://www.haskell.org/mailman/listinfo/ghc-devs > >

6 5

Patch/feature proposal: Provide access to the runStmt sandbox ThreadID
by Edsko de Vries 21 Jun '13

21 Jun '13

Hi all, This proposal is related to http://hackage.haskell.org/trac/ghc/ticket/1381, which Simon Marlow closed through commit https://github.com/ghc/ghc/commit/02c4ab049adeb77b8ee0e3b98fbf0f3026eee453. The problem, in a nutshell, is "how do we terminate a code snippet started with runStmt"? Before Simon's patch the only way was to disable ghc's sandboxing, so that the snippet would run in the same thread as the thread that called runStmt in the first place, and then send an asynchronous exception to that thread. This is the approach we used to take. It's a little tricky to get right (have to make sure that these exceptions are thrown only at the right times), but we thought we had it working okay. Until, that is, we realized we had a very nasty problem: snippets were being unexpected interrupted. To debug this, we introduced a CustomUserInterruption data type to serve as the exception that we were throwing. This had two purposes: first, we would be use that if we saw a CustomUserInterrupt that it could only have come from one particular throw, and second, the CustomUserInterrupt carried an integer which was incremented on every throw so that we never threw the same exception twice. What we realized is that snippets were being interrupted by *old* exceptions; that is, exceptions that we had thrown to *previous* snippets (and had been caught too). This should obviously never happen. Ian managed to reproduce this behaviour in a completely different setting ( http://hackage.haskell.org/trac/ghc/ticket/5902#comment:5) and we think that something similar (unsafePerformIO related) must be happening inside ghc. Moreover, a further conjecture is that this only happens when runStmt is still compiling the snippet to be executed (as opposed to the snippet actually executing) -- i.e., that the exception gets stuck in the bowels of ghc somewhere. We don't have any hard evidence for this, other than the problem has not appeared again with the proposed patch (but that might be luck, as it depends on timing). The patch as we currently have it is against 7.4.2, so pre Simon's change for the sandbox behaviour -- but I don't think that Simon's changes affect the above problem. The core of our patch is -sandboxIO :: DynFlags -> MVar Status -> IO [HValue] -> IO Status -sandboxIO dflags statusMVar thing = +sandboxIO :: DynFlags -> MVar Status -> MVar (Maybe ThreadId) -> IO [HValue] -> IO Status +sandboxIO dflags statusMVar tidMVar thing = mask $ \restore -> -- fork starts blocked - let runIt = liftM Complete $ try (restore $ rethrow dflags thing) + let thing' = gbracket (myThreadId >>= putMVar tidMVar . Just) + (\() -> modifyMVar_ tidMVar (\_ -> return Nothing)) + (\() -> thing) + runIt = liftM Complete $ try (restore $ rethrow dflags thing') in if dopt Opt_GhciSandbox dflags then do tid <- forkIO $ do res <- runIt putMVar statusMVar res -- empty: can't block That is, sandboxIO takes an additional MVar (Maybe ThreadId): 1. Initially this MVar should be empty. The MVar gets initialized to Just the Thread ID of the sandbox when the user code starts running. This means that if an secondary thread attempts to read the MVar (in order to kill the snippet), that secondary thread will block until the user code starts running -- it will not interrupt ghc when compiling the snippet. 2. When the user code exists the MVar is updated to be Nothing. This means that if the auxiliary thread reads the MVar and finds Nothing it knows that the snippet has already terminated. 3. When the auxiliary thread finds Just a thread ID (it must use withMVar rather than readMVar to avoid a race condition) it can safely throw the asynchronous exception. The remainder of the patch just propagates these changes up so that runStmt takes this MVar as an argument, too. Probably when integrating the patch into ghc it would be better to leave runStmt along and provide a runStmt' that takes the additional argument. Again, any and all feedback on the above would be appreciated. Edsko

2 2

haddock test suite
by Richard Eisenberg 21 Jun '13

21 Jun '13

Hi all, I've implemented the proposal at http://hackage.haskell.org/trac/ghc/wiki/NewAxioms/Nonlinearity, which involved a few changes to HsDecl datatypes (FamilyDecl, TyFamInstDecl, etc.), thus requiring a change to Haddock. I tried to be as minimal as I could there, but prudence tells me I should run the haddock test suite. How do I do so? The instructions seem to say I should just `cabal test`, but that requires `cabal build`, which fails, seemingly because of import path issues: cam-05-unx:10:59:54 /5playpen/eir/ghc/head/utils/haddock> cabal build Building haddock-2.13.2... Preprocessing library haddock-2.13.2... src/Haddock/Convert.hs:29:8: Could not find module `CoAxiom' Use -v to see a list of the files searched for. Any advice? Separately, I see that Haddock does not generate any output for type family instances. I'd be happy to bring someone up to speed about the new type families and such so that this gap could be closed. Thanks! Richard

2 3

Proposal: better library management ideas (was: how to checkout proper submodules)
by Austin Seipp 20 Jun '13

20 Jun '13

So, there seems to be a fairly clear majority favor in terms of doing something I think. The question then, is what. I'm fairly convinced from Ian's response earlier that submodules *can* be dangerous if you're using a lot of high-traffic packages, especially the ability to trample each other might be bad. I could see this happening for base for example if two people are working on large features and do not coordinate a merge. Git's own merge facility doesn't suffer nearly as bad from this problem and we can figure out how we want that to happen later. However, it seems like every high-volume package is for better or worse, intimately tied to GHC. These packages are also the most problematic to rollback 'in sync' with GHC. As Geoffrey mentioned, this also becomes even MORE impossible if you use merges without fast-forwards or rebases, because dates no longer correlate accurately. These package include base, and testsuite. Probably nofib as well. In some sense, I agree with Malcolm that 'base' being GHC only is maybe unfortunate. But maybe it's not (I'll talk more about this later,) and maybe in the mean time we shouldn't lie to ourselves. So first off, I'd like to propose something I guess, which seems, to me, the best approach for one if we want to avoid developer pain with as many wins as possible in the long run. I hope this doesn't sound actively radical or anything, but it's going to totally sound actively radical (though I don't think it is): --> Let's just put base and testsuite inside the GHC repository directly. No submodules, no floating repos. Just put it directly inside and make a super commit, I guess. GHC becomes the de facto repository. And hey, why not nofib? I know, I know. People really want to split the maintenance burdens I guess, and ideologically the Haskell community is all about clean separation but, please? All of GHC HQ are the de facto maintainers of this stuff anyway. And as Jan mentioned, testsuite is really *so* crucial GHC should have it inline. The testsuite is perhaps the most important of all. There are other candidates for this treatment too, really. For example, why is template-haskell, ghc-prim, and hpc split out? GHC is the only thing that supports them. template-haskell is especially super-intrusive of an extension to support, and arguably hpc as well. integer-simple and integer-gmp follow the exact same story. Same with hoopl and dph. They're all ours. We own them. Just put them all inside GHC and be done with it. Having active fragmentation in the VCS is not necessary when there need be none. These packages de-facto ship with GHC and are very tied to it. I think people might be really opposed to a mega repository or something, but honestly? There's less maintenance, cross-package changes can work correctly and be tracked correctly in terms of history. It's less work for maintainers. It's less to explain and frankly, less to mess up. All of this I think is a huge win. OK, so radical idea is out there. Let's look at some numbers. I think ultimately anything will be a bit painful, because... $ cd ~/ghc/ghc-work $ grep -v "\#" ./packages | head --lines="-1" | wc -l 39 There are 39 sub packages which GHC requires (the -1 is because GHC itself is listed as the final entry.) These aren't all libraries of course. But that's a massive number of dependencies really, so managing them is a pain. How many are submodules already? $ grep -v "\#" packages | head --lines="-1" | awk '{print $3}' | grep "^-" | wc -l 14 So there are 14 submodules, and 25 packages that are free floating. This is a very very large amount of dependent packages. I guess that's just the price we pay. Let's say that hypothetically, we fold all those packages I said into GHC (base, testsuite, nofib, template-haskell, integer-simple, integer-gmp, hpc, ghc-prim.) That leaves 14 submodules and 17 floaters. I actually believe that most of the submodules right now are a fairly good trade off, because as designed they've all got upstreams. That's good. But what about things that are *not* submodules? Let's look at the commits over all the floaters in 1 year. The command is "git log --since="1 year ago" --format=oneline . | wc -l" * ghc-tarballs: 1 * hsc2hs: 11 * haddock: 147 * array: 10 * base: 306 * deepseq: 6 * directory: 19 * filepath: 3 * ghc-prim: 9 * haskell98: 11 * haskell2010: 7 * hoopl: 13 * hpc: 13 * integer-gmp: 29 * integer-simple: 8 * old-time: 5 * old-locale: 3 * process: 40 * template-haskell: 19 * unix: 32 * testsuite: 825 * nofib: 50 * parallel: 5 * stm: 31 * dph: 95 Remember, a lot of the commits in several of these repositories are somewhat closely tied to GHC commits. Testsuite especially, so the numbers lie a little. But *now* let's take out all the ones we wanted to fold in. * ghc-tarballs: 1 * hsc2hs: 11 * haddock: 147 * array: 10 * deepseq: 6 * directory: 19 * filepath: 3 * haskell98: 11 * haskell2010: 7 * old-time: 5 * old-locale: 3 * process: 40 * unix: 32 * parallel: 5 * stm: 31 These are all incredibly low traffic with the exception of haddock, because I was generous and listed it anyway (even though I shouldn't because it uses the GHC API.) stm/parallel are also pretty generous I'd say. Now let's think about this. Most of these could be converted to submodules with very little loss possibly. They are not very actively touched in the process of most development cycles and after looking at a lot of the changes. It's unlikely you'll hit many merge conflicts or weird situations. And even if you do, it's probably not going to happen *often*. It's even possible a lot of these could also become upstreams with separate maintainers. A lot of these are not dependent on GHC necessarily in theory or practice possibly: unix, process, deepseq, array, directory, filepath, etc. Someone could maintain them and developers work with them. Would anyone want to be a maintainer? (I heard some people clamoring for GitHub. Become a maintainer and you can host it where you want :P) Or we could also fold them in too - mega repository style - and just say GHC HQ is the de-facto maintainer, as it is now. If someone wants to step up, we can split it out later. That would just leave 14 sub repositories which are pretty well taken care of with upstreams. Maybe a few more if some people come onboard and can maintain things. This would reduce our problems a lot I feel. Other things like ./sync-all could change to support branching and other basic multi-repo facilities as Jan said, and that's not totally unreasonable either I think. It's about making the normal case easy. We're often concerned with things being at the right granularity and sharing stuff maybe, but I think the trend is pretty frighteningly clear at this point in time - GHC is the de facto implementation of Haskell, and the number of maintainers isn't especially high. And maintaining it is a lot of work (it's truly a World Class™ programming language implementation, after all.) And having 39 repositories is scary. If that's the case, I'd say we should optimize where it counts and minimize our own burden and make it easy to track our changes, and make our workflows as simple as possible. Yes, hypothetically a competitor can come along and give us a run for our money and maybe they'll want to use base and the testsuite and all that other stuff and we'll own it and whatnot. And duplication of work etc etc. And that'll be sad. Or not. And they'll do their complete own thing and run with it. UHC has its own base and testsuite, as does JHC for example. Perhaps sharing things like that is the exception, not the rule or regular occurrence. Ultimately a software project is as much about ideals, and what we believe is worth working on with our time - just as it is about what code you're writing or using right now. Perhaps we should not hinge our development strategies on these tactics any longer when the pattern seems to be darn clear. This proposal is fairly radical. It would require the agreement of almost every single developer, because several of us have varying degrees of ownership over parts of the source that concern all of them. But like I said, it seems the majority would agree something should change, and I don't think we should give up finding it, so let's just see where our ideas take us. And I think the wins would be enormous. I also appreciate you all dealing with the novels I've written over the past few days. -- Regards, Austin - PGP: 4096R/0x91384671

10 26

Running GHC Tests for Cross-Compiler?
by Stephen Paul Weber 19 Jun '13

19 Jun '13

Anyone know the best way to build the GHC test suite so that it can be run on another device (so that I can run them for my cross-compiler)? -- Stephen Paul Weber, @singpolyma See <http://singpolyma.net> for how I prefer to be contacted edition right joseph

2 1

Literate Markdown
by Trevor Elliott 19 Jun '13

19 Jun '13

Hi All, I've implemented support for literate markdown in a branch [1]. After doing a search on the bug tracker to see if anyone had already worked on this I came across bug #4836 [2], which details some unexpected interactions between the markdown format, unlit and CPP. Specifically, unlit attempts to keep CPP in the comment sections of a literate file, which clashes with the way that headings are defined in markdown. The approach that I took is that when GHC is processing markdown files, it passes the -r and -# flags to unlit, causing it to remove CPP in the comment parts of the file. I've added support for .markdown and .md as source file extensions supported by GHC to be able to detect when this is necessary. CPP can still be used within code blocks, giving a path forward for the case where the file was generated by something that wishes to leave #line markers. I've documented the changes on the wiki at [3]. Does this approach sound reasonable? If so, should I attach a patch to #4836, or create a new ticket? Thanks! --trevor [1] https://github.com/elliottt/ghc/tree/literate-markdown [2] http://hackage.haskell.org/trac/ghc/ticket/4836 [3] http://hackage.haskell.org/trac/ghc/wiki/LiterateMarkdown

1 0