- ghc-devs - Haskell.org

Proposal: better library management ideas (was: how to checkout proper submodules)
by Austin Seipp 20 Jun '13

20 Jun '13

So, there seems to be a fairly clear majority favor in terms of doing something I think. The question then, is what. I'm fairly convinced from Ian's response earlier that submodules *can* be dangerous if you're using a lot of high-traffic packages, especially the ability to trample each other might be bad. I could see this happening for base for example if two people are working on large features and do not coordinate a merge. Git's own merge facility doesn't suffer nearly as bad from this problem and we can figure out how we want that to happen later. However, it seems like every high-volume package is for better or worse, intimately tied to GHC. These packages are also the most problematic to rollback 'in sync' with GHC. As Geoffrey mentioned, this also becomes even MORE impossible if you use merges without fast-forwards or rebases, because dates no longer correlate accurately. These package include base, and testsuite. Probably nofib as well. In some sense, I agree with Malcolm that 'base' being GHC only is maybe unfortunate. But maybe it's not (I'll talk more about this later,) and maybe in the mean time we shouldn't lie to ourselves. So first off, I'd like to propose something I guess, which seems, to me, the best approach for one if we want to avoid developer pain with as many wins as possible in the long run. I hope this doesn't sound actively radical or anything, but it's going to totally sound actively radical (though I don't think it is): --> Let's just put base and testsuite inside the GHC repository directly. No submodules, no floating repos. Just put it directly inside and make a super commit, I guess. GHC becomes the de facto repository. And hey, why not nofib? I know, I know. People really want to split the maintenance burdens I guess, and ideologically the Haskell community is all about clean separation but, please? All of GHC HQ are the de facto maintainers of this stuff anyway. And as Jan mentioned, testsuite is really *so* crucial GHC should have it inline. The testsuite is perhaps the most important of all. There are other candidates for this treatment too, really. For example, why is template-haskell, ghc-prim, and hpc split out? GHC is the only thing that supports them. template-haskell is especially super-intrusive of an extension to support, and arguably hpc as well. integer-simple and integer-gmp follow the exact same story. Same with hoopl and dph. They're all ours. We own them. Just put them all inside GHC and be done with it. Having active fragmentation in the VCS is not necessary when there need be none. These packages de-facto ship with GHC and are very tied to it. I think people might be really opposed to a mega repository or something, but honestly? There's less maintenance, cross-package changes can work correctly and be tracked correctly in terms of history. It's less work for maintainers. It's less to explain and frankly, less to mess up. All of this I think is a huge win. OK, so radical idea is out there. Let's look at some numbers. I think ultimately anything will be a bit painful, because... $ cd ~/ghc/ghc-work $ grep -v "\#" ./packages | head --lines="-1" | wc -l 39 There are 39 sub packages which GHC requires (the -1 is because GHC itself is listed as the final entry.) These aren't all libraries of course. But that's a massive number of dependencies really, so managing them is a pain. How many are submodules already? $ grep -v "\#" packages | head --lines="-1" | awk '{print $3}' | grep "^-" | wc -l 14 So there are 14 submodules, and 25 packages that are free floating. This is a very very large amount of dependent packages. I guess that's just the price we pay. Let's say that hypothetically, we fold all those packages I said into GHC (base, testsuite, nofib, template-haskell, integer-simple, integer-gmp, hpc, ghc-prim.) That leaves 14 submodules and 17 floaters. I actually believe that most of the submodules right now are a fairly good trade off, because as designed they've all got upstreams. That's good. But what about things that are *not* submodules? Let's look at the commits over all the floaters in 1 year. The command is "git log --since="1 year ago" --format=oneline . | wc -l" * ghc-tarballs: 1 * hsc2hs: 11 * haddock: 147 * array: 10 * base: 306 * deepseq: 6 * directory: 19 * filepath: 3 * ghc-prim: 9 * haskell98: 11 * haskell2010: 7 * hoopl: 13 * hpc: 13 * integer-gmp: 29 * integer-simple: 8 * old-time: 5 * old-locale: 3 * process: 40 * template-haskell: 19 * unix: 32 * testsuite: 825 * nofib: 50 * parallel: 5 * stm: 31 * dph: 95 Remember, a lot of the commits in several of these repositories are somewhat closely tied to GHC commits. Testsuite especially, so the numbers lie a little. But *now* let's take out all the ones we wanted to fold in. * ghc-tarballs: 1 * hsc2hs: 11 * haddock: 147 * array: 10 * deepseq: 6 * directory: 19 * filepath: 3 * haskell98: 11 * haskell2010: 7 * old-time: 5 * old-locale: 3 * process: 40 * unix: 32 * parallel: 5 * stm: 31 These are all incredibly low traffic with the exception of haddock, because I was generous and listed it anyway (even though I shouldn't because it uses the GHC API.) stm/parallel are also pretty generous I'd say. Now let's think about this. Most of these could be converted to submodules with very little loss possibly. They are not very actively touched in the process of most development cycles and after looking at a lot of the changes. It's unlikely you'll hit many merge conflicts or weird situations. And even if you do, it's probably not going to happen *often*. It's even possible a lot of these could also become upstreams with separate maintainers. A lot of these are not dependent on GHC necessarily in theory or practice possibly: unix, process, deepseq, array, directory, filepath, etc. Someone could maintain them and developers work with them. Would anyone want to be a maintainer? (I heard some people clamoring for GitHub. Become a maintainer and you can host it where you want :P) Or we could also fold them in too - mega repository style - and just say GHC HQ is the de-facto maintainer, as it is now. If someone wants to step up, we can split it out later. That would just leave 14 sub repositories which are pretty well taken care of with upstreams. Maybe a few more if some people come onboard and can maintain things. This would reduce our problems a lot I feel. Other things like ./sync-all could change to support branching and other basic multi-repo facilities as Jan said, and that's not totally unreasonable either I think. It's about making the normal case easy. We're often concerned with things being at the right granularity and sharing stuff maybe, but I think the trend is pretty frighteningly clear at this point in time - GHC is the de facto implementation of Haskell, and the number of maintainers isn't especially high. And maintaining it is a lot of work (it's truly a World Class™ programming language implementation, after all.) And having 39 repositories is scary. If that's the case, I'd say we should optimize where it counts and minimize our own burden and make it easy to track our changes, and make our workflows as simple as possible. Yes, hypothetically a competitor can come along and give us a run for our money and maybe they'll want to use base and the testsuite and all that other stuff and we'll own it and whatnot. And duplication of work etc etc. And that'll be sad. Or not. And they'll do their complete own thing and run with it. UHC has its own base and testsuite, as does JHC for example. Perhaps sharing things like that is the exception, not the rule or regular occurrence. Ultimately a software project is as much about ideals, and what we believe is worth working on with our time - just as it is about what code you're writing or using right now. Perhaps we should not hinge our development strategies on these tactics any longer when the pattern seems to be darn clear. This proposal is fairly radical. It would require the agreement of almost every single developer, because several of us have varying degrees of ownership over parts of the source that concern all of them. But like I said, it seems the majority would agree something should change, and I don't think we should give up finding it, so let's just see where our ideas take us. And I think the wins would be enormous. I also appreciate you all dealing with the novels I've written over the past few days. -- Regards, Austin - PGP: 4096R/0x91384671

10 26

Running GHC Tests for Cross-Compiler?
by Stephen Paul Weber 19 Jun '13

19 Jun '13

Anyone know the best way to build the GHC test suite so that it can be run on another device (so that I can run them for my cross-compiler)? -- Stephen Paul Weber, @singpolyma See <http://singpolyma.net> for how I prefer to be contacted edition right joseph

2 1

Literate Markdown
by Trevor Elliott 19 Jun '13

19 Jun '13

Hi All, I've implemented support for literate markdown in a branch [1]. After doing a search on the bug tracker to see if anyone had already worked on this I came across bug #4836 [2], which details some unexpected interactions between the markdown format, unlit and CPP. Specifically, unlit attempts to keep CPP in the comment sections of a literate file, which clashes with the way that headings are defined in markdown. The approach that I took is that when GHC is processing markdown files, it passes the -r and -# flags to unlit, causing it to remove CPP in the comment parts of the file. I've added support for .markdown and .md as source file extensions supported by GHC to be able to detect when this is necessary. CPP can still be used within code blocks, giving a path forward for the case where the file was generated by something that wishes to leave #line markers. I've documented the changes on the wiki at [3]. Does this approach sound reasonable? If so, should I attach a patch to #4836, or create a new ticket? Thanks! --trevor [1] https://github.com/elliottt/ghc/tree/literate-markdown [2] http://hackage.haskell.org/trac/ghc/ticket/4836 [3] http://hackage.haskell.org/trac/ghc/wiki/LiterateMarkdown

1 0

Integer constant folding in the presence of new primops
by Jan Stolarek 19 Jun '13

19 Jun '13

I'm finishing my patches for T6135, but I still have some work to do on integer-gmp and integer-simpl libraries. I changed the previously existing functions that compare Integers: {-# NOINLINE #-} eqInteger :: Integer -> Integer -> Bool eqInteger = ... to something like this: {-# NOINLINE eqIntegerPrim #-} eqIntegerPrim :: Integer -> Integer -> Int# eqIntegerPrim ... -- same as before, but uses new primops eqInteger :: Integer -> Integer -> Bool eqInteger a b = tagToEnum# (a `eqIntegerPrim` b) I noticed that in some cases this implementation prevents constant folding for Integers. I'm trying to figure out the proper way to fix this. I am considering two ways of doing it: - add new folding rules for eqIntegerPrim and other functions, leave the existing rules for eqInteger. This should fold: (100012 :: Integer) == 100012 to True (as it does now) - add new folding rules for eqIntegerPrim and other functions, remove the existing rules for eqInteger. This would fold (100012 :: Integer) == 100012 to tagToEnum# 1# Which approach is better? Another question related to this: how do I run validation with integer-simpl enabled? Wiki describes how to enable integer-simpl for the build, but I think that validation is always run against integer-gmp? Or am I wrong here? Janek

5 11

Windows failures
by Simon Peyton-Jones 19 Jun '13

19 Jun '13

Ian This is 'sh validate" on 64bit Windows. Rather a lot of failures. Might you look? The perf tests seem to fail because it's comparing with the *32* bit numbers, even though this is definitely a *64* bit laptop. Whether the entire build system thinks it's a 32 bit machine, or just the testsuite, I can't tell. Thanks Simon Unexpected failures: cabal/cabal01 cabal01 [bad exit code] (normal) cabal/cabal04 cabal04 [bad exit code] (normal) driver dynHelloWorld [exit code non-0] (dyn) driver/dynamicToo dynamicToo002 [bad stdout] (normal) driver/dynamicToo dynamicToo003 [bad exit code] (normal) dynlibs T4464 [bad exit code] (normal) dynlibs T5373 [bad exit code] (normal) ghc-api/T7478 T7478 [bad exit code] (normal) ghci.debugger/scripts break008 [bad exit code] (ghci) ghci.debugger/scripts dynbrk009 [bad exit code] (ghci) ghci/scripts T5975a [bad stderr] (ghci) llvm/should_compile T5486 [exit code non-0] (optllvm) llvm/should_compile T5681 [exit code non-0] (optllvm) llvm/should_compile T6158 [exit code non-0] (optllvm) llvm/should_compile T7571 [exit code non-0] (optllvm) llvm/should_compile T7575 [exit code non-0] (optllvm) perf/compiler T1969 [stat not good enough] (normal) perf/compiler T3294 [stat not good enough] (normal) perf/compiler T4801 [stat too good] (normal) perf/haddock haddock.Cabal [stat not good enough] (normal) perf/haddock haddock.base [stat not good enough] (normal) perf/haddock haddock.compiler [stat not good enough] (normal) perf/should_run T7797 [stat too good] (normal) rts T4850 [bad stdout] (normal) rts derefnull [bad exit code] (normal) rts divbyzero [bad exit code] (normal) runghc T7859 [bad stderr] (normal) safeHaskell/check/pkg01 ImpSafeOnly01 [exit code non-0] (normal) safeHaskell/check/pkg01 ImpSafeOnly02 [exit code non-0] (normal) safeHaskell/check/pkg01 ImpSafeOnly03 [stderr mismatch] (normal) safeHaskell/check/pkg01 ImpSafeOnly04 [exit code non-0] (normal) safeHaskell/check/pkg01 ImpSafeOnly05 [stderr mismatch] (normal) safeHaskell/check/pkg01 ImpSafeOnly06 [exit code non-0] (normal) safeHaskell/check/pkg01 ImpSafeOnly07 [stderr mismatch] (normal) safeHaskell/check/pkg01 ImpSafeOnly08 [stderr mismatch] (normal) safeHaskell/check/pkg01 ImpSafeOnly09 [stderr mismatch] (normal) safeHaskell/check/pkg01 ImpSafeOnly10 [exit code non-0] (normal) safeHaskell/check/pkg01 safePkg01 [bad exit code] (normal)

2 2

build error on Mac
by Kazu Yamamoto 19 Jun '13

19 Jun '13

Hi, I got an build error on Mac today: make -r --no-print-directory -f ghc.mk phase=final all "inplace/bin/ghc-stage1" -o utils/dll-split/dist-install/build/tmp/dll-split -hisuf dyn_hi -osuf dyn_o -hcsuf dyn_hc -fPIC -dynamic -H32m -O -hide-all-packages -i -iutils/dll-split/. -iutils/dll-split/dist-install/build -iutils/dll-split/dist-install/build/autogen -Iutils/dll-split/dist-install/build -Iutils/dll-split/dist-install/build/autogen -optP-include -optPutils/dll-split/dist-install/build/autogen/cabal_macros.h -package base-4.7.0.0 -package containers-0.5.0.0 -package filepath-1.3.0.2 -XHaskell98 -no-user-package-db -rtsopts -odir utils/dll-split/dist-install/build -hidir utils/dll-split/dist-install/build -stubdir utils/dll-split/dist-install/build -optl-Wl,-headerpad_max_install_names utils/dll-split/dist-install/build/Main.dyn_o ld: unknown option: --undefined=_ghczmprim_GHCziTypes_Izh_static_info collect2: ld returned 1 exit status make[1]: *** [utils/dll-split/dist-install/build/tmp/dll-split] Error 1 make: *** [all] Error 2 --Kazu

3 7

Nightly builds
by Simon Peyton-Jones 18 Jun '13

18 Jun '13

Hi Ian What's the state of our nightly builds? * I'd rather not rely on build infrastructure here at MSR, because I am not capable of maintaining it, and Geoff is leaving in August. (Geoff is, however, willing to continue to help on infrastructure issues, after his move to Drexel; thanks Geoff.) * I believe that our nightly snapshots are still dead. Are these constructed by buildbots? * How are the buildbots doing? Is all this awaiting our shift of abott? Or is it orthogonal? Do we need help from others? If so who? Thanks Simon Microsoft Research Limited (company number 03369488) is registered in England and Wales Registered office is at 21 Station Road, Cambridge, CB1 2FB

3 2

profiling
by Kazu Yamamoto 18 Jun '13

18 Jun '13

Hi, The following command can install acme-http with profiling if GHC 7.6.3 is used: % cabal install -p --ghc-options="-prof -fprof-auto" --enable-executable-profiling -j10 acme-http But if GHC head is used, the following error occurs: % cabal install -p --ghc-options="-prof -fprof-auto" --enable-executable-profiling -j10 acme-http In function `sofa_closure': (.data+0xe8): undefined reference to `CCCS' /ghc-head/lib/ghc-7.7.20130616/Cabal-1.17.0/libHSCabal-1.17.0_p.a(Setup__1680.p_o):(.data+0x88): more undefined references to `CCCS' follow Is this a bug? I'm using Linux 3 and Cabal version is: % cabal --version cabal-install version 1.16.0.2 using version 1.16.0 of the Cabal library --Kazu

2 2

Keeping GHC's history intact
by Iavor Diatchki 18 Jun '13

18 Jun '13

Hello, I'd noticed that occasionally my branch of GHC would fail to check-out due to missing references to the Cabal repo. For a while this was puzzling me, because as far as I know we don't do rebasing of published changes, but as I was reading another thread I found the problem. Ian explains the current workflow for collaborating with Cabal: * Initially ghc's Cabal repo is at the same commit as upstream * We make a local commit 123 in Cabal to fix some bug * Cabal upstream makes a commit 456 to fix the same bug differently * We jump to commit 456, in such a way that we don't end up merging with our 123 commit every time we pull from Cabal in the future The last step in this work-flow---where we jump directly from 123 to 456---loses information: the Cabal repo has no way of knowing that some previous version of GHC still depends on 123. As a result, commit 123 can get GC-ed, which breaks GHC's history (i.e., we can't go back to GHC as it was back when it depended on 123). I'd like to propose the following workflow for repos where GHC developers need to make temporary changes that are not intended to be merged into upstream. 1. Create a new branch 'ghc-tmp-changes' in the foreign repo (e.g., Cabal). This branch is where we'll make the temporary changes, and its history will keep track of all previous temporary changes we made, so that our history stays intact. 2. The GHC sub-module will point to: 2.1 some upstream commit, if we don't currently use any temporary changes; 2.2 some commit on the `ghc-tmp-changes` branch, if we are depending on some temporary fix. 3. To start a new set of temporary changes (i.e., our sub-module is currently pointing to upstream commit `111`) 3.1 Merge `111` into `ghc-tmp-changes` by using strategy `theirs`. This makes sure that `ghc-tmp-changes` is up to date. If this is not the first temporary change to make, then we can skip this step. 3.2 Make the change and commit it on branch `ghc-tmp-changes`, resulting in commit `222`. 3.3 Update the sub-module to point to `222`. 4. When our temporary changes are no longer needed (i.e., they got fixed in upstream in some other way), then we simply point the sub-module to the relevant upstream commit. I hope this helps, -Iavor

1 0

cabal uses dynamic-linking for lifted-base
by Kazu Yamamoto 18 Jun '13

18 Jun '13

Hi, When I try to install libfted-base, cabal uses dynamic-linking while it uses static-linking for other libraries. So, I got the following error: % cabal install lifted-base /usr/bin/ld: cannot find -lHSmonad-control-0.3.2.1-ghc7.7.20130616 /usr/bin/ld: cannot find -lHStransformers-base-0.4.1-ghc7.7.20130616 /usr/bin/ld: cannot find -lHSbase-unicode-symbols-0.2.2.4-ghc7.7.20130616 If I explicitly enable/disable dynamic-linking, lifted-base can be installed. That is, the following two commands are valid: % cabal install lifted-base --enable-shared % cabal install lifted-base --disable-shared Why dynamic-linking is used for lifted-base (only)? --Kazu

1 0