- ghc-devs - Haskell.org

RE: [commit: ghc] master: Bump haddock.base max_bytes_used (8df7fea)
by Simon Peyton Jones 01 Sep '14

01 Sep '14

Urk. It's quite surprising that this particular change would increase allocation significantly. I wonder whether it just pushed it over the threshold. Simon | -----Original Message----- | From: ghc-commits [mailto:ghc-commits-bounces@haskell.org] On Behalf Of | git(a)git.haskell.org | Sent: 01 August 2014 18:57 | To: ghc-commits(a)haskell.org | Subject: [commit: ghc] master: Bump haddock.base max_bytes_used (8df7fea) | | Repository : ssh://git@git.haskell.org/ghc | | On branch : master | Link : | http://ghc.haskell.org/trac/ghc/changeset/8df7fea7cf8a32d54ac3d6772432273 | 8595bf421/ghc | | >--------------------------------------------------------------- | | commit 8df7fea7cf8a32d54ac3d67724322738595bf421 | Author: Joachim Breitner <mail(a)joachim-breitner.de> | Date: Fri Aug 1 19:55:52 2014 +0200 | | Bump haddock.base max_bytes_used | | It has reliably increased with commit 1ae5fa45, and has been stable | since then, so it does not seem to be a fluke. I did not investigate | why | that commit might have increased this value. | | | >--------------------------------------------------------------- | | 8df7fea7cf8a32d54ac3d67724322738595bf421 | testsuite/tests/perf/haddock/all.T | 13 +++++++------ | 1 file changed, 7 insertions(+), 6 deletions(-) | | diff --git a/testsuite/tests/perf/haddock/all.T | b/testsuite/tests/perf/haddock/all.T | index b17d472..49321b9 100644 | --- a/testsuite/tests/perf/haddock/all.T | +++ b/testsuite/tests/perf/haddock/all.T | @@ -17,13 +17,14 @@ test('haddock.base', | # 2014-01-22: 168 (x86/Linux - new haddock) | # 2014-06-29: 156 (x86/Linux) | ,stats_num_field('max_bytes_used', | - [(wordsize(64), 115113864, 10) | - # 2012-08-14: 87374568 (amd64/Linux) | - # 2012-08-21: 86428216 (amd64/Linux) | - # 2012-09-20: 84794136 (amd64/Linux) | - # 2012-11-12: 87265136 (amd64/Linux) | - # 2013-01-29: 96022312 (amd64/Linux) | + [(wordsize(64), 127954488, 10) | + # 2012-08-14: 87374568 (amd64/Linux) | + # 2012-08-21: 86428216 (amd64/Linux) | + # 2012-09-20: 84794136 (amd64/Linux) | + # 2012-11-12: 87265136 (amd64/Linux) | + # 2013-01-29: 96022312 (amd64/Linux) | # 2013-10-18: 115113864 (amd64/Linux) | + # 2014-07-31: 127954488 (amd64/Linux), correlates with | 1ae5fa45 | ,(platform('i386-unknown-mingw32'), 58557136, 10) | # 2013-02-10: 47988488 (x86/Windows) | # 2013-11-13: 58557136 (x86/Windows, | 64bit machine) | | _______________________________________________ | ghc-commits mailing list | ghc-commits(a)haskell.org | http://www.haskell.org/mailman/listinfo/ghc-commits

6 11

Trouble compiling fibon
by David Feuer 31 Aug '14

31 Aug '14

I'm trying to compile the fibon benchmark suite, but I'm getting a non-specific permission error. Can anyone give me a clue? == make boot - --no-print-directory; in /home/dfeuer/src/ghc-slowmod/nofib/fibon/Hackage/Bzlib ------------------------------------------------------------------------ // Codec/Compression/BZip/Stream.hsc make[2]: execvp: //: Permission denied make[2]: *** [Codec/Compression/BZip/Stream.hs] Error 127 Failed making boot in Bzlib: 1

1 0

Raft of optimiser changes
by Simon Peyton Jones 30 Aug '14

30 Aug '14

I've just pushed a bunch of Core-to-Core optimisation changes that have been sitting in my tree for ages. The aggregate effect on nofib is very modest, but they are mostly aimed at corner cases, and consolidation. Program Size Allocs Runtime Elapsed TotalMem Min -7.2% -3.1% -7.8% -7.8% -14.8% Max +5.6% +1.3% +20.0% +19.7% +50.0% Geometric Mean -0.3% -0.1% +1.7% +1.7% +0.2% The runtime increases are spurious - I checked. A couple of perf/compiler tests (i.e. GHC's own performance) improve significantly, which is a good sign. I have a few more to come but wanted to get this lot out of my hair. Simon a1a400ed * Testsuite wibbles 39ccdf91 * White space only 6c6b001e * Remove dead lookup_dfun_id (merge-o) a0b2897e * Simple refactor of the case-of-case transform bb877266 * Performance changes 082e41b4 * Testsuite wibbles 1122857e * Run float-inwards immediately before the strictness analyser. 86a2ebf8 * Comments only 6d48ce29 * Make tidyProgram discard speculative specialisation rules fa582cc4 * Fix an egregious bug in the NonRec case of bindFreeVars b9e49d3e * Add -fspecialise-aggressively dce70957 * Compiler performance increases -- yay! a3e207f6 * More SPEC rules fire baa3c9a3 * Wibbles to "...plus N others" error message about instances in scope 99178c1f * Specialise monad functions, and make them INLINEABLE 2ef997b8 * Slightly improve fusion rules for 'take' 949ad67e * Don't float out (classop dict e1 e2) 34363330 * Move the Enum Word instance into GHC.Enum 4c03791f * Specialise Eq, Ord, Read, Show at Int, Char, String 9cf5906b * Make worker/wrapper work on INLINEABLE things 8f099374 * Make maybeUnfoldingTemplate respond to DFunUnfoldings 3af1adf9 * Kill unused setUnfoldingTemplate 6e0f6ede * Refactor unfoldings e9cd1d5e * Less voluminous output when printing continuations

2 1

Can't install 32-bit ghc-7.8.1 on 64-bit xubuntu 14.04
by kyra 30 Aug '14

30 Aug '14

Hi, I have unpacked ghc-7.8.1-i386-unknown-linux-deb7.tar.xz to ~/data/Work/ghc-7.8.1-i386. Now I have the following: awson@awsonvirt:~/data/Work/ghc-7.8.1-i386$ ./configure --prefix=/home/awson/data/ghc-7.8.1-i386 checking for path to top of build tree... utils/ghc-pwd/dist-install/build/tmp/ghc-pwd-bindist: line 3: /home/awson/data/Work/ghc-7.8.1-i386/utils/ghc-pwd/dist-install/build/tmp/ghc-pwd: No such file or directory configure: error: cannot determine current directory ghc-pwd is definitely there but it does not start it seems. Perhaps, the system lacks some shared libs or something. There absolutely no problems with 64-bit ghc. Regards, Kyra

5 8

Why isn't ($) inlining when I want?
by David Feuer 30 Aug '14

30 Aug '14

tl;dr I added a simplifier run with inlining enabled between specialization and floating out. It seems incapable of inlining saturated applications of ($), and I can't figure out why. These are inlined later, when phase 2 runs. Am I running the simplifier wrong or something? I'm working on this simple little fusion pipeline: {-# INLINE takeWhile #-} takeWhile p xs = build builder where builder c n = foldr go n xs where go x r = if p x then x `c` r else n foo c n x = foldr c n . takeWhile (/= (1::Int)) $ [-9..10] There are some issues with the enumFrom definition that break things. If I use a fusible unfoldr that produces some numbers instead, that issue goes away. Part of that problem (but not all of it) is that the simplifier doesn't run to apply rules between specialization and full laziness, so there's no opportunity for the specialization of enumFromTo to Int to trigger the rewrite to a build form and fusion with foldr before full laziness tears things apart. The other problem is that inlining doesn't happen at all before full laziness, so things defined using foldr and/or build aren't actually exposed as such until afterwards. Therefore I decided to try adding a simplifier run with inlining between specialization and floating out: runWhen do_specialise CoreDoSpecialising, runWhen full_laziness $ CoreDoSimplify max_iter (base_mode { sm_phase = InitialPhase , sm_names = ["PostGentle"] , sm_rules = rules_on , sm_inline = True , sm_case_case = False }), runWhen full_laziness $ CoreDoFloatOutwards FloatOutSwitches { floatOutLambdas = Just 0, floatOutConstants = True, floatOutPartialApplications = False }, The weird thing is that for some reason this doesn't inline ($), even though it appears to be saturated. Using the modified thing with (my version of) unfoldr: foo c n x = (foldr c n . takeWhile (/= (1::Int))) $ unfoldr (potato 10) (-9) potato :: Int -> Int -> Maybe (Int, Int) potato n m | m <= n = Just (m, m) | otherwise = Nothing I get this out of the specializer: foo foo = \ @ t_a1Za @ c_a1Zb c_a1HT n_a1HU _ -> $ (. (foldr c_a1HT n_a1HU) (takeWhile (let { ds_s21z ds_s21z = I# 1 } in \ ds_d1Zw -> neInt ds_d1Zw ds_s21z))) (let { n_s21x n_s21x = I# 10 } in unfoldr (\ m_a1U7 -> case leInt m_a1U7 n_s21x of _ { False -> Nothing; True -> Just (m_a1U7, m_a1U7) }) ($fNumInt_$cnegate (I# 9))) and then I get this out of my extra simplifier run: foo foo = \ @ t_a1Za @ c_a1Zb c_a1HT n_a1HU _ -> $ (\ x_a20f -> foldr (\ x_a1HR r_a1HS -> case case x_a1HR of _ { I# x_a20R -> tagToEnum# (case x_a20R of _ { __DEFAULT -> 1; 1 -> 0 }) } of _ { False -> n_a1HU; True -> c_a1HT x_a1HR r_a1HS }) n_a1HU x_a20f) (let { b'_a1ZS b'_a1ZS = $fNumInt_$cnegate (I# 9) } in $ (build) (\ @ b1_a1ZU c_a1ZV n_a1ZW -> letrec { go_a1ZX go_a1ZX = \ b2_a1ZY -> case case case b2_a1ZY of _ { I# x_a218 -> tagToEnum# (<=# x_a218 10) } of _ { False -> Nothing; True -> Just (b2_a1ZY, b2_a1ZY) } of _ { Nothing -> n_a1ZW; Just ds_a203 -> case ds_a203 of _ { (a1_a207, new_b_a208) -> c_a1ZV a1_a207 (go_a1ZX new_b_a208) } }; } in go_a1ZX b'_a1ZS)) That is, neither the $ in the code nor the $ that was inserted when inlining unfoldr got inlined themselves, even though both appear to be saturated. As a result, foldr/build doesn't fire, and full laziness tears things apart. Later on, in simplifier phase 2, $ gets inlined. What's preventing this from happening in the PostGentle phase I added? David Feuer

4 11

build fixes
by Simon Peyton Jones 29 Aug '14

29 Aug '14

I've pushed patches that should finally fix the build. (including improved performance in the compiler itself!) Sorry about the breakage yesterday Simon

4 5

HEADS UP: full rebuild necessary
by Edward Z. Yang 29 Aug '14

29 Aug '14

Duncan has landed his changes to remove GHC's dep on Cabal, so you'll need to do a clean and full rebuild once you pull from master. Cheers, Edward

3 4

RE: Fusion
by Simon Peyton Jones 29 Aug '14

29 Aug '14

I have added a section "Ticky-ticky quick start" to our ticky-ticky profiling page, to explain how I go about dealing with the problem you describe https://ghc.haskell.org/trac/ghc/wiki/Debugging/TickyTicky Simon | -----Original Message----- | From: David Feuer [mailto:david.feuer@gmail.com] | Sent: 20 August 2014 09:33 | To: Simon Peyton Jones | Subject: Re: Fusion | | I'll be happy to try to expand it with some examples. I'm wondering if | you could help me figure something out: the (simple) cons/build rule | we discussed, along with the similar cons/augment rule, | | "cons/build" forall (x::a) (g::forall b . (a->b->b)->b->b) . x : | build g = build (\c n -> c x (g c n)) | "cons/augment" forall (x::a) (g::forall b . (a->b->b)->b->b) | (xs::[a]) . x : augment g xs = augment (\c n -> c x (g c n)) xs | | somehow *increase* allocation substantially (11.7%) in the "event" | NoFib test, and also significantly (3.6%) in constraints, somewhat | (2.4%) in nucleic2 [remember this] and 1.4% in ansi. I am having a | heck of a time trying to figure out how to track these down, and | burning loads of time recompiling GHC over and over again. On the flip | side of things, the wang test reduces allocation by 45.8% if I use | these rules (both of which fire), but only when I also use -fsimple- | list-literals for nofib/spectral/hartel/nucleic2/Main.hs. | nucleic2 still performs a little more allocation. | | On Wed, Aug 20, 2014 at 2:56 AM, Simon Peyton Jones | <simonpj(a)microsoft.com> wrote: | > Great start, thank you. Can I suggest that for each question you | give | > a concrete example? Otherwise only experts, who already know a lot | about | > rules, will understand the question, let alone the answer. I’d be | happy to | > take another look then. | > | > | > Simon | > | > | > | > From: David Feuer [mailto:david.feuer@gmail.com] | > Sent: 19 August 2014 23:30 | > To: Simon Peyton Jones | > Cc: Haskell Libraries | > Subject: Re: Fusion | > | > | > | > I've started a page at | > https://ghc.haskell.org/trac/ghc/wiki/FoldrBuildNotes | > Please feel free to add, correct, etc. | > | > On Aug 19, 2014 3:10 AM, "Simon Peyton Jones" | <simonpj(a)microsoft.com> wrote: | > | > David | > | > You've been doing all this work on improving fusion, and you | probably | > have a very good idea now about how it works, and how GHC's | libraries | > use phases and RULES to achieve it. A kind of design pattern, if you | > like; tips and tricks. | > | > I wonder if you'd feel able to write a GHC wiki page describing what | > you have learned, with examples and explanation about why it is done | that way. | > If you did this, someone who follows in your footsteps wouldn't need | > to re-learn everything. And maybe someone will say "oh, there's one | > pattern you have omitted, here it is". | > | > Thanks | > | > Simon

2 1

Moving Haddock *development* out of GHC tree
by Mateusz Kowalczyk 29 Aug '14

29 Aug '14

Hello, A slightly long e-mail but I ask that you voice your opinion if you ever changed GHC API. You can skim over the details, simply know that it saves me vast amount of time, allows me to try and find contributors and doesn't impact GHC negatively. It seems like a win-win scenario for GHC and Haddock. GHC team's workflow does not change and will not require any new commitment: I do all the work and I think it's a 1 line change in sync-all when transition is ready. Here it is: It is no secret that many core Haskell projects lack developer hands and Haddock is no exception: the current maintainers are Simon Hengel and myself. Simon does not have much time so currently all the issues and updates are up to me. Ideally I would like if some more people could come and hack on Haddock but there are a couple of problems with trying to recruit folk for this: 1. Interacting with GHC API is not the easiest thing. This is Haddock's problem but I thought I'd mention it here. 2. Haddock resides directly in the GHC tree and it is currently *required* that it compiles with GHC HEAD. This is a huge barrier of entry for anyone: today I wanted to make a fairly simple change but it still took me 3 validate runs to be at least somewhat confident that I didn't break much in GHC. On top of this I had help from Edward Z. Yang on IRC and information from him on what the issue exactly was. If I was to do everything alone it would have taken even more validates. A validate is not fast on machine by any means, it takes an hour or two. Here is what I want to do unless there are major objections: I want to move the active development away from GHC tree. Below is how it would work. For simplicity please imagine that we have *just* released 7.8.3. * Haddock development would concentrate on supporting the last public release of GHC: I stop developing against GHC HEAD and currently would develop against 7.8.3. * GHC itself checks out Haddock as a submodule as it does now. The only difference is that it points at whatever commit worked last. Let us assume it is the Haddock 2.14.3 release commit. The vital difference from current state is that GHC will no longer track changes in master branch. * Now when GHC API changes things proceed as they normally do: whoever is responsible for the changes, pops into the Haddock submodule applies the patches necessary for Haddock to build with HEAD and everyone is happy. What does *not* happen is these patches don't go into master: I ignore them and keep working with 7.8.3. * When a GHC release rolls around, I update Haddock to work with the new API so that people with new release can still use it. Once it works against new API, GHC can start tracking from that commit onwards and proceed as usual. Here are the advantages: * I don't have to work against GHC HEAD. This means I don't have to build GHC HEAD and I don't need to worry about GHC API changes. I don't waste 2-4 hours building before hacking and validating after hacking to make any minor changes and to make sure I haven't broken anything. * More importantly, anyone who wants to write a patch for Haddock can now do so easily, all they need is recent compiler rather than being forced to build HEAD. Building and validating against HEAD is a **huge** barrier of entry. * I only have to care about GHC API changes once a release and not twice a week. I think PatternSynonyms have changed 4 times in a month but the end result at release time is the same and that's what people care about. * It is less work for anyone changing GHC API: they only have to deal with their own changes and not my changes which add features or whatever. * If I break something in Haddock HEAD, GHC is not affected. * If Haddock's binary interface doesn't change, we may even allow more versions of GHC be compatible through CPP and other such trickery. If we were to do it today, it would be an increased burden on the GHC team to deal with those. * I can release as often as I want against the same compiler version. Currently doing this requires backporting features (see v2.14 branch) which is a massive pain. I no longer have to tell the users ‘yes, your bug is fixed but to get it you need to compile GHC HEAD or wait 6-12 months until next GHC release’. I have to do this a lot. Here are the disadvantages and why I think they don't make a big difference: * GHC HEAD doesn't get any new-and-cool features that we might implement. I say this doesn't matter because no one uses varying GHC HEAD versions to develop actual software, documentation and all. What I mean to say is that the only user of the Haddock that's developed in GHC tree is GHC itself. The only case where GHC actually used in-tree Haddock was when Herbert generated documentation for base-4.7 early for me to eye before the release. Even this doesn't matter because so close to the release I'll already have the existing GHC API integrated anyway. Again, it does not matter if GHC HEAD itself doesn't get pretty operator rendering or whatever right when I implement it because no one cares about it until it's release time. I know that many people simply HADDOCK_DOCS=NO to save time. The actual users only care about Haddock that works with 7.6.x, 7.8.x, 7.10.x; only GHC cares about in-betweens and only for the purpose of being able to build and validate. * GHC team can't easily contribute features and get the back immediately. In part it doesn't matter because of the previous point and in the last year or so there were no features contributed directly from GHC except those necessary to keep Haddock compiling. This just means there's no demand for such close relationship. * Haddock-affecting changes in GHC parser don't ‘take effect’ straight away. This is my loss and considering the infrequency at which such changes happen, it's a tiny price to pay to have to wait until release. * …that's it, no other disadvantages that I can think of, but that's why I'm sending it to the list to review! What's worth mentioning is that the no-external-dependencies thing still applies because even though we no longer need to compile against HEAD, we still need to compile against the tree at release time. In summary: My life gets easier because I stop wasting it on playing with whole GHC tree, GHC team's life gets easier because they don't have to deal with the changes I make. My life gets even easier because I only have to make big API updates once a release. I can actually start looking for contributors. When a release rolls around, GHC and Haddock ‘meet up’, we make sure it all works, release happens, GHC starts tracking from that point and we part ways until the next release. What do you think? If there are no major objections in one week then I will assume I am good to go with this. Transition from current setup: If I receive some patches I was promised then I will then make a 2.14.4 bugfix/compat release make sure that master is up to date and then create something like GHC-tracking branch from master and track that. I will then abandon that branch and not push to it unless it is GHC release time. The next commit in master will bring Haddock to a state where it works with 7.8.3: yes, this means removing all new API stuff until 7.10 or 7.8.4 or whatever. GHC API changes go onto GHC-tracking while all the stuff I write goes master. When GHC makes a release or is about to, I make master work with that and make GHC-tracking point to that instead. Thanks! -- Mateusz K.

10 30

Does the 'stage=2' setting not work anymore in build.mk?
by Dr. ERDI Gergo 29 Aug '14

29 Aug '14

Hi, I tried setting 'stage=2' in my mk/build.mk file, but the stage 1 compiler is still getting rebuilt (and of course this causes the stage 2 compiler to be rebuilt from scratch...). I am using BuildFlavour=devel2. What am I missing? Thanks, Gergo

3 7