- Glasgow-haskell-users

Re: memory fragmentation with ghc-7.6.1
by Ben Gamari 01 Oct '12

01 Oct '12

Simon Marlow <marlowsd(a)gmail.com> writes: > On 28/09/12 17:36, Ben Gamari wrote: >> Unfortunately, after poking around I found a few obvious problems with >> both the code and my testing configuration which explained the >> performance drop. Things seem to be back to normal now. Sorry for the >> noise! Great job on the new codegen. > > That's good to hear, thanks for letting me know! > Of course! That being said, I have run in to a bit of a performance issue which could be related to the runtime system. In particular, as I scale up in thread count (from 6 up to 48, the core count of the machine) in my program[1] (test data available), I'm seeing the total runtime increase, as well as a corresponding increase in CPU-seconds used. This despite the RTS claiming consistently high (~94%) productivity. Meanwhile Threadscope shows that nearly all of my threads are working busily with very few STM retries and no idle time. This in an application which should scale reasonably well (or so I believe). Attached below you will find a crude listing of various runtime statistics over a variety of thread counts (ranging into what should probably be regarded as the absurdly large). The application is a parallel Gibbs sampler for learning probabilistic graphical models. It involves a set of worker threads (updateWorkers) pulling work units off of a common TQueue. After grabbing a work unit, the thread will read a reference to the current global state from an IORef. It will then begin a long-running calculation, resulting in a small value (forced to normal form with deepseq) which it then communicates back to a global update thread (diffWorker) in the form of a lambda through another TQueue. The global update thread then maps the global state (the same as was read from the IORef earlier) through this lambda with atomicModifyIORef'. This is all implemented in [2]. I do understand that I'm asking a lot of the language and I have been quite impressed by how well Haskell and GHC have stood up to the challenge thusfar. That being said, the behavior I'm seeing seems a bit strange. If synchronization overhead were the culprit, I'd expect to observe STM retries or thread blocking, which I do not see (by eye it seems that STM retries occur on the order of 5/second and worker threads otherwise appear to run uninterrupted except for GC; GHC event log from a 16 thread run available here[3]). If GC were the problem, I would expect this to be manifested in the productivity, which it is clearly not. Do you have any idea what else might be causing such extreme performance degradation with higher thread counts? I would appreciate any input you would have to offer. Thanks for all of your work! Cheers, - Ben [1] https://github.com/bgamari/bayes-stack/v2 [2] https://github.com/bgamari/bayes-stack/blob/v2/BayesStack/Core/Gibbs.hs [3] http://goldnerlab.physics.umass.edu/~bgamari/RunCI.eventlog Performance of Citation Influence model on lda-handcraft data set 1115 arcs, 702 nodes, 50 items per node average 100 sweeps in blocks of 10, 200 topics Running with +RTS -A1G ghc-7.7 9c15249e082642f9c4c0113133afd78f07f1ade2 Cores User time (s) Walltime (s) CPU % Productivity ====== ============= ============= ====== ============= 2 488.66 269.41 188% 93.7% 3 533.43 195.28 281% 94.1% 4 603.92 166.94 374% 94.3% 5 622.40 138.16 466% 93.8% 6 663.73 123.00 558% 94.2% 7 713.96 114.17 647% 94.0% 8 724.66 101.98 736% 93.7% 9 802.75 100.59 826% . 10 865.05 97.69 917% . 11 966.97 99.09 1010% . 12 1238.42 114.28 1117% 13 1242.43 106.53 1206% 14 1428.59 112.48 1310% 15 1299.40 97.52 1387% 16 1559.99 108.86 1481% 17 1972.02 126.49 1604% 18 2157.03 130.91 1696% 19 1966.24 115.29 1770% . 20 2693.64 146.76 1887% . 21 3051.16 158.48 1990% . 22 4100.88 199.18 2109% 93.7% 23 4156.94 193.38 2201% 93.5% 24 4780.46 212.13 2303% 94.3% 25 5733.64 242.78 2407% . 26 7806.47 313.92 2526% . 27 6368.32 249.65 2596% . 28 8563.18 320.26 2717% -sstderr output from 2 cores: ====================================== 214,358,463,960 bytes allocated in the heap 2,261,367,592 bytes copied during GC 71,053,384 bytes maximum residency (7 sample(s)) 2,077,816 bytes maximum slop 2299 MB total memory in use (0 MB lost due to fragmentation) Tot time (elapsed) Avg pause Max pause Gen 0 100 colls, 100 par 28.58s 14.27s 0.1427s 0.3345s Gen 1 7 colls, 6 par 3.36s 1.74s 0.2486s 0.3241s Parallel GC work balance: 25.06% (serial 0%, perfect 100%) TASKS: 4 (1 bound, 3 peak workers (3 total), using -N2) SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled) INIT time 0.03s ( 0.03s elapsed) MUT time 474.28s (252.98s elapsed) GC time 31.94s ( 16.01s elapsed) EXIT time 0.12s ( 0.12s elapsed) Total time 506.49s (269.26s elapsed) Alloc rate 451,969,681 bytes per MUT second Productivity 93.7% of total user, 176.2% of total elapsed gc_alloc_block_sync: 108391 whitehole_spin: 0 gen[0].sync: 7392 gen[1].sync: 97126 -sstderr output from 24 cores: ======================================= 262,335,150,000 bytes allocated in the heap 1,323,818,912 bytes copied during GC 67,648,048 bytes maximum residency (7 sample(s)) 2,927,176 bytes maximum slop 25172 MB total memory in use (0 MB lost due to fragmentation) Tot time (elapsed) Avg pause Max pause Gen 0 29 colls, 29 par 162.40s 6.77s 0.2335s 0.3469s Gen 1 7 colls, 6 par 118.34s 5.32s 0.7600s 1.2448s Parallel GC work balance: 5.61% (serial 0%, perfect 100%) TASKS: 27 (1 bound, 26 peak workers (26 total), using -N24) SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled) INIT time 0.34s ( 0.34s elapsed) MUT time 4606.75s (197.59s elapsed) GC time 280.74s ( 12.09s elapsed) EXIT time 0.40s ( 0.40s elapsed) Total time 4888.61s (210.80s elapsed) Alloc rate 56,945,837 bytes per MUT second Productivity 94.3% of total user, 2185.8% of total elapsed gc_alloc_block_sync: 5397994 whitehole_spin: 0 gen[0].sync: 32282 gen[1].sync: 5072

3 2

ANNOUNCE: GHC version 7.6.1
by Ian Lynagh 30 Sep '12

30 Sep '12

============================================================= The (Interactive) Glasgow Haskell Compiler -- version 7.6.1 ============================================================= The GHC Team is pleased to announce a new major release of GHC, 7.6.1. Here are some of the highlights of the 7.6 branch since 7.4: * Polymorphic kinds and data promotion are now fully implemented and supported features. * Windows 64bit is now a supported platform. * It is now possible to defer type errors until runtime using the -fdefer-type-errors flag. * The RTS now supports changing the number of capabilities at runtime with Control.Concurrent.setNumCapabilities. Full release notes are here: http://www.haskell.org/ghc/docs/7.6.1/html/users_guide/release-7-6-1.html How to get it ~~~~~~~~~~~~~ The easy way is to go to the web page, which should be self-explanatory: http://www.haskell.org/ghc/ We supply binary builds in the native package format for many platforms, and the source distribution is available from the same place. Packages will appear as they are built - if the package for your system isn't available yet, please try again later. Background ~~~~~~~~~~ Haskell is a standard lazy functional programming language. GHC is a state-of-the-art programming suite for Haskell. Included is an optimising compiler generating good code for a variety of platforms, together with an interactive system for convenient, quick development. The distribution includes space and time profiling facilities, a large collection of libraries, and support for various language extensions, including concurrency, exceptions, and foreign language interfaces (C, whatever). GHC is distributed under a BSD-style open source license. A wide variety of Haskell related resources (tutorials, libraries, specifications, documentation, compilers, interpreters, references, contact information, links to research groups) are available from the Haskell home page (see below). On-line GHC-related resources ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Relevant URLs on the World-Wide Web: GHC home page http://www.haskell.org/ghc/ GHC developers' home page http://hackage.haskell.org/trac/ghc/ Haskell home page http://www.haskell.org/ Supported Platforms ~~~~~~~~~~~~~~~~~~~ The list of platforms we support, and the people responsible for them, is here: http://hackage.haskell.org/trac/ghc/wiki/Contributors Ports to other platforms are possible with varying degrees of difficulty. The Building Guide describes how to go about porting to a new platform: http://hackage.haskell.org/trac/ghc/wiki/Building Developers ~~~~~~~~~~ We welcome new contributors. Instructions on accessing our source code repository, and getting started with hacking on GHC, are available from the GHC's developer's site run by Trac: http://hackage.haskell.org/trac/ghc/ Mailing lists ~~~~~~~~~~~~~ We run mailing lists for GHC users and bug reports; to subscribe, use the web interfaces at http://www.haskell.org/mailman/listinfo/glasgow-haskell-users http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs There are several other haskell and ghc-related mailing lists on www.haskell.org; for the full list, see http://www.haskell.org/mailman/listinfo/ Some GHC developers hang out on #haskell on IRC, too: http://www.haskell.org/haskellwiki/IRC_channel Please report bugs using our bug tracking system. Instructions on reporting bugs can be found here: http://www.haskell.org/ghc/reportabug

12 18

Simplification of instances
by David Waern 28 Sep '12

28 Sep '12

Hi, was there a change in how GHC simplifies instances sometime between 7.4.1 and 7.6.1? Haddock used to produce this: http://hackage.haskell.org/packages/archive/kan-extensions/2.7/doc/html/Dat… but is now producing this: http://hackage.haskell.org/packages/archive/kan-extensions/3.1/doc/html/Dat… Haddock uses tcRnGetInfo to get the instances. Thanks, David

2 2

memory fragmentation with ghc-7.6.1
by John Lato 26 Sep '12

26 Sep '12

Hello, We've noticed that some applications exhibit significantly worse memory usage when compiled with ghc-7.6.1 compared to ghc-7.4, leading to out of memory errors in some cases. Running one app with +RTS -s, I see this: ghc-7.4 525,451,699,736 bytes allocated in the heap 53,404,833,048 bytes copied during GC 39,097,600 bytes maximum residency (2439 sample(s)) 1,547,040 bytes maximum slop 628 MB total memory in use (0 MB lost due to fragmentation) ghc-7.6 512,535,907,752 bytes allocated in the heap 53,327,184,712 bytes copied during GC 40,038,584 bytes maximum residency (2391 sample(s)) 1,456,472 bytes maximum slop 3414 MB total memory in use (2744 MB lost due to fragmentation) The total memory in use (consistent with 'top's output) is much higher when built with ghc-7.6, due entirely to fragmentation. I've filed a bug report (http://hackage.haskell.org/trac/ghc/ticket/7257, http://hpaste.org/74987) but I was wondering if anyone else has noticed this? I'm not entirely sure what's triggering this behavior (some applications work fine), although I suspect it has to do with allocation of pinned memory. John L.

4 5

Overloading Lists
by Achim Krause 25 Sep '12

25 Sep '12

As was mentioned in Haskell Cafe some days ago, there is some work going on on overloading lists by George Giorgidze, Jeroen Weijers and me. We are implementing two approaches in fact. The first one is using a typeclass called FromList, defined as class FromList l where type (Elem l) fromList :: [Elem l] -> l And explicit lists and arithmetic sequences get just desugared in the obvious way using this, so, for example, [1,2,3] gets desugared to (fromList [1,2,3]). The disadvantages of this (although being very easy to implement and use) is that this goes through recursive lists each time it gets used. Since explicit lists need not to be static (i.e. can contain variables), this can really matter. The second approach thus is the following (kinda motivated from the way parallel arrays are desugared): There is a class class Singleton l where type (Elem l) singleton :: Elem l -> l and now explicit lists like [1,2,3] get desugared to (singleton 1) `mappend` (singleton 2) `mappend` (singleton 3). Arithmetic sequences are simply done by providing a class "GenericEnum" containing slightly more general versions of the classical from, fromThen, fromTo, fromThenTo, and the desugaring then really is exactly the same as in the non-overloaded case. I wrote a patch for the GHC which is supposed to do this stuff. It compiles, and the typechecking and desugaring of arithmetic sequences works just fine. However, the explicit lists give me a runtime error, when used on a simple test file containing explicit list notation: Can't find interface-file declaration for variable base:Data.Monoid.mempty Probable cause: bug in .hi-boot file, or inconsistent .hi file Use -ddump-if-trace to get an idea of which file caused the error In the expression: [] In an equation for `testlist': testlist = [] At first I thought that I forgot something about adding a new module to the PrelNames module, so I did the following: I created a class Monoid2 (with functions mempty2 and mappend2) in the module GHC.Exts, because I know this module to work with wired-ins (For example, the fromList from the previous approach, the genericEnum and the singleton were also put there). The new error message became: Can't find interface-file declaration for variable base:GHC.Exts.mempty2 Probable cause: bug in .hi-boot file, or inconsistent .hi file Use -ddump-if-trace to get an idea of which file caused the error In the expression: [] In an equation for `testlist': testlist = [] I double-checked the stuff I did in PrelNames.hs. Since I was running short of Uniques (the code says that Template Hakell uses IdUniques 200-499), I started to use IdUniques from 500 on, but not for memptyName and mappendName, so I'm pretty sure that PrelNames should work the way I did it. A quick search for "Can't find interface-file" shows that the error originates from TcInterface. I assume now, that the error comes from how I typecheck ExplicitLists: tcExpr (ExplicitList _ witness exprs) res_ty = case witness of Nothing -> do { (coi, elt_ty) <- matchExpectedListTy res_ty ; exprs' <- mapM (tc_elt elt_ty) exprs ; return $ mkHsWrapCo coi (ExplicitList elt_ty Nothing exprs') } where tc_elt elt_ty expr = tcPolyExpr expr elt_ty Just (singleton, mempty, mappend) -> do { elt_ty <- newFlexiTyVarTy liftedTypeKind ; mempty' <- tcSyntaxOp OverloadedListOrigin mempty res_ty ; mappend' <- tcSyntaxOp OverloadedListOrigin mappend (mkFunTys [res_ty, res_ty] res_ty) ; singleton' <- tcSyntaxOp OverloadedListOrigin singleton (mkFunTy elt_ty res_ty) ; exprs' <- mapM (tc_elt elt_ty) exprs ; return $ ExplicitList elt_ty (Just (singleton',mempty',mappend')) exprs'} where tc_elt elt_ty expr = tcPolyExpr expr elt_ty A few remarks about that: I changed the ExplicitList constructor by adding a witness field of the type Maybe (SyntaxExpr, SyntaxExpr, SyntaxExpr). The renamer leaves this at Nothing, if the overloaded lists extension is turned off, otherwise it changes it to Just (singleton, mempty, mappend) where these are variables containing the respective names. The Nothing case in the above code is just whats done for regular lists in the original code. The Just case is supposed to do the following: If an overloaded list has type a, then the respective mempty should also have type a, and the respective mappend should have type a -> a -> a. If there are no instances with these types, then we should get an error, because overloaded lists should always be monoids, according to our desugaring rules. There should now be some element type, appearing in the type signature of singleton. Therefore we introduce a type variable elt_ty and match singleton against the function elt_ty -> res_ty, this should give the instance we need and infer the element type. Then we match all elements of the list against this type just as is done in the original code. My observations: If I interchange the mempty line with the mappend line, we get a similar error, but about mappend. If I comment both of these lines out, the error vanishes. (The last point is not that easy to check, because we need to return something, but if you replace the Just in the return thing by Nothing, everything works.) So the error has to do something with these two lines (which is not that surprising, because this is the only point where mempty should become instantiated). I also tried to interchange these lines, this did not do anything (except when the mappend line comes before the mempty line the error talks about mappend2 instead of mempty2). I do not have any ideas what to try and what could be wrong. I DID a complete clean build, so it's really about the code. The code as it is now is on github, my account is "achimkrause". If anyone wants to test it, please do not forget to patch the base and the dph repository also: base contains my changes to GHC.Exts, and in dph I had to make some imports of GHC.Exts qualified, because there are also singleton functions. Please feel free to ask questions. I would be very happy if you could help us figure this out, we are fairly sure that this is some kind of common error. Cheers, Achim

1 0

ghc api and .o files
by Evan Laforge 25 Sep '12

25 Sep '12

I'm using the GHC api to run a little REPL for my app. This involves linking in some compiled C, and as long as it's in a library it's fine, I just pass -lwhatever to parseDynamicFlags and it's happy. However, I also have a plain .o file, and from poking at the ghci source, it looks like that works by consing onto v_Ld, which is not exported from StaticFlags. I was just about to go verify this, but it looks like it just changed! So maybe .o files can now be passed to parseStaticFlags? Or even better, parseDynamicFlags? However, it seems to have happened very recently, and 7.6.1 is still using the v_Ld global. Does this mean that there's no way to link in an arbitrary .o? I suppose I can work around it by packing the .o into a libx.a and then passing -lx. Relatedly, I've noticed that OS X is forgiving when you don't link in a needed object. It will let me run code, but if I call a function that's not linked in I get a crash. However, linux immediately prints "unknown symbol `etc.'". The OS X behaviour is more convenient because it's easy to avoid calling the missing functions, and difficult to figure out how to cut all the dependencies so they're not needed, but the linux way is certainly safer. Does anyone know why this difference exists? Just curious. As an aside, I recently upgraded to 7.6.1 from 7.0.3 and the API changed in a numer of ways. Is there any documentation for those changes? I'm guessing not, since there's not much documentation for the API itself, except of course the code. I noticed that GHC.getWarnings disappeared, and seems to have been replaced with either printing directly to stderr or throwing an exception, e.g. HscTypes.handleFlagWarnings. Other than the expected hassles from depending on an internal API, 7.6.1 has been great so far, except of course hackage had to go down as soon as it was time to cabal install everything. thanks!

3 5

How do I build GHC 7.6 from source?
by Iavor Diatchki 21 Sep '12

21 Sep '12

Hello, I was just trying to build the GHC-7.6 branch from source and the build failed with type-errors, because the libraries used by GHC have moved on since the release, and "sync all" just gets the most recent version. Is there a "fingerprint" somewhere so that I can checkout the libraries in a suitable form for building 7.6? More generally, I run into this problem all the time with people trying to build the branch I work on, so it really would be nice if we started using sub-modules to keep track of this dependency. I talked to folks at ICFP about this, and one idea was that we could check in the "fingerprint" for a branch into the repo, and track the dependencies this way, but this is exactly what git's submodule machinery does, so it seems pointless to implement the functionality which is already there with a standard interface. Thoughts? -Iavor

4 6

Why is Bag's Data instance "broken"?
by Philip Holzenspies 21 Sep '12

21 Sep '12

Dear GHCers, I'm performing traversals over GHC-API results (HsSyn et al). For this purpose, I'm using SYB generics. I found that I couldn't use "ext1Q" for a function with type "Data x => Bag x -> String", i.e. that this function was never applied. The source of Bag's instance of the Data class seems to explain why: instance Data a => Data (Bag a) where gfoldl k z b = z listToBag `k` bagToList b -- traverse abstract type abstractly toConstr _ = abstractConstr $ "Bag("++show (typeOf (undefined::a))++")" gunfold _ _ = error "gunfold" dataTypeOf _ = mkNoRepType "Bag" Is there a rationale to not allow gunfolds and to keep toConstr abstract? More to the point for my needs, is there a reason to not allow dataCast1 casting of Bags? Regards, Philip

3 8

Type operators in GHC
by Simon Peyton-Jones 19 Sep '12

19 Sep '12

Dear GHC users As part of beefing up the kind system, we plan to implement the "Type operators" proposal for Haskell Prime http://hackage.haskell.org/trac/haskell-prime/wiki/InfixTypeConstructors GHC has had type operators for some kind, so you can say data a :+: b = Left a | Right b but you can only do that for operators which start with ":". As part of the above wiki page you can see the proposal to broaden this to ALL operators, allowing data a + b = Left a | Right b Although this technically inconsistent the value page (as the wiki page discussed), I think the payoff is huge. (And "A foolish consistency is the hobgoblin of little minds", Emerson) This email is (a) to highlight the plan, and (b) to ask about flags. Our preferred approach is to *change* what -XTypeOperators does, to allow type operators that do not start with :. But that will mean that *some* (strange) programs will stop working. The only example I have seen in tc192 of GHC's test suite {-# LANGUAGE TypeOperators #-} comp :: Arrow (~>) => (b~>c, c~>d)~>(b~>d) comp = arr (uncurry (>>>)) Written more conventionally, the signature would look like comp :: Arrow arr => arr (arr b c, arr c d) (arr b d) comp = arr (uncurry (>>>)) or, in infix notation {-# LANGUAGE TypeOperators #-} comp :: Arrow arr => (b `arr` c, c `arr` d) `arr` (b `arr` d) comp = arr (uncurry (>>>)) But tc192 as it stands would become ILLEGAL, because (~>) would be a type *constructor* rather than (as now) a type *variable*. Of course it's easily fixed, as above, but still a breakage is a breakage. It would be possible to have two flags, so as to get - Haskell 98 behaviour - Current TypeOperator behaviuor - New TypeOperator behaviour but it turns out to be Quite Tiresome to do so, and I would much rather not. Can you live with that? http://chrisdone.com/posts/2010-10-07-haskelldb-and-typeoperator-madness.ht…

13 24

Kind Demotion
by Ashley Yakeley 18 Sep '12

18 Sep '12

Now that we have type promotion, where certain types can become kinds, I find myself wanting kind demotion, where kinds are also types. So for instance there would be a '*' type, and all types of kind * would be demoted to values of it. Is that feasible? -- Ashley Yakeley

2 4