- ghc-devs - Haskell.org

Increased memory usage with GHC 7.10.1
by Jan Stolarek 02 Apr '15

02 Apr '15

Forall hi, I just uprgaded both of my machines to use GHC 7.10.1. I keep sandboxed installations of GHC and this means I had to rebuild Agda and Idris because the binaries built with GHC 7.8.4 were stored inside deactivated 7.8.4 sandbox. Sadly, I had problems building both Agda and Idris due to GHC taking up all of available memory. With Idris the problematic module was Idris.ElabTerm (~2900LOC). The interesting part of the story is that when I do a clean build of Idris GHC consumes all of memory when compiling that module and I have to kill the build. But when I restart the build after killing GHC the module is compiled using a reasonable amount of memory and within reasonable time. With Agda the problematic module is Agda.TypeChecking.Serialise (~2000LOC). The trick with killing the build and restarting it didn't work in this case. I had to compile Agda with GHC 7.8.4 (which works without problems though the mentioned module still requires a lot of memory) and alter my setup so that Agda binary is not stored inside GHC sandbox. I wonder if any of you came across similar issues with GHC 7.10.1? Do we have any performance data that allows to compare memory usage and performance of GHC 7.10.1 with previous stable releases? All of the above happened on 64bit Debian Wheezy with 2GB of RAM. Janek --- Politechnika Łódzka Lodz University of Technology Treść tej wiadomości zawiera informacje przeznaczone tylko dla adresata. Jeżeli nie jesteście Państwo jej adresatem, bądź otrzymaliście ją przez pomyłkę prosimy o powiadomienie o tym nadawcy oraz trwałe jej usunięcie. This email contains information intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient or if you have received this message in error, please notify the sender and delete it from your system.

5 7

Typeable
by Simon Peyton Jones 02 Apr '15

02 Apr '15

Iavor, Richard, and others We always thought it was the Right Thing to generate the runtime representation for a TyCon (used in Typeable type representations) at the definition site of the type, not "on the fly". I thought I'd do this. But it was a LOT harder than I expected. I have spent a ridiculous amount of time on it over the last three weeks. Sigh. However (a) it's done, (b) I did a lot of other tidying up (as usual). So I'm keen to use it. Would you like to take a look? It's on branch wip/T9858-typeable-spj. Relevant log messages below. Unsurprisingly, some perf/compiler tests get a bit slower (see below). More surprisingly, there are two perf/should_run tests that get worse perf/should_run T5205 [stat not good enough] (normal) perf/should_run lazy-bs-alloc [stat not good enough] (normal) This is much more surprising: neither seems to have anything much to do with Typeable. But I can't investigate before I go on holiday. Would anyone else like to? It's all up to date wrt HEAD. Simon commit fcd18c42b6ae2e73a9f4c67dbe6d994b0f8468c8 Author: Simon Peyton Jones <simonpj(a)microsoft.com> Date: Thu Mar 26 11:06:12 2015 +0000 Remove a redndant 'return' commit a6200759fd2f3f0e901cb2092e148bf31abb08b4 Author: Simon Peyton Jones <simonpj(a)microsoft.com> Date: Mon Mar 23 14:53:53 2015 +0000 Error message wibbles These are associated with - new instance reporting - typeable changes in -ddump-simpl output - -ddump-types being a bit less verbose - some renaming of type variables in debugger output (no idea why this happens) commit fe510d911b6cf61d6884b8b42d8771d4aea3229f Author: Simon Peyton Jones <simonpj(a)microsoft.com> Date: Mon Mar 23 14:50:23 2015 +0000 Generate Typeble info at definition sites This patch implements the idea floated in #9858, namely that we should generate type-representation information at the data type declaration site, rather than when solving a Typeable constraint. However, this turned out quite a bit harder than I expected. I still think it's the right thing to do, and it's done now, but it was quite a struggle. See particularly * Note [Grand plan for Typeable] in TcTypeable (which is a new module) * Note [The overall promotion story] in DataCon (clarifies existing stuff) The most painful bit was that to generate Typeable instances (ie TyConRepName bindings) for every TyCon is tricky for types in ghc-prim etc: - we need to have enough data types around to *define* a TyCon - many of these types are wired-in Also, to minimise the code generated for each data type, I wanted to generate pure data, not CAFs with unpackCString# stuff floating about. Performance ~~~~~~~~~~~ Three perf/compiler tests start to allocate quite a bit more. This isn't surprising, because they all allocate zillions of data types, with practically no other code, esp T1969 T1969: GHC allocates 30% more T5642: GHC allocates 14% more T9872d: GHC allocates 5% more I'm treating this as acceptable. The payoff comes in Typeable-heavy code. Remaining to do ~~~~~~~~~~~~~~~ * I think that "TyCon" and "Module" are over-generic names to use for the runtime type representations used in GHC.Typeable. Better might be "TrTyCon" and "TrModule". But I have not yet done this * Add more info the the "TyCon" e.g. source location where it was defined * Use the new "Module" type to help with Trac #10068 * It would be possible to generate TyConRepName (ie Typeable instances) selectively rather than all the time. We'd need to persist the information in interface files. Lacking a motivating reason I have not done this, but it would not be difficult. Refactoring ~~~~~~~~~~~ As is so often the case, I ended up refactoring more than I intended. In particular * In TyCon, - a type *family* (whether type or data) is repesented by a FamilyTyCon - a algebraic data type (including data/newtype instances) is represented by AlgTyCon This wasn't true before; a data family was represented as an AlgTyCon. There are some corresponding changes in IfaceSyn. Also get rid of the (unhelpfully named) tyConParent. * In TyCon define 'Promoted', isomorphic to Maybe, used when things are optionally promoted; and use it elsewhere in GHC. * Each TyCon, including promoted TyCons, contains its TyConRepName, if it has one. This is, in effect, the name of its Typeable instance. * I added PatSynId, DefMethId, and ReflectionId to the IdInfo.IdDetails type. They are used for debugging only, namely to suppress excessive output in -ddump-types. * Tidy up the generation of PrelInfo.knownKeyNames * Move newImplicitBinder from IfaceEnv to BuildTyCl * PrelNames.conName renamed to dcQual for consistency with varQual, tcQual * Move mkDefaultMethodIds, mkRecSelBinds from TcTyClsDecls to TcTyDecls commit 021e6f583a1159b0a3581ab1713d560cadc2bdf7 Author: Simon Peyton Jones <simonpj(a)microsoft.com> Date: Mon Mar 23 14:32:31 2015 +0000 tcRnDeclsi can use tcRnSrcDecls I'm not sure why tcRnDeclsi didn't call tcRnSrcDecls before, but now it does. About 20 lines of code vanish. Hooray. commit 6473d110ab1aa22a5933e405b59e3f597562ce02 Author: Simon Peyton Jones <simonpj(a)microsoft.com> Date: Fri Mar 20 12:38:42 2015 +0000 Implement lookupGlobal in TcEnv, and use it This localises the (revolting) initTcForLookup function, exposing instead the more civilised interface for lookupGlobal commit 7078a11f50e3af9139dd5ceef032e89047677833 Author: Simon Peyton Jones <simonpj(a)microsoft.com> Date: Fri Mar 20 12:36:22 2015 +0000 Comments and white space commit 48512df5751a07fd503f0ba523e5504d09ee258d Author: Simon Peyton Jones <simonpj(a)microsoft.com> Date: Fri Mar 20 12:27:59 2015 +0000 Improve the error messages for class instance errors See Note [Displaying potential instances]. simonpj@cam-05-unx:~/code/HEAD-5$

2 3

How to better parallelize GHC build.
by Karel Gardas 01 Apr '15

01 Apr '15

Folks, first of all, I remember someone already mentioned issue with decreased parallelism of the GHC build recently somewhere but I cann't find it now. Sorry, for that since otherwise I would use this thread if it was on this mailing list. Anyway, while working on SPARC NCG I'm using T2000 which provides 32 threads/8 core UltraSPARC T1 CPU. The property of this machine is that it's really slow on single-threaded work. To squeeze some perf from it man really needs to push 32 threads of work on it. Now, it really hurts my nerves to see it's lazy building/running just one or two ghc processes. To verify the fact I've created simple script to collect number of ghc processes over time and putting this to graph. The result is in the attached picture. The graph is result of running: gmake -j64 anyway, the average number of running ghc processes is 4.4 and the median value is 2. IMHO such low number not only hurts build times on something like CMT SPARC machine, but also on let say a cluster of ARM machines using NFS and also on common engineering workstations which provide these days (IMHO!) around 8-16 cores (and double the threads number). My naive idea(s) for fixing this issue is (I'm assuming no Haskell file imports unused imports here, but perhaps this may be also investigated): 1) provide explicit dependencies which guides make to build in more optimal way 2) hack GHC's make depend to kind of compute explicit dependencies from (1) in an optimal way automatically 3) someone already mentioned using shake for building ghc. I don't know shake but perhaps this is the right direction? 4) hack GHC to compile needed hi file directly in its memory if hi file is not (yet!) available (issue how to get compiling options right here). Also I don't know hi file semantics yet so bear with me on this. Is there anything else which may be done to fix that issue? Is someone already working on some of those? (I mean those reasonable from the list)? Thanks! Karel

5 7

Re: traverse_
by Herbert Valerio Riedel 31 Mar '15

31 Mar '15

On 2015-03-30 at 07:05:56 +0200, Fumiaki Kinoshita wrote: [...] > I found out that (<>) (in Data.Monoid) is missing, also. It would be nice > to reexamine Prelude to export things we want to export. Fwiw, (<>) was actually left-out as it wasn't required (it's just a an alias for `mappend`), *and* to keep our options open (or at least not make it more difficult) in terms of possible migration-plans available for the case we'd be moving 'Semigroup' to base/Prelude at some point in the future.

3 4

Re: traverse_
by Herbert Valerio Riedel 31 Mar '15

31 Mar '15

On 2015-03-31 at 06:27:32 +0200, Chris Wong wrote: [...] > While it would be useful to have it in the Prelude, traverse_ doesn't > fit either of the motivations above. So I think its omission is > intended here. That is not to say that it can't be proposed for inclusion... IOW, just turn the cautious "'traverse_' is not exported from Prelude [..] Is this intentional?"-question into an explicit/proper library proposal! Cheers, hvr

1 0

traverse_
by Kazu Yamamoto 30 Mar '15

30 Mar '15

Hi, I have just installed GHC 7.10.1. I'm a little bit surprised because 'traverse_' is not exported from Prelude. But 'traverse' is exported. Is this intentional? I expected that I can forget xxxM, xxxM_, xxxA, and xxxA_ functions with GHC 7.10.1. I want to use 'traverse_' instead of 'mapM_'. --Kazu

1 0

HP 2015.2.0.0 and GHC 7.10
by Randy Polen 30 Mar '15

30 Mar '15

I am helping Mark with the Haskell Platform, doing the Windows builds (32- and 64-bit). I want you to be aware of a problem I am encountering, and solicit suggestions and possible help. In building for HP 2015.2.0.0 on Windows 7, 64-bit (haven't gotten to 32-bit yet but likely the same problem will occur), I seem to be hitting the 32K limit for the length of arguments to a process, encountered while cabal is invoking haddock to build the docs for the OpenGLRaw package. For HP2014.2.0.0, the argument list was ~25K (from looking at my old build logs) but now is ~36K, which exceeds the maximum for CreateProcess (not a limit of the command-line, but of the OS call itself). Is there a way to build haddock docs for a single package but in multiple haddock invocations (maybe building a .haddock file for portions, then combining them, with the goal that the command line is kept short)? Seems this would also require a corresponding cabal change, as cabal is the invocator when this happens. Barring any existing mechanism, the typical solution to this problem on the Windows OS is (when possible, of course) to modify the program to accept a "response file" of command-line arguments. In this case, we could add an option to haddock to accept either a complete "response file" (i.e., allowing *all* options and arguments to come from a file) or just a file containing the files to process. Either of these changes to haddock are rather trivial to write (but adding another option implies more testing, documentation, other cases to handle, etc.). Since haddock ships with the ghc release, that's another wrinkle for this particular release. The other implication of such a solution is that cabal would need a change to utilize this change for it to be effective, checking haddock's version for support of this new haddock-flag, and either use it if the haddock version supports it, or do it optionally (which implies a new flag for cabal's haddock sub-command). This change to cabal is also rather trivial to implement (this is not to imply insensitivity to the incurred cost of each line of code, nor to the added burden of user-visible changes such as a command-line option). (Less desirable possibilities, mentioned only for completeness: skip the documentation for OpenGLRaw for this version of the Haskell Platform; split up the OpenGLRaw package itself in some way.) Other possible solutions and work-arounds? Thoughts on either using haddock in a different way (and the cabal change that would be required to break up the doc build into multiple steps for a single package)? Thoughts on the "response file" solution? Randy

3 4

Platform 2015.2.0.0 Alpha 2 for debian based systems
by Mark Lentczner 30 Mar '15

30 Mar '15

I've uploaded a build of *Haskell Platform 2015.2.0.0 Alpha 2* for generic debian linux systems. It includes GHC 7.10.2, using the generic linux bindist. This is a tarball, not a package. Installation instructions: cd / sudo tar xvf ...downloaded-tarfile... sudo /usr/local/haskell/ghc-7.10.1-x86-64/bin/activate-hs Notes: - Built on Ubuntu 14.04LTS - Needs the following packages installed: build-essential libbsd-dev libgmp-dev libtinfo-dev zlib1g-dev - The OpenGL packages need the OpenGL libs, which on a non-GUI system like a server can be gotten by installing: freeglut3-dev libgl1-mesa-dev libglu1-mesa-dev - Mark

1 1

OS X builds of GHC 7.8.4, 7.10.1-rc2, and 7.10.1-rc3
by Miëtek Bak 29 Mar '15

29 Mar '15

An OS X build of GHC 7.10.1-rc3 is now available, courtesy of Mark Lentczner. This is in addition to the GHC 7.8.4 and 7.10.1-rc2 builds Mark has made available on February 2. Thanks, Mark. GHC 7.8.4 for OS X can already be downloaded from haskell.org, even though it’s not mentioned on the GHC 7.8.4 webpage: https://downloads.haskell.org/~ghc/7.8.4/ghc-7.8.4-x86_64-apple-darwin.tar.… To avoid running up Mark’s bandwidth bill, I’m omitting the URL to his staging area. I’ve mirrored all three builds in Halcyon public storage: 7.8.4: https://halcyon.global.ssl.fastly.net/original/ghc-7.8.4-x86_64-apple-darwi… 7.10.1-rc2: https://halcyon.global.ssl.fastly.net/original/ghc-7.10.0.20150123-x86_64-a… 7.10.1-rc3: https://halcyon.global.ssl.fastly.net/original/ghc-7.10.0.20150316-x86_64-a… Binary builds of GHC, repackaged with documentation removed, are also available in Halcyon — on OS X 10.10, 10.9, 10.8, and a considerable number of Linux distributions. This includes GHC 7.10.1-rc3, 7.10.1-rc2, 7.8.4, 7.8.3, 7.8.2, 7.6.3, 7.6.1, 7.4.2, 7.2.2, and 7.0.4. Using Halcyon, installing GHC together with cabal-install is expected to take 20-30 seconds: https://halcyon.sh/tutorial/#install-ghc-and-cabal -- Miëtek https://mietek.io

1 1

Automating GHC build for Windows
by Roman Kuznetsov 29 Mar '15

29 Mar '15

Hello *, As I am still new in this, I will ask. Were there any attempt to automate GHC build process on Windows with some kind of CI engine, like Hudson, Jenkins, etc.? It seems to be rather helpful to publish build results if not after each commit, but at least once every day. WDYT? -- Sincerely yours, Roman Kuznetsov

7 16