As Neil says, I'm hoping that the new Shake-based build system will make a big difference. It's not certain that we'll switch to it, but I very much hope that we will. Fortunately we can work it side-by-side with the old system, so I hope it'll just be a question of switching over because it is manifestly better. Andrey can give a status report. Simon | -----Original Message----- | From: ghc-devs [mailto:ghc-devs-bounces@haskell.org] On Behalf Of Karel | Gardas | Sent: 07 March 2015 10:50 | To: ghc-devs | Subject: How to better parallelize GHC build. | | | Folks, | | first of all, I remember someone already mentioned issue with decreased | parallelism of the GHC build recently somewhere but I cann't find it | now. Sorry, for that since otherwise I would use this thread if it was | on this mailing list. | | Anyway, while working on SPARC NCG I'm using T2000 which provides 32 | threads/8 core UltraSPARC T1 CPU. The property of this machine is that | it's really slow on single-threaded work. To squeeze some perf from it | man really needs to push 32 threads of work on it. Now, it really hurts | my nerves to see it's lazy building/running just one or two ghc | processes. To verify the fact I've created simple script to collect | number of ghc processes over time and putting this to graph. The result | is in the attached picture. The graph is result of running: | | gmake -j64 | | anyway, the average number of running ghc processes is 4.4 and the | median value is 2. IMHO such low number not only hurts build times on | something like CMT SPARC machine, but also on let say a cluster of ARM | machines using NFS and also on common engineering workstations which | provide these days (IMHO!) around 8-16 cores (and double the threads | number). | | My naive idea(s) for fixing this issue is (I'm assuming no Haskell file | imports unused imports here, but perhaps this may be also investigated): | | 1) provide explicit dependencies which guides make to build in more | optimal way | | 2) hack GHC's make depend to kind of compute explicit dependencies from | (1) in an optimal way automatically | | 3) someone already mentioned using shake for building ghc. I don't know | shake but perhaps this is the right direction? | | 4) hack GHC to compile needed hi file directly in its memory if hi file | is not (yet!) available (issue how to get compiling options right here). | Also I don't know hi file semantics yet so bear with me on this. | | | Is there anything else which may be done to fix that issue? Is someone | already working on some of those? (I mean those reasonable from the | list)? | | Thanks! | Karel