#910: --make should have a -j flag for parallel building -------------------------------------+------------------------------------ Reporter: igloo | Owner: Type: feature request | Status: patch Priority: normal | Milestone: _|_ Component: Compiler | Version: 6.4.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: None/Unknown | Difficulty: Unknown Test Case: N/A | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Changes (by parcs): * status: new => patch Comment: I've been working on this for a while, including this year's GSoC. Since the GHC 7.8 feature freeze is imminent, I would like to post a stable subset of my current progress for review and possible inclusion to GHC 7.8. This subset of changes includes the bare minimum required to build multiple modules in parallel, and the parallel upsweep itself. Each patch is for the most part self-explanatory, I hope. https://github.com/parcs/ghc/commits/ghc-parmake-gsoc The speedups provided by the parallel upsweep are decent: I can realize a 1.8x speedup when compiling the Cabal library with `-j3 -O2`, and a 2.4x speedup when compiling 7 independent, relatively large modules with `-j3 -O0`, for example. The performance/thread ratio seems to peak at -j3 (which instructs the runtime to use 3 capabilities). The performance of the sequential upsweep is not significantly impacted by these patches: compiling the Cabal library with `-O2` takes about 1% longer with the patches than without. I have not tested these patches on any platform other than x86_64/Linux, but I have no reason to believe that behavior would differ on other platforms. Nonetheless, testing on other platforms is necessary and appreciated. These changes are well-tested and stable, but there is a single bug that escapes me which can be triggered by the testsuite. When running the testsuite with e.g. `EXTRA_HC_OPTS=-j2`, there is about a 1/1000 chance that a compiler process will exit with {{{ ghc-stage2: GHC.Event.Manager.loop: state is already Finished }}} On a separate machine, I don't ever trigger the aforementioned bug, but instead the compiler process on rare occasion never exits. The testsuite script eventually kills the process, causing the particular test to fail with 'exit code non-0'. These are likely bugs in the IO manager triggered by the changing of the number of capabilities at runtime. If I instead explicitly set the number of capabilities at startup with `EXTRA_HC_OPTS=-j2 +RTS -N2 -RTS` then neither bug manifests. I don't yet understand the IO manager well enough to fix this issue though. Other than that though, this feature Just Works. One simply has to pass the `-jN` flag to GHC and the build will just finish faster. Questions or comments? I have likely not explained everything I ought to explain. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/910#comment:22 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler