Good evening all! Currently when being ran './validate' script (without any parameters): - builds ghc using 2 parallel jobs - runs testsuite using 2 parallel jobs I propose to change the default value to amount of available CPUs: - build ghc using N+1 parallel jobs - run testsuite using N+1 parallel jobs Pros: - first-time users will get faster ./validate - seasoned users will need less tweaking for buildbots Cons: - for imbalanced boxes (32 cores, 8GB RAM) it might be quite painful to drag box out of swap What do you think about it? Actual patch: https://phabricator.haskell.org/D146 Thanks! -- Sergei
On Tue, Aug 12, 2014 at 10:31 PM, Sergei Trofimovich wrote:
Good evening all!
Currently when being ran './validate' script (without any parameters): - builds ghc using 2 parallel jobs - runs testsuite using 2 parallel jobs
I propose to change the default value to amount of available CPUs: - build ghc using N+1 parallel jobs - run testsuite using N+1 parallel jobs
Pros: - first-time users will get faster ./validate - seasoned users will need less tweaking for buildbots
Cons: - for imbalanced boxes (32 cores, 8GB RAM) it might be quite painful to drag box out of swap
What do you think about it?
Isn't the memory use also a problem on boxes with a much lower number of cores (e.g. 7.8 space leak(s))? On one machine with 1GB per core, I had to limit cabal install's parallelism when using 7.8. Assuming the patch goes in, is there a way to limit parallel jobs on the command line?
On Wed, 13 Aug 2014 11:39:56 +0200
Tuncer Ayaz
On Tue, Aug 12, 2014 at 10:31 PM, Sergei Trofimovich wrote:
Good evening all!
Currently when being ran './validate' script (without any parameters): - builds ghc using 2 parallel jobs - runs testsuite using 2 parallel jobs
I propose to change the default value to amount of available CPUs: - build ghc using N+1 parallel jobs - run testsuite using N+1 parallel jobs
Pros: - first-time users will get faster ./validate - seasoned users will need less tweaking for buildbots
Cons: - for imbalanced boxes (32 cores, 8GB RAM) it might be quite painful to drag box out of swap
What do you think about it?
Isn't the memory use also a problem on boxes with a much lower number of cores (e.g. 7.8 space leak(s))?
On one machine with 1GB per core, I had to limit cabal install's parallelism when using 7.8.
It's true in general, but I would not expect such a massive growth on ghc source. Current -Rghc-timing shows ~300MBs per ghc process on amd64. The fallout examples are HsSyn and cabal's PackageDescription modules. ghc's build system is a bit different from Cabal's: - Cabal runs one 'ghc --make' instance for a whole package. It leads to massive RAM usage in case of a multitude of modules (highlighting-kate and qthaskell come to mind). - ghc's buld system uses one 'ghc -c' execution for a single .hs file (roughly)
Assuming the patch goes in, is there a way to limit parallel jobs on the command line?
The mechanism to set limit manually is the same as before: CPUS=8 ./validate It's the default that is proposed to be changed. -- Sergei
participants (2)
-
Sergei Trofimovich -
Tuncer Ayaz