GHC: compile using multiple cores?

newer
WX: linking to system libraries...

older
ANN: network-bytestring 0.1.2

Peter Verswyvelen

9 Apr 2009 9 Apr '09

10:16 a.m.

Is it possible to use all CPU cores when compiling with GHC and/or Cabal?

Attachments:

attachment.html (text/html — 115 bytes)

Show replies by date

Achim Schneider

9 Apr 9 Apr

10:51 a.m.

Peter Verswyvelen wrote:

...

Is it possible to use all CPU cores when compiling with GHC and/or Cabal?

Nope. Last thing I heard is that file-parallel compilation is low priority as not much would be gained anyway due to excessive cross-package stuff that's done and much stricter dependencies than say C (which you can compile in about any order you like). I guess if such a thing happens, it'd be most likely in the form of strategically placed `par`'s inside of compiler stages. -- (c) this sig last receiving data processing entity. Inspect headers for copyright history. All rights reserved. Copying, hiring, renting, performance and/or quoting of this signature prohibited.

Don Stewart

11:30 a.m.

barsoap:

...

Peter Verswyvelen wrote:

...
Is it possible to use all CPU cores when compiling with GHC and/or Cabal?

Nope. Last thing I heard is that file-parallel compilation is low priority as not much would be gained anyway due to excessive cross-package stuff that's done and much stricter dependencies than say C (which you can compile in about any order you like). I guess if such a thing happens, it'd be most likely in the form of strategically placed `par`'s inside of compiler stages.

Not with cabal, with GHC, yes: assuming you have enough modules. Use ghc -M to dump a makefile, and then make -j20 (or whatever you have) -- Don

Neil Mitchell

11:51 a.m.

...

Not with cabal, with GHC, yes: assuming you have enough modules. Use ghc -M to dump a makefile, and then make -j20 (or whatever you have)

There is a performance penalty to running ghc on separate files vs --make. If your number of core's is limited --make may be better. I'd love someone to figure out what the cross over point is :-) As a related question, how does GHC implement -j3? For my programs, if I want to run in parallel, I have to type +RTS -N3. Can I use the same trick as GHC? Thanks Neil

Andrea Vezzosi

7:08 p.m.

The main bottleneck right now is that each ghc process has to read the package.conf, which afaiu is done with Read and it's awfully slow, especially if you have many packages installed. I've started seeing total time improvements when approaching ~300% CPU usage and only the extralibs installed. On Thu, Apr 9, 2009 at 5:51 PM, Neil Mitchell wrote:

...

...
Not with cabal, with GHC, yes: assuming you have enough modules. Use ghc -M to dump a makefile, and then make -j20 (or whatever you have)

There is a performance penalty to running ghc on separate files vs --make. If your number of core's is limited --make may be better. I'd love someone to figure out what the cross over point is :-)

As a related question, how does GHC implement -j3? For my programs, if I want to run in parallel, I have to type +RTS -N3. Can I use the same trick as GHC?

Thanks

Neil _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Peter Verswyvelen

7:24 p.m.

That should be fairly easy to optimize I guess? Maybe even using read-only shared memory to share the parsed database in native binary format? On Fri, Apr 10, 2009 at 1:08 AM, Andrea Vezzosi wrote:

...

The main bottleneck right now is that each ghc process has to read the package.conf, which afaiu is done with Read and it's awfully slow, especially if you have many packages installed. I've started seeing total time improvements when approaching ~300% CPU usage and only the extralibs installed.

On Thu, Apr 9, 2009 at 5:51 PM, Neil Mitchell wrote:

...
...
Not with cabal, with GHC, yes: assuming you have enough modules. Use ghc -M to dump a makefile, and then make -j20 (or whatever you have)

There is a performance penalty to running ghc on separate files vs --make. If your number of core's is limited --make may be better. I'd love someone to figure out what the cross over point is :-)

As a related question, how does GHC implement -j3? For my programs, if I want to run in parallel, I have to type +RTS -N3. Can I use the same trick as GHC?

Thanks

Neil _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

5888

Age (days ago)

5888

Last active (days ago)

List overview

Download

5 comments

5 participants

participants (5)

Achim Schneider
Andrea Vezzosi
Don Stewart
Neil Mitchell
Peter Verswyvelen