Cabal: Can only one processor core be used? - Haskell-Cafe - Haskell.org

newer
Hackage: "Build: PlanningFailed"

Cabal: Can only one processor core be used?

older
Cabal install fails, don't...

Volker Wysk

2 Apr 2024 2 Apr '24

7:22 a.m.

Hi I've searched the web and the cabal user guide on how to use more processor cores for compilation via cabal. I couldn't find anything. It looks like only one processor core can be used with cabal, even though (I've read on the web) GHC supports parallel compilation. Cabal compilation times can be a pain. It can't be that you can use only one processor core with cabal, can it ..? Cheers, Volker

Reply

Sign in to reply online Use email software

Show replies by date

Alexey Vagarenko

2 Apr 2 Apr

7:37 a.m.

Hi, There is -j flag for this: https://cabal.readthedocs.io/en/3.4/cabal-project.html#cfg-field-jobs On Tue, Apr 2, 2024 at 4:23 PM Volker Wysk wrote:

Hi

I've searched the web and the cabal user guide on how to use more processor cores for compilation via cabal. I couldn't find anything.

It looks like only one processor core can be used with cabal, even though (I've read on the web) GHC supports parallel compilation.

Cabal compilation times can be a pain. It can't be that you can use only one processor core with cabal, can it ..?

Cheers, Volker _______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

Reply

Sign in to reply online Use email software

coot＠coot.me

7:47 a.m.

Let's also mention that there's work to have a single semaphore which governs the concurrency in a better way which allows the tooling to dynamically decide how many cores to use for compiling a package when multiple of them are being compiled. Regards, Marcin Sent with Proton Mail secure email. On Tuesday, 2 April 2024 at 13:37, Alexey Vagarenko wrote:

Hi,

There is -j flag for this: https://cabal.readthedocs.io/en/3.4/cabal-project.html#cfg-field-jobs

On Tue, Apr 2, 2024 at 4:23 PM Volker Wysk wrote:

...
Hi

I've searched the web and the cabal user guide on how to use more processor cores for compilation via cabal. I couldn't find anything.

It looks like only one processor core can be used with cabal, even though (I've read on the web) GHC supports parallel compilation.

Cabal compilation times can be a pain. It can't be that you can use only one processor core with cabal, can it ..?

Cheers, Volker _______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

Reply

Sign in to reply online Use email software

Volker Wysk

7:49 a.m.

Am Dienstag, dem 02.04.2024 um 16:37 +0500 schrieb Alexey Vagarenko:

Hi,

There is -j flag for this: https://cabal.readthedocs.io/en/3.4/cabal-project.html#cfg-field-jobs

Okay, great. But only whole packages will be compiled in parallel, as Henning has pointed out. Cheers, Volker

On Tue, Apr 2, 2024 at 4:23 PM Volker Wysk wrote:

...
Hi

I've searched the web and the cabal user guide on how to use more processor cores for compilation via cabal. I couldn't find anything.

It looks like only one processor core can be used with cabal, even though (I've read on the web) GHC supports parallel compilation.

Cabal compilation times can be a pain. It can't be that you can use only one processor core with cabal, can it ..?

Cheers, Volker _______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

Reply

Sign in to reply online Use email software

Bryan Richter

8:06 a.m.

BTW the "semaphore" solution mentioned was introduced in this proposal: https://github.com/ghc-proposals/ghc-proposals/pull/540 Looks like it may be present in the latest Cabal (3.10) and latest GHC (9.8)? I'm curious to know if anyone's using it yet. :) On Tue, 2 Apr 2024 at 14:50, Volker Wysk wrote:

Am Dienstag, dem 02.04.2024 um 16:37 +0500 schrieb Alexey Vagarenko:

...
Hi,

There is -j flag for this: https://cabal.readthedocs.io/en/3.4/cabal-project.html#cfg-field-jobs

Okay, great. But only whole packages will be compiled in parallel, as Henning has pointed out.

Cheers, Volker

...
On Tue, Apr 2, 2024 at 4:23 PM Volker Wysk wrote:

...
Hi

I've searched the web and the cabal user guide on how to use more processor cores for compilation via cabal. I couldn't find anything.

It looks like only one processor core can be used with cabal, even though (I've read on the web) GHC supports parallel compilation.

Cabal compilation times can be a pain. It can't be that you can use only one processor core with cabal, can it ..?

Cheers, Volker _______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

_______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

Reply

Sign in to reply online Use email software

Brandon Allbery

11:18 a.m.

It's only in HEAD at present and will be in 3.12. On Tue, Apr 2, 2024 at 8:07 AM Bryan Richter via Haskell-Cafe < haskell-cafe@haskell.org> wrote:

BTW the "semaphore" solution mentioned was introduced in this proposal: https://github.com/ghc-proposals/ghc-proposals/pull/540

Looks like it may be present in the latest Cabal (3.10) and latest GHC (9.8)? I'm curious to know if anyone's using it yet. :)

On Tue, 2 Apr 2024 at 14:50, Volker Wysk wrote:

...
Am Dienstag, dem 02.04.2024 um 16:37 +0500 schrieb Alexey Vagarenko:

...
Hi,

There is -j flag for this: https://cabal.readthedocs.io/en/3.4/cabal-project.html#cfg-field-jobs

Okay, great. But only whole packages will be compiled in parallel, as Henning has pointed out.

Cheers, Volker

...
On Tue, Apr 2, 2024 at 4:23 PM Volker Wysk

wrote:

...
...
...
Hi

I've searched the web and the cabal user guide on how to use more processor cores for compilation via cabal. I couldn't find anything.

It looks like only one processor core can be used with cabal, even though (I've read on the web) GHC supports parallel compilation.

Cabal compilation times can be a pain. It can't be that you can use only one processor core with cabal, can it ..?

Cheers, Volker _______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

_______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

-- brandon s allbery kf8nh allbery.b@gmail.com

Reply

Sign in to reply online Use email software

Andrew Lelechenko

5:32 p.m.

I’ve been using semaphore concurrency builds since GHC 9.8 was released. Works like a charm, very nice, but yes, one has to build cabal-install HEAD or wait until cabal-3.12 is out (which is soon AFAIU). Best regards, Andrew

On 2 Apr 2024, at 16:18, Brandon Allbery wrote:

It's only in HEAD at present and will be in 3.12.

On Tue, Apr 2, 2024 at 8:07 AM Bryan Richter via Haskell-Cafe wrote: BTW the "semaphore" solution mentioned was introduced in this proposal: https://github.com/ghc-proposals/ghc-proposals/pull/540

Looks like it may be present in the latest Cabal (3.10) and latest GHC (9.8)? I'm curious to know if anyone's using it yet. :)

On Tue, 2 Apr 2024 at 14:50, Volker Wysk wrote:

...
Am Dienstag, dem 02.04.2024 um 16:37 +0500 schrieb Alexey Vagarenko:

...
Hi,

There is -j flag for this: https://cabal.readthedocs.io/en/3.4/cabal-project.html#cfg-field-jobs

Okay, great. But only whole packages will be compiled in parallel, as Henning has pointed out.

Cheers, Volker

...
On Tue, Apr 2, 2024 at 4:23 PM Volker Wysk wrote:

...
Hi

I've searched the web and the cabal user guide on how to use more processor cores for compilation via cabal. I couldn't find anything.

It looks like only one processor core can be used with cabal, even though (I've read on the web) GHC supports parallel compilation.

Cabal compilation times can be a pain. It can't be that you can use only one processor core with cabal, can it ..?

Cheers, Volker _______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

_______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

_______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

-- brandon s allbery kf8nh allbery.b@gmail.com _______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

Reply

Sign in to reply online Use email software

Brandon Allbery

5:44 p.m.

You don't have to build it yourself, there's a `cabal-head` release on github where you can download it or use: ghcup install cabal --force -u https://github.com/haskell/cabal/releases/download/cabal-head/cabal-head-Lin... head On Tue, Apr 2, 2024 at 5:33 PM Andrew Lelechenko < andrew.lelechenko@gmail.com> wrote:

I’ve been using semaphore concurrency builds since GHC 9.8 was released. Works like a charm, very nice, but yes, one has to build cabal-install HEAD or wait until cabal-3.12 is out (which is soon AFAIU).

Best regards, Andrew

...
On 2 Apr 2024, at 16:18, Brandon Allbery wrote:

It's only in HEAD at present and will be in 3.12.

On Tue, Apr 2, 2024 at 8:07 AM Bryan Richter via Haskell-Cafe < haskell-cafe@haskell.org> wrote: BTW the "semaphore" solution mentioned was introduced in this proposal: https://github.com/ghc-proposals/ghc-proposals/pull/540

Looks like it may be present in the latest Cabal (3.10) and latest GHC (9.8)? I'm curious to know if anyone's using it yet. :)

On Tue, 2 Apr 2024 at 14:50, Volker Wysk wrote:

...
Am Dienstag, dem 02.04.2024 um 16:37 +0500 schrieb Alexey Vagarenko:

...
Hi,

There is -j flag for this:

https://cabal.readthedocs.io/en/3.4/cabal-project.html#cfg-field-jobs

...
Okay, great. But only whole packages will be compiled in parallel, as Henning has pointed out.

Cheers, Volker

...
On Tue, Apr 2, 2024 at 4:23 PM Volker Wysk

wrote:

...
...
...
Hi

I've searched the web and the cabal user guide on how to use more processor cores for compilation via cabal. I couldn't find anything.

It looks like only one processor core can be used with cabal, even though (I've read on the web) GHC supports parallel compilation.

Cabal compilation times can be a pain. It can't be that you can use only one processor core with cabal, can it ..?

Cheers, Volker _______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

_______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

-- brandon s allbery kf8nh allbery.b@gmail.com _______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

_______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

-- brandon s allbery kf8nh allbery.b@gmail.com

Reply

Sign in to reply online Use email software

Henning Thielemann

7:38 a.m.

On Tue, 2 Apr 2024, Volker Wysk wrote:

I've searched the web and the cabal user guide on how to use more processor cores for compilation via cabal. I couldn't find anything.

$ cabal install -j3 or $ fgrep jobs ~/.cabal/config jobs: 3

Cabal compilation times can be a pain. It can't be that you can use only one processor core with cabal, can it ..?

There are two places where you can increase the use of multiple cores. The one above controls the number of packages to be compiled in parallel, whereas $ cabal install --ghc-option=-j3 means that each instance of GHC compiles up to three modules in parallel. As far as I know it is currently not possible to interleave compilation of modules and packages. I.e. optimally we would only need one 'jobs' option and then Cabal and GHC manage to fill the working queue with any module from any package. This would certainly require to run GHC in a server mode. However, without a fast storage for temporary files you will hardly see an actual speedup. I use to mount a ram disk with tmpfs at /ram and then compile with $ cabal install -j3 --ghc-option=-j3 --builddir=/ram/cabal

Reply

Sign in to reply online Use email software

Volker Wysk

7:55 a.m.

Am Dienstag, dem 02.04.2024 um 13:38 +0200 schrieb Henning Thielemann:

On Tue, 2 Apr 2024, Volker Wysk wrote:

...
I've searched the web and the cabal user guide on how to use more processor cores for compilation via cabal. I couldn't find anything.

$ cabal install -j3

or

$ fgrep jobs ~/.cabal/config jobs: 3

...
Cabal compilation times can be a pain. It can't be that you can use only one processor core with cabal, can it ..?

There are two places where you can increase the use of multiple cores. The one above controls the number of packages to be compiled in parallel, whereas

$ cabal install --ghc-option=-j3

means that each instance of GHC compiles up to three modules in parallel.

This sounds better than compiling only whole packages in parallel.

As far as I know it is currently not possible to interleave compilation of modules and packages. I.e. optimally we would only need one 'jobs' option and then Cabal and GHC manage to fill the working queue with any module from any package. This would certainly require to run GHC in a server mode.

However, without a fast storage for temporary files you will hardly see an actual speedup. I use to mount a ram disk with tmpfs at /ram and then compile with

$ cabal install -j3 --ghc-option=-j3 --builddir=/ram/cabal

That's important. And so you're saying that an SSD isn't fast enough. So I'll probably set up a RAM disk as well. Thanks Volker

Reply

Sign in to reply online Use email software

413

Age (days ago)

413

Last active (days ago)

Download

9 comments

7 participants

tags

participants (7)

Alexey Vagarenko
Andrew Lelechenko
Brandon Allbery
Bryan Richter
coot＠coot.me
Henning Thielemann
Volker Wysk