Re: Compilation time

Ryan Scott

7 Jul 2017 7 Jul '17

11:26 a.m.

Another things to note about 8.2 build times is that a bulk of the time is probably being spent during the linking phase. This is because GHC 8.2 has switched over to split-sections by default, but unfortunately, old versions of the BFD linker take a long time to link code that uses split-sections (see the discussion at [1]). There is ongoing work (which should hopefully land before the final 8.2 release) to use gold instead of BFD as the default linker when available, which will reduce linking times dramatically. Ryan S. ----- [1] https://ghc.haskell.org/trac/ghc/ticket/13739

Attachments:

attachment.html (text/html — 711 bytes)

Show replies by date

Ben Gamari

7 Jul 7 Jul

12:28 p.m.

New subject: Compilation time

Ryan Scott writes:

...

Another things to note about 8.2 build times is that a bulk of the time is probably being spent during the linking phase. This is because GHC 8.2 has switched over to split-sections by default, but unfortunately, old versions of the BFD linker take a long time to link code that uses split-sections (see the discussion at [1]). There is ongoing work (which should hopefully land before the final 8.2 release) to use gold instead of BFD as the default linker when available, which will reduce linking times dramatically.

Indeed it has already landed. See #13541. In short, ./configure will now choose to use ld.gold or ld.lld if available (although this can be disabled using the --disable-ld-override configure flag). Cheers, - Ben

Evan Laforge

6:21 p.m.

New subject: Compilation time

On Fri, Jul 7, 2017 at 9:28 AM, Ben Gamari wrote:

...

Ryan Scott writes:

...
Another things to note about 8.2 build times is that a bulk of the time is probably being spent during the linking phase. This is because GHC 8.2 has switched over to split-sections by default, but unfortunately, old versions of the BFD linker take a long time to link code that uses split-sections (see the discussion at [1]). There is ongoing work (which should hopefully land before the final 8.2 release) to use gold instead of BFD as the default linker when available, which will reduce linking times dramatically.

Indeed it has already landed. See #13541.

In short, ./configure will now choose to use ld.gold or ld.lld if available (although this can be disabled using the --disable-ld-override configure flag).

Just from curiosity, does this apply to OS X? Of course, gold is ELF-only, so it surely doesn't apply, but there's still lld. OS X uses clang to compile so I thought it might already use lld, but the 'ld -v' output looks different, and lld.llvm.org implies it's a replacement for OS X ld, so maybe not. But it doesn't look like GNU ld either, so maybe it's not affected by the BFD bug? I'll try 8.2 on OS X and see if the link time changes.

Ben Gamari

7:33 p.m.

New subject: Compilation time

Evan Laforge writes:

...

On Fri, Jul 7, 2017 at 9:28 AM, Ben Gamari wrote:

...
In short, ./configure will now choose to use ld.gold or ld.lld if available (although this can be disabled using the --disable-ld-override configure flag).

Just from curiosity, does this apply to OS X? Of course, gold is ELF-only, so it surely doesn't apply, but there's still lld. OS X uses clang to compile so I thought it might already use lld, but the 'ld -v' output looks different, and lld.llvm.org implies it's a replacement for OS X ld, so maybe not. But it doesn't look like GNU ld either, so maybe it's not affected by the BFD bug?

I'll try 8.2 on OS X and see if the link time changes.

8.2 will prefer both gold and lld over bfd ld. However two conditions must hold for these to be used, * The ld.lld/ld.gold executable must be in $PATH (or explicitly named by passing the LD variable to configure) * $CC must understand the `-fuse-ld={gold,lld}` option. For (IMHO quite silly) political reasons, gcc doesn't support `-fuse-ld=lld`. Debian happens to patch gcc to add support but I don't know how common this is in other distributions. Unfortunately, some earlier `gcc` versions didn't fail if given a `-fuse-ld` option that they didn't understand. Sadly we have no reliable way to detect this, so in this case we may end up passing a `-fuse-ld` option that gcc simply ignores. In the case of OS X we use Apple's own home-grown linker. I'm not sure whether/how OS X's gcc wrapper treats `-fuse-ld` (beyond that it doesn't error if the flag is given). I also don't know whether lld is currently a capable replacement for OS X ld. Cheers, - Ben

Moritz Angermann

9:45 p.m.

New subject: Compilation time

For those of us who are on macOS, you usually do not have any gcc, it’s all clang behind the curtains. E.g. see $ /usr/bin/gcc --version Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1 Apple LLVM version 8.1.0 (clang-802.0.42) Target: x86_64-apple-darwin16.6.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin ld, is apples ld64 (see https://opensource.apple.com/source/ld64/) $ /usr/bin/ld -v @(#)PROGRAM:ld PROJECT:ld64-278.4 configured to support archs: armv6 armv7 armv7s arm64 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em (tvOS) LTO support using: LLVM version 8.1.0, (clang-802.0.42) TAPI support using: Apple TAPI version 1.33.11 if you end up having clang+llvm-4.0.0-x86_64-apple-darwin/bin or similar in your path, you WILL have `ld.lld`, but that one is an ELF linker, which on a MachO platform won’t be very useful (and will therefore fail to link MachO objects). Llvm’s `lld` will provide you with the following semi helpful note: $ lld lld is a generic driver. Invoke ld.lld (Unix), ld (Mac) or lld-link (Windows) instead. however, my copy of clang+llvm, does only ship ld.lld, and lld-link, but not `ld`. Thus, if you end up having the llvm tools (and ld.lld) in your PATH, you will need to set `--disable-ld-override` during configure on macOS, or your ghc build will fail, because you end up trying to link MachO object files with an ELF only linker. Cheers, Moritz

...

On Jul 8, 2017, at 7:33 AM, Ben Gamari wrote:

Evan Laforge writes:

...
On Fri, Jul 7, 2017 at 9:28 AM, Ben Gamari wrote:

...
In short, ./configure will now choose to use ld.gold or ld.lld if available (although this can be disabled using the --disable-ld-override configure flag).

Just from curiosity, does this apply to OS X? Of course, gold is ELF-only, so it surely doesn't apply, but there's still lld. OS X uses clang to compile so I thought it might already use lld, but the 'ld -v' output looks different, and lld.llvm.org implies it's a replacement for OS X ld, so maybe not. But it doesn't look like GNU ld either, so maybe it's not affected by the BFD bug?

I'll try 8.2 on OS X and see if the link time changes.

8.2 will prefer both gold and lld over bfd ld. However two conditions must hold for these to be used,

* The ld.lld/ld.gold executable must be in $PATH (or explicitly named by passing the LD variable to configure)

* $CC must understand the `-fuse-ld={gold,lld}` option. For (IMHO quite silly) political reasons, gcc doesn't support `-fuse-ld=lld`. Debian happens to patch gcc to add support but I don't know how common this is in other distributions.

Unfortunately, some earlier `gcc` versions didn't fail if given a `-fuse-ld` option that they didn't understand. Sadly we have no reliable way to detect this, so in this case we may end up passing a `-fuse-ld` option that gcc simply ignores.

In the case of OS X we use Apple's own home-grown linker. I'm not sure whether/how OS X's gcc wrapper treats `-fuse-ld` (beyond that it doesn't error if the flag is given). I also don't know whether lld is currently a capable replacement for OS X ld.

Cheers,

- Ben _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Ben Gamari

10:37 p.m.

New subject: Compilation time

Moritz Angermann writes:

...

For those of us who are on macOS,

...

Thus, if you end up having the llvm tools (and ld.lld) in your PATH, you will need to set `--disable-ld-override` during configure on macOS, or your ghc build will fail, because you end up trying to link MachO object files with an ELF only linker.

Yikes, this is quite bad. We'll need to teach autoconf to recognize this before the release. Thanks for bringing this up! Cheers, - Ben

Karel Gardas

8 Jul 8 Jul

2:47 a.m.

New subject: Compilation time

On 07/ 8/17 01:33 AM, Ben Gamari wrote:

...

8.2 will prefer both gold and lld over bfd ld. However two conditions must hold for these to be used,

* The ld.lld/ld.gold executable must be in $PATH (or explicitly named by passing the LD variable to configure)

* $CC must understand the `-fuse-ld={gold,lld}` option. For (IMHO quite silly) political reasons, gcc doesn't support `-fuse-ld=lld`. Debian happens to patch gcc to add support but I don't know how common this is in other distributions.

Unfortunately, some earlier `gcc` versions didn't fail if given a `-fuse-ld` option that they didn't understand. Sadly we have no reliable way to detect this, so in this case we may end up passing a `-fuse-ld` option that gcc simply ignores.

I've run into this issue too, but it looks like the issue is not in gcc, but in ghc's aclocal.m4 (one '$' missing). Attached patch solves this on OpenBSD 6.1-current where HEAD fails building on linker error. It passes build-id option to the gcc's linker (bfd ld) which does not support it as configure detects lld is presented on this system too. Sorry for not being able to push that through usual arc. Thanks, Karel

2873

Age (days ago)

2874

Last active (days ago)

List overview

Download

6 comments

5 participants

participants (5)

Ben Gamari
Evan Laforge
Karel Gardas
Moritz Angermann
Ryan Scott