[GHC] #16084: Improve link times on Windows
#16084: Improve link times on Windows -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: bug | Status: new Priority: high | Milestone: Component: Compiler | Version: 8.6.3 Keywords: | Operating System: Windows Architecture: | Type of failure: Compile-time Unknown/Multiple | performance bug Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- Currently the Windows CI builds are typically lasting over 3 hours. The primary cause of this appears to be poor performance in `ld.bfd`, especially when linking testsuite tests. One option to fix this is to try using LLD for linking. Unfortunately the msys2 `gcc` does not support `-fuse-ld=lld`. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/16084> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#16084: Improve link times on Windows -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: bug | Status: new Priority: high | Milestone: Component: Compiler | Version: 8.6.3 Resolution: | Keywords: Operating System: Windows | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): I have opened an `msys2` [[https://github.com/Alexpux/MSYS2-packages/issues/1531|ticket]] requesting that `gcc` ship with support for `-fuse-ld=lld`. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/16084#comment:1> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#16084: Improve link times on Windows -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: bug | Status: upstream Priority: high | Milestone: Component: Compiler | Version: 8.6.3 Resolution: | Keywords: Operating System: Windows | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * status: new => upstream -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/16084#comment:2> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#16084: Improve link times on Windows -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: bug | Status: upstream Priority: high | Milestone: Component: Compiler | Version: 8.6.3 Resolution: | Keywords: Operating System: Windows | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * cc: Phyx (added) -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/16084#comment:3> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#16084: Improve link times on Windows -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: bug | Status: upstream Priority: high | Milestone: Component: Compiler | Version: 8.6.3 Resolution: | Keywords: Operating System: Windows | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): Tamar has done a bit of working optimising `ld.bfd`: https://github.com/Mistuke/binutils-gdb -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/16084#comment:4> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#16084: Improve link times on Windows -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: bug | Status: upstream Priority: high | Milestone: Component: Compiler | Version: 8.6.3 Resolution: | Keywords: Operating System: Windows | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by awson): Replying to [ticket:16084 bgamari]:
One option to fix this is to try using LLD for linking. Unfortunately the msys2 `gcc` does not support `-fuse-ld=lld`.
LLD won't magically help here. Latest improvements by Martin Storsjö (mstorsjo) have made LLD able to link some mingw **gcc**-generated code, but, alas, when assembling **GHC**-generated assembly, mingw **gas** produces (I believe this is a sort of "optimisation") non-standard relocations, which LLD is unable to deal with (honestly, I haven't checked if this is still the case, but it was definitely so a year or two ago). For quite a bit of time I have a GHC port which works flawlessly against native Windows SDK and uses clang in MSVC mode and MS or LLD linker. After recent LLD improvements by mstorsjo I've decided to try a less intrusive approach — use clang in mingw mode and LLD linker. That required quite a bit of work and I finally managed to produce stage2 GHC executable, which appeared to be severely broken — it is unable to do anything. TL;DR the current state of affairs is that `lld` **can't** serve as a drop-in replacement for `ld` not only when GNU binutils toolchain is used, but even when `clang` is used as an assembler. Perhaps, the latter might work against mingw SDK built using LLVM tools (e.g. https://github.com/mstorsjo/llvm-mingw), but I never tried this. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/16084#comment:5> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#16084: Improve link times on Windows -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: bug | Status: upstream Priority: high | Milestone: Component: Compiler | Version: 8.6.3 Resolution: | Keywords: Operating System: Windows | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by adamse): * cc: adamse (added) -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/16084#comment:6> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#16084: Improve link times on Windows -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: bug | Status: upstream Priority: high | Milestone: Component: Compiler | Version: 8.6.3 Resolution: | Keywords: Operating System: Windows | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari):
LLD won't magically help here. Latest improvements by Martin Storsjö (mstorsjo) have made LLD able to link some mingw gcc-generated code, but, alas, when assembling GHC-generated assembly, mingw gas produces (I believe this is a sort of "optimisation") non-standard relocations.
Frankly I wonder how difficult it would be to add support for these relocations. Given that LLVM code is generally pretty approachable I suspect that would be the easiest path forward. In the meantime I have been pursuing testing Tamar's binutils patch. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/16084#comment:7> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#16084: Improve link times on Windows -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: bug | Status: upstream Priority: high | Milestone: Component: Compiler | Version: 8.6.3 Resolution: | Keywords: Operating System: Windows | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by awson): I think it's easy, but (as I've already mentioned) the other problems exist. Mingw binutils introduce a lot of non-standard things, which `LLD` didn't support at all until recently, now some support have appeared, but AFAIUI, things go best when mingw sdk is built by LLVM toolsuite, not by mingw gcc/binutils. And a general problem is that wrong linkage bugs are *very* hard to debug, this is why I haven't ever tried to continue my mingw clang/lld experiment — it might consume an unpredictable amount of time to understand what is wrong with the generated ghc executable — all the symptoms are that the image file is invalid, since the OS can't even load it properly. Replying to [comment:7 bgamari]:
Frankly I wonder how difficult it would be to add support for these relocations. Given that LLVM code is generally pretty approachable I suspect that would be the easiest path forward.
In the meantime I have been pursuing testing Tamar's binutils patch.
-- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/16084#comment:8> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#16084: Improve link times on Windows -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: bug | Status: upstream Priority: high | Milestone: Component: Compiler | Version: 8.6.3 Resolution: | Keywords: Operating System: Windows | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by Ben Gamari <ben@…>): In [changeset:"bb06c6b12c4a24c049fa8fbc9c470f12d625ca18/ghc" bb06c6b1/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="bb06c6b12c4a24c049fa8fbc9c470f12d625ca18" gitlab-ci: Allow Windows to fail for now While we sort out #16084. }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/16084#comment:9> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#16084: Improve link times on Windows -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: bug | Status: upstream Priority: high | Milestone: Component: Compiler | Version: 8.6.3 Resolution: | Keywords: Operating System: Windows | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by Ben Gamari <ben@…>): In [changeset:"942b501972139ed95b49b75cb8c0523b460f6d10/ghc" 942b5019/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="942b501972139ed95b49b75cb8c0523b460f6d10" gitlab-ci: Try only building Windows in the quick flavour It seems no matter how many machines I throw at Windows it's constantly behind. Perhaps the quick build flavour will be fast enough to allow us to keep until while we sort out our toolchain issues (#16084). }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/16084#comment:10> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#16084: Improve link times on Windows -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: bug | Status: upstream Priority: high | Milestone: Component: Compiler | Version: 8.6.3 Resolution: | Keywords: Operating System: Windows | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): I have pushed a new branch testing a new `binutils` build provided by Tamar to `wip/fix-windows-ci` (see !495). -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/16084#comment:11> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
participants (1)
-
GHC