[GHC] #15051: -split-objs generates excessively many files on Windows
#15051: -split-objs generates excessively many files on Windows --------------------------------------+--------------------------------- Reporter: kanetw | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Keywords: | Operating System: Windows Architecture: x86_64 (amd64) | Type of failure: None/Unknown Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: --------------------------------------+--------------------------------- A BuildFlavour = perf build generated ~18k .S files for ghc- prim:GHC.Types, which caused it to basically get stuck compiling all those assembly files. Removing -split-objs (via SplitObjs = NO) significantly sped up compilation. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15051 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15051: -split-objs generates excessively many files on Windows ---------------------------------+-------------------------------------- Reporter: kanetw | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.5 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Changes (by kanetw): * version: 8.2.2 => 8.5 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15051#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15051: -split-objs generates excessively many files on Windows ---------------------------------+-------------------------------------- Reporter: kanetw | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.6.1 Component: Compiler | Version: 8.5 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Changes (by Phyx-): * priority: normal => highest Comment: Default builds are now broken. it's now generating over 18k worth of split markers, which means Windows builds don't finish in any reasonable amount of time. Does GHC prim really have over 18k worth of symbols now in a single module? Other platforms don't notice this because they use split-sections. Which we can't use on Windows due to a lack of a fast linking process and non-existent x86 implementation. *sigh* perhaps it's time to fix this. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15051#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15051: -split-objs generates excessively many files on Windows ---------------------------------+-------------------------------------- Reporter: kanetw | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.6.1 Component: Compiler | Version: 8.5 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by Phyx-): @kanetw do you happen to know which hash you noticed this on first? It seems to affect the 8.4.1 release tag already. I guess @bgamari just waited hours and hours for the builds to finish? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15051#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15051: -split-objs generates excessively many files on Windows ---------------------------------+-------------------------------------- Reporter: kanetw | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.6.1 Component: Compiler | Version: 8.5 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by kanetw): Um, whatever built was most recent 7 weeks ago. I can give you the hash once I'm back home, but that won't be for another week. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15051#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15051: -split-objs generates excessively many files on Windows ---------------------------------+-------------------------------------- Reporter: kanetw | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.6.1 Component: Compiler | Version: 8.5 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by bgamari): It looks like the source of the symbols is the `Typeable` `KindRep`s generated for the tuple types. However, I'm rather surprised that we are just seeing this now; this logic has been present since 8.2. Are you sure this is a new problem? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15051#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15051: -split-objs generates excessively many files on Windows ---------------------------------+-------------------------------------- Reporter: kanetw | Owner: Phyx- Type: bug | Status: new Priority: highest | Milestone: 8.6.1 Component: Compiler | Version: 8.5 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Changes (by bgamari): * owner: (none) => Phyx- -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15051#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15051: -split-objs generates excessively many files on Windows -------------------------------------+------------------------------------- Reporter: kanetw | Owner: Phyx- Type: bug | Status: new Priority: highest | Milestone: 8.6.1 Component: Compiler | Version: 8.5 (CodeGen) | Resolution: | Keywords: Operating System: Windows | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by Phyx-): * failure: None/Unknown => Compile-time performance bug * component: Compiler => Compiler (CodeGen) * architecture: x86_64 (amd64) => Unknown/Multiple Comment: This is beyond annoying. The windows tarballs are useless after this change. A full compile takes a day and a half in validate mode. even compiling a simple module now takes seconds. So I'm just going to kill SPLIT_OBJS as this change has effectively made it useless. Compiling a simple Cabal module alone takes 900 fork/exec calls. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15051#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15051: -split-objs generates excessively many files on Windows -------------------------------------+------------------------------------- Reporter: kanetw | Owner: Phyx- Type: bug | Status: new Priority: highest | Milestone: 8.6.1 Component: Compiler | Version: 8.5 (CodeGen) | Resolution: | Keywords: Operating System: Windows | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4915 Wiki Page: | Phab:D4916 -------------------------------------+------------------------------------- Changes (by Phyx-): * differential: => Phab:D4915 Phab:D4916 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15051#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15051: -split-objs generates excessively many files on Windows -------------------------------------+------------------------------------- Reporter: kanetw | Owner: Phyx- Type: bug | Status: new Priority: highest | Milestone: 8.6.1 Component: Compiler | Version: 8.5 (CodeGen) | Resolution: | Keywords: Operating System: Windows | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4915 Wiki Page: | Phab:D4916 -------------------------------------+------------------------------------- Comment (by bgamari): Yes, I think just disabling (and ultimately removing) split objects is the right idea here. This is something I've been looking to do for quite some time. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15051#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15051: -split-objs generates excessively many files on Windows -------------------------------------+------------------------------------- Reporter: kanetw | Owner: Phyx- Type: bug | Status: new Priority: highest | Milestone: 8.6.1 Component: Compiler | Version: 8.5 (CodeGen) | Resolution: | Keywords: Operating System: Windows | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4915 Wiki Page: | Phab:D4916 -------------------------------------+------------------------------------- Comment (by Phyx-): It's unfortunate that it leaves the Windows port in a very icky situation. It either gets slower, gets much slower, or gets bigger. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15051#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15051: -split-objs generates excessively many files on Windows -------------------------------------+------------------------------------- Reporter: kanetw | Owner: Phyx- Type: bug | Status: new Priority: highest | Milestone: 8.6.1 Component: Compiler | Version: 8.5 (CodeGen) | Resolution: | Keywords: Operating System: Windows | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4915 Wiki Page: | Phab:D4916 -------------------------------------+------------------------------------- Comment (by bgamari): I agree, but disk space is reasonably cheap now. It seems to me like accepting a temporary increase in size and focusing effort on fixing split sections is a better use of time than trying to fix what is ultimately a dead-end. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15051#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15051: -split-objs generates excessively many files on Windows -------------------------------------+------------------------------------- Reporter: kanetw | Owner: Phyx- Type: bug | Status: new Priority: highest | Milestone: 8.6.1 Component: Compiler | Version: 8.5 (CodeGen) | Resolution: | Keywords: Operating System: Windows | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4915 Wiki Page: | Phab:D4916 -------------------------------------+------------------------------------- Comment (by Phyx-): True, from my own checks the increase in size doesn't seem to be that much. so Yeah I also think disabling it is the way forwards. Also because fundamentally there's nothing really that can be done about it. split-objs just won't scale. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15051#comment:12 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15051: -split-objs generates excessively many files on Windows
-------------------------------------+-------------------------------------
Reporter: kanetw | Owner: Phyx-
Type: bug | Status: new
Priority: highest | Milestone: 8.6.1
Component: Compiler | Version: 8.5
(CodeGen) |
Resolution: | Keywords:
Operating System: Windows | Architecture:
Type of failure: Compile-time | Unknown/Multiple
performance bug | Test Case:
Blocked By: | Blocking:
Related Tickets: | Differential Rev(s): Phab:D4915
Wiki Page: | Phab:D4916
-------------------------------------+-------------------------------------
Comment (by Ben Gamari
#15051: -split-objs generates excessively many files on Windows -------------------------------------+------------------------------------- Reporter: kanetw | Owner: Phyx- Type: bug | Status: new Priority: highest | Milestone: 8.6.1 Component: Compiler | Version: 8.5 (CodeGen) | Resolution: | Keywords: Operating System: Windows | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4915 Wiki Page: | Phab:D4916 -------------------------------------+------------------------------------- Comment (by Zemyla): I've noticed this problem too, but I'm not sure getting rid of split-objs is the solution. The problem, it seems to me, is that it's (a) splitting with a Perl script, and (b) calling gcc with each and every single individual file produced. Is it possible to get rid of the Perl script, at least on Windows, by specializing the split-objects procedure for Windows x86/x64 builds and then using gcc to build many assembly files at a time? Actually, now that I think about it, why is it even using gcc to convert the assembly files into object files in the first place? Doesn't it convert the code directly into an object file when split-objs isn't used? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15051#comment:14 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15051: -split-objs generates excessively many files on Windows -------------------------------------+------------------------------------- Reporter: kanetw | Owner: Phyx- Type: bug | Status: closed Priority: highest | Milestone: 8.6.1 Component: Compiler | Version: 8.5 (CodeGen) | Resolution: fixed | Keywords: Operating System: Windows | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4915 Wiki Page: | Phab:D4916 -------------------------------------+------------------------------------- Changes (by bgamari): * status: new => closed * resolution: => fixed Comment: comment:13 merged to `ghc-8.6` in f14c087a8b88c39e3567af1dde7c2368a5391333. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15051#comment:15 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15051: -split-objs generates excessively many files on Windows -------------------------------------+------------------------------------- Reporter: kanetw | Owner: Phyx- Type: bug | Status: closed Priority: highest | Milestone: 8.6.1 Component: Compiler | Version: 8.5 (CodeGen) | Resolution: fixed | Keywords: Operating System: Windows | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4915 Wiki Page: | Phab:D4916 -------------------------------------+------------------------------------- Comment (by Phyx-):
I've noticed this problem too, but I'm not sure getting rid of split- objs is the solution. The problem, it seems to me, is that it's (a) splitting with a Perl script, and (b) calling gcc with each and every single individual file produced.
Is it possible to get rid of the Perl script, at least on Windows, by specializing the split-objects procedure for Windows x86/x64 builds and
Actually, now that I think about it, why is it even using gcc to convert
(a) is not the problem. The perl script just does a simple linear scan looking for split markers and breaks it up. The overall runtime of the split script is negligible. (b) It won't work without this. split-objs just exploits the fact that linkers pull in library code on demand. If no symbols in an object file in an archive is needed it's not pulled in. split-section uses "linker stubs" to get the same effect, the tiny object files are pre-linked together to get one giant object file where each original .s file is a new stub. Passing all the .s files to the assembler will cause it to merge the contents of the sections and you'd get one linker stub. The net effect would be the same as not splitting to begin with. While you could force the creation of a linker stub with a `.file` directive for each part, the result still won't be the same as your `.text` section header will still be merged. then using gcc to build many assembly files at a time? Not really, as I've explained above. the assembly files into object files in the first place? Doesn't it convert the code directly into an object file when split-objs isn't used? No, GHC doesn't have an assembler or static linker. It always uses an external program for both of these cases. It only has a runtime linker for GHCi and Template Haskell. As it stands, `-split-objs` is simply a dead approach, it's better to invest the time into getting `-split-sections` working, which doesn't rely on hacks. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15051#comment:16 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15051: -split-objs generates excessively many files on Windows
-------------------------------------+-------------------------------------
Reporter: kanetw | Owner: Phyx-
Type: bug | Status: closed
Priority: highest | Milestone: 8.6.1
Component: Compiler | Version: 8.5
(CodeGen) |
Resolution: fixed | Keywords:
Operating System: Windows | Architecture:
Type of failure: Compile-time | Unknown/Multiple
performance bug | Test Case:
Blocked By: | Blocking:
Related Tickets: | Differential Rev(s): Phab:D4915
Wiki Page: | Phab:D4916
-------------------------------------+-------------------------------------
Comment (by Ben Gamari
participants (1)
-
GHC