[GHC] #8405: experiment with using function-sections for linking (for smaller libs and executables)
#8405: experiment with using function-sections for linking (for smaller libs and executables) ------------------------------------+------------------------------------- Reporter: carter | Owner: Type: task | Status: new Priority: normal | Milestone: 7.10.1 Component: Compiler | Version: 7.6.3 Keywords: | Operating System: Unknown/Multiple Architecture: Unknown/Multiple | Type of failure: None/Unknown Difficulty: Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | ------------------------------------+------------------------------------- currently the only support we have for making small (ish) executables using split objects, which 1. is not on by default, for various good reasons 2. can seriously penalize compile times 3. in tandem, something like strip is often used, but on some platforms/targets (such as the iOS cross compiler), strip will actually break things might be worth seriously experimenting with function-sections / gc- sections in the object files we generate. example slide deck on this http://elinux.org/images/2/2d/ELC2010-gc- sections_Denys_Vlasenko.pdf -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8405> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8405: experiment with using function-sections for linking (for smaller libs and executables) -------------------------------------+------------------------------------ Reporter: carter | Owner: Type: task | Status: new Priority: normal | Milestone: 7.10.1 Component: Compiler | Version: 7.6.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: None/Unknown | Difficulty: Unknown Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Changes (by refold): * cc: the.dead.shall.rise@… (added) -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8405#comment:1> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8405: experiment with using function-sections for linking (for smaller libs and executables) -------------------------------------+------------------------------------ Reporter: carter | Owner: Type: task | Status: new Priority: normal | Milestone: 7.10.1 Component: Compiler | Version: 7.6.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: None/Unknown | Difficulty: Unknown Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by refold): 4. Apparently doesn't work with `-fllvm` (#8300) -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8405#comment:2> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8405: experiment with using function-sections for linking (for smaller libs and executables) -------------------------------------+------------------------------------ Reporter: carter | Owner: Type: task | Status: new Priority: normal | Milestone: 7.10.1 Component: Compiler | Version: 7.6.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: None/Unknown | Difficulty: Unknown Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by refold): Looks like --gc-sections is not supported on Windows: http://sourceware.org/bugzilla/show_bug.cgi?id=11539 -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8405#comment:3> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8405: experiment with using function-sections for linking (for smaller libs and executables) -------------------------------------+------------------------------------ Reporter: carter | Owner: Type: task | Status: new Priority: normal | Milestone: 7.10.1 Component: Compiler | Version: 7.6.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: None/Unknown | Difficulty: Unknown Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Changes (by erikd): * cc: mle+hs@… (added) -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8405#comment:4> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8405: experiment with using function-sections for linking (for smaller libs and executables) -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: task | Status: new Priority: normal | Milestone: 7.12.1 Component: Compiler | Version: 7.6.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by olsner): Another benefit of this would be getting rid of the satanic splitter, build system support for split objects, and any other splitting-related hacks there might be laying around. FWIW, I've been hacking at a patch to do this in the native code generator (https://github.com/olsner/ghc/commits/function-sections). It passes almost all of the test suite now, and generated binaries for a simple "hello world" did get smaller. But that's compared to a build without split objs - for some reason function sections doesn't seem to help nearly as much as split-objs... So that's something I'll want to fix first. I have also only tested this on Linux so far. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8405#comment:6> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8405: experiment with using function-sections for linking (for smaller libs and executables) -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: task | Status: new Priority: normal | Milestone: 7.12.1 Component: Compiler | Version: 7.6.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by rwbarton): Yay, happy to hear someone is looking at this! -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8405#comment:7> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8405: experiment with using function-sections for linking (for smaller libs and executables) -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: task | Status: new Priority: normal | Milestone: 7.12.1 Component: Compiler | Version: 7.6.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by thomie): Awesome. How about the size of the .a files? With `-split-objs`, `.a` files tend to get a factor of 2 to 2.5x larger. This is one of the reasons why a ghc installation is as large as it is (~900Mb for me). -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8405#comment:8> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8405: experiment with using function-sections for linking (for smaller libs and executables) -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: task | Status: new Priority: normal | Milestone: 7.12.1 Component: Compiler | Version: 7.6.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by olsner): Comparing the size of libraries/*/dist-install/build/*.a from each tree, the sections build was 71MB and the split objects build was 99MB, compared to 41MB for a non-split build. Slightly less bloat than split-objs at least. Meanwhile, I found the reason for the missing savings compared to split- objs: I had missed that SRT table generation was generating split or monolithic SRT tables depending on the split-objs flag. For now, I made the split SRTs unconditional (since the section splitting is also always on), and that made my "hello world" 10% smaller with sections than with split-objs (20% diff after stripping). -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8405#comment:9> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8405: experiment with using function-sections for linking (for smaller libs and executables) -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: task | Status: new Priority: normal | Milestone: 7.12.1 Component: Compiler | Version: 7.6.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by refold): Replying to [comment:9 olsner]:
[...] that made my "hello world" 10% smaller with sections than with
split-objs (20% diff after stripping). Very cool, thanks for working on this! Link times with `--gc-sections` should be also much faster than with split-objs, especially for something like OpenGL. And it looks like `--gc-sections` [https://sourceware.org/git/gitweb.cgi?p =binutils-gdb.git;h=0f088b2a9417b1d4ed597849ffa671eba25f5051 is now finally supported on Windows] (though the patch still needs testing). -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8405#comment:10> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8405: experiment with using function-sections for linking (for smaller libs and executables) -------------------------------------+------------------------------------- Reporter: carter | Owner: olsner Type: task | Status: new Priority: normal | Milestone: 7.12.1 Component: Compiler | Version: 7.6.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Changes (by olsner): * owner: => olsner Comment: It seems GNU ld actually has scalability issues with very large numbers of sections too, unfortunately. Found out because I was building libHSghc with split sections, and getting 10 minute link times on some test cases with plugins linking statically against ghc... Since split-objs is specifically disabled for that code I simply disabled split-sections as well. A much cleaned up version of the patch now at https://github.com/olsner/ghc/commits/function-sections-2 - this adds a -split-sections flag that controls the splitting when compiling Haskell code. --gc-sections is always added to the linker command line. (Vague rationale being that after building libraries with split sections, we'll want to bring space savings to users without manual steps or extra flags building their programs/libraries.) -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8405#comment:11> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8405: experiment with using function-sections for linking (for smaller libs and executables) -------------------------------------+------------------------------------- Reporter: carter | Owner: olsner Type: task | Status: new Priority: normal | Milestone: 7.12.1 Component: Compiler | Version: 7.6.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by refold): Replying to [comment:11 olsner]:
It seems GNU ld actually has scalability issues with very large numbers of sections too, unfortunately.
Have you tried the gold linker? -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8405#comment:12> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8405: experiment with using function-sections for linking (for smaller libs and executables) -------------------------------------+------------------------------------- Reporter: carter | Owner: olsner Type: task | Status: new Priority: normal | Milestone: 8.0.1 Component: Compiler | Version: 7.6.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: Phab:D1242 -------------------------------------+------------------------------------- Changes (by olsner): * differential: => Phab:D1242 -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8405#comment:14> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8405: experiment with using function-sections for linking (for smaller libs and executables) -------------------------------------+------------------------------------- Reporter: carter | Owner: olsner Type: task | Status: new Priority: normal | Milestone: 8.0.1 Component: Compiler | Version: 7.6.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D1242 -------------------------------------+------------------------------------- Comment (by Ben Gamari <ben@…>): In [changeset:"3021cc06f84109c6e7a529318d690c520748f936/ghc" 3021cc06/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="3021cc06f84109c6e7a529318d690c520748f936" Add rts/Linker support for more than 64k sections Since some ELF fields ran out of range to represent that many sections, they've been extended with magic numbers that indicate that the full value is stored in another field. This will be necessary for GHCi with -split-sections on ELF platforms that don't use GNU ld. Reviewers: austin, bgamari, simonmar, erikd Reviewed By: bgamari, simonmar, erikd Subscribers: thomie Differential Revision: https://phabricator.haskell.org/D1357 GHC Trac Issues: #8405 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8405#comment:15> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8405: experiment with using function-sections for linking (for smaller libs and executables) -------------------------------------+------------------------------------- Reporter: carter | Owner: olsner Type: task | Status: new Priority: normal | Milestone: 8.0.1 Component: Compiler | Version: 7.6.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D1242 -------------------------------------+------------------------------------- Comment (by Ben Gamari <ben@…>): In [changeset:"4a32bf925b8aba7885d9c745769fe84a10979a53/ghc" 4a32bf9/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="4a32bf925b8aba7885d9c745769fe84a10979a53" Implement function-sections for Haskell code, #8405 This adds a flag -split-sections that does similar things to -split-objs, but using sections in single object files instead of relying on the Satanic Splitter and other abominations. This is very similar to the GCC flags -ffunction-sections and -fdata-sections. The --gc-sections linker flag, which allows unused sections to actually be removed, is added to all link commands (if the linker supports it) so that space savings from having base compiled with sections can be realized. Supported both in LLVM and the native code-gen, in theory for all architectures, but really tested on x86 only. In the GHC build, a new SplitSections variable enables -split-sections for relevant parts of the build. Test Plan: validate with both settings of SplitSections Reviewers: dterei, Phyx, austin, simonmar, thomie, bgamari Reviewed By: simonmar, thomie, bgamari Subscribers: hsyl20, erikd, kgardas, thomie Differential Revision: https://phabricator.haskell.org/D1242 GHC Trac Issues: #8405 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8405#comment:16> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8405: experiment with using function-sections for linking (for smaller libs and executables) -------------------------------------+------------------------------------- Reporter: carter | Owner: olsner Type: task | Status: closed Priority: normal | Milestone: 8.0.1 Component: Compiler | Version: 7.6.3 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D1242 Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * status: new => closed * resolution: => fixed Comment: We now have support for this. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8405#comment:17> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8405: experiment with using function-sections for linking (for smaller libs and executables) -------------------------------------+------------------------------------- Reporter: carter | Owner: olsner Type: task | Status: closed Priority: normal | Milestone: 8.0.1 Component: Compiler | Version: 7.6.3 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D1242 Wiki Page: | -------------------------------------+------------------------------------- Comment (by refold): Awesome, thanks to everyone involved! -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8405#comment:18> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8405: experiment with using function-sections for linking (for smaller libs and executables) -------------------------------------+------------------------------------- Reporter: carter | Owner: olsner Type: task | Status: closed Priority: normal | Milestone: 8.0.1 Component: Compiler | Version: 7.6.3 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D1242 Wiki Page: | -------------------------------------+------------------------------------- Comment (by carter): Wow. Great! Would this make it practical to disable stripping on libraries and executables (so that dwarf metadata could be available by default?) -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8405#comment:19> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8405: experiment with using function-sections for linking (for smaller libs and executables) -------------------------------------+------------------------------------- Reporter: carter | Owner: olsner Type: task | Status: closed Priority: normal | Milestone: 8.0.1 Component: Compiler | Version: 7.6.3 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D1242 Wiki Page: | -------------------------------------+------------------------------------- Comment (by olsner): Replying to [comment:19 carter]:
Would this make it practical to disable stripping on libraries and executables (so that dwarf metadata could be available by default?)
I didn't know that was a problem, so if this feature fixes anything it's purely a bonus :) I suspect the debug info could get pretty large regardless of this patch, so many users might still not want it. (Though it might be improved compared to `-split-objs`?) Replying to [comment:10 refold]:
And it looks like `--gc-sections` [https://sourceware.org/git/gitweb.cgi?p=binutils- gdb.git;h=0f088b2a9417b1d4ed597849ffa671eba25f5051 is now finally supported on Windows] (though the patch still needs testing).
Since I had my system in Windows today, I decided to test this out - but ran into some problems. First problem is that linking executables with `SplitSections=YES` gets you a "too many sections" error. But when making an executable, those sections should be getting merged into just a handful of sections... I think this is `ld`'s fault, so I submitted https://sourceware.org/bugzilla/show_bug.cgi?id=19254 for it. It also seems the `--gc-sections` patch was committed after binutils 2.25 branched off, so we need to upgrade binutils to use this. (Probably to a yet-unreleased version that also has a fix for the sections issue...) -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8405#comment:20> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8405: experiment with using function-sections for linking (for smaller libs and executables) -------------------------------------+------------------------------------- Reporter: carter | Owner: olsner Type: task | Status: closed Priority: normal | Milestone: 8.0.1 Component: Compiler | Version: 7.6.3 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D1242 Wiki Page: | -------------------------------------+------------------------------------- Comment (by Tamar Christina <tamar@…>): In [changeset:"7f77e4e9b301493db0782cf2d129cf62dbcd5af6/ghc" 7f77e4e/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="7f77e4e9b301493db0782cf2d129cf62dbcd5af6" Fix Windows builds after D1242 Summary: Only use the merge_sections.ld linker script if SplitSections is enabled. I thought I had a way to make the linker script actually work on Windows and produce object files instead of image files (by using the "INSERT AFTER" linker script command to get the default script's output format setting), but that fix caused ghci to crash with an illegal instruction on startup. Gave up and made a simpler fix of just disabling this for normal builds, as it's only really relevant with SplitSections enabled anyway. Reviewers: austin, thomie, bgamari Reviewed By: austin, thomie Subscribers: Phyx, thomie Differential Revision: https://phabricator.haskell.org/D1505 GHC Trac Issues: #8405 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8405#comment:21> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8405: experiment with using function-sections for linking (for smaller libs and executables) -------------------------------------+------------------------------------- Reporter: carter | Owner: olsner Type: task | Status: closed Priority: normal | Milestone: 8.0.1 Component: Compiler | Version: 7.6.3 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D1242 Wiki Page: | -------------------------------------+------------------------------------- Comment (by ezyang): Link-time performance observation here: #11285; when compiling `Setup.sh`, split sections makes ld.bfd run four times slower than split objects, but ld.gold two times faster than split objects. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8405#comment:22> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8405: experiment with using function-sections for linking (for smaller libs and executables) -------------------------------------+------------------------------------- Reporter: carter | Owner: olsner Type: task | Status: closed Priority: normal | Milestone: 8.0.1 Component: Compiler | Version: 7.6.3 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D1242 Wiki Page: | -------------------------------------+------------------------------------- Comment (by Ben Gamari <ben@…>): In [changeset:"a50082c115bed1891b2e5aac4a21462935f4f0d6/ghc" a50082c/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="a50082c115bed1891b2e5aac4a21462935f4f0d6" Apply SplitSections to all C compilations Previously this was added only to the RTS's C files (those are the bulk of it though), but there are C bits in ghc-prim, integer-gmp and base too. Followup for #8405, allows the large table of character properties in base to be stripped when not used. Test Plan: validate Reviewers: austin, bgamari, simonmar Reviewed By: bgamari Subscribers: thomie, snowleopard Differential Revision: https://phabricator.haskell.org/D3121 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8405#comment:23> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
participants (1)
-
GHC