#9832: Get rid of PERL dependency of `ghc-split` -------------------------------------+------------------------------------- Reporter: hvr | Owner: rwbarton Type: task | Status: new Priority: high | Milestone: 7.12.1 Component: Driver | Version: Resolution: | Keywords: perl Operating System: Windows | Architecture: Unknown/Multiple Type of failure: | Difficulty: Difficult (2-5 None/Unknown | days) Test Case: | Blocked By: Blocking: | Related Tickets: Differential Revisions: | -------------------------------------+------------------------------------- Changes (by rwbarton): * owner: => rwbarton Comment: I am in this area currently (planning to add split-objs support to the LLVM backend, which will require some refactoring somewhere), so will mark myself owner for now at least. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9832#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9832: Get rid of PERL dependency of `ghc-split` -------------------------------------+------------------------------------- Reporter: hvr | Owner: rwbarton Type: task | Status: new Priority: high | Milestone: 7.12.1 Component: Driver | Version: Resolution: | Keywords: perl Operating System: Windows | Architecture: Unknown/Multiple Type of failure: | Difficulty: Difficult (2-5 None/Unknown | days) Test Case: | Blocked By: Blocking: | Related Tickets: Differential Revisions: | -------------------------------------+------------------------------------- Comment (by ezyang): By the way, if you do (2), be *very* careful about memory usage. GHC outputs C-- in a streaming fashion, so if you accidentally accumulate everything in memory as you are splitting objects you will spike memory usage. In principle it should be possible to make sure it streams properly, but putting the splitter in a separate process certainly makes this job easier. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9832#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9832: Get rid of PERL dependency of `ghc-split` -------------------------------------+------------------------------------- Reporter: hvr | Owner: Type: task | Status: new Priority: high | Milestone: 7.12.1 Component: Driver | Version: Resolution: | Keywords: perl Operating System: Windows | Architecture: Type of failure: None/Unknown | Unknown/Multiple Blocked By: | Test Case: Related Tickets: | Blocking: | Differential Revisions: -------------------------------------+------------------------------------- Changes (by rwbarton): * owner: rwbarton => Comment: No, I'm not actively working on this and I don't intend to in the near future. Are you planning to just rewrite ghc-split in Haskell, or to change its interface in some way? If the latter, could you describe the changes you plan to make here so I can comment on whether they would be suitable for the LLVM backend also? No point in redesigning twice. Thanks. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9832#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9832: Get rid of PERL dependency of `ghc-split` -------------------------------------+------------------------------------- Reporter: hvr | Owner: Phyx- Type: task | Status: new Priority: high | Milestone: 7.12.1 Component: Driver | Version: Resolution: | Keywords: perl Operating System: Windows | Architecture: Type of failure: None/Unknown | Unknown/Multiple Blocked By: | Test Case: Related Tickets: | Blocking: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by Phyx-): Just a small update in case anyone is watching. I've mostly got this done. Ended up inlining one of the PCRE bindings since rewriting this without Perl Regex would have ended up in something a lot bigger and complexer. Currently missing the apple-Darwin one implementation and then I'll start cleaning up the code and performing more thorough testing. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9832#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9832: Get rid of PERL dependency of `ghc-split` -------------------------------------+------------------------------------- Reporter: hvr | Owner: Phyx- Type: task | Status: new Priority: high | Milestone: 7.12.1 Component: Driver | Version: Resolution: | Keywords: perl Operating System: Windows | Architecture: Type of failure: None/Unknown | Unknown/Multiple Blocked By: | Test Case: Related Tickets: | Blocking: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by thomie):

Does validate allow for testing other binaries than GHC? You can try adding a `run_command` test. See Building/RunningTests/Adding.

You'll need to tell the testsuite where to find the splitter. See find_tool and canonicaliseExecutable in `testsuite/mk/boilerplate.mk`. Alternatively, use a `compile` test that sets `-split-objs`. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9832#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9832: Get rid of PERL dependency of `ghc-split` -------------------------------------+------------------------------------- Reporter: hvr | Owner: Phyx- Type: task | Status: new Priority: high | Milestone: 7.12.1 Component: Driver | Version: Resolution: | Keywords: perl Operating System: Windows | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Changes (by thomie): * cc: simonmar (added) Comment:

Ended up inlining one of the PCRE bindings since rewriting this without Perl Regex would have ended up in something a lot bigger and complexer.

I [https://github.com/Mistuke/ghc/tree/trac-9832-replace-split-perl- script-with-haskell noticed] that this adds 30.000 lines of C and 1700 lines of Haskell. Not sure if that's any better than just bundling Perl on Windows + a 300 line Perl script. What is the problem we're trying so solve here? If it's binary size: we've been shipping Python and fortran compilers as well, maybe start with stripping those (#9014). Perhaps we can get rid of split-objs completely? I found the following quote by @simonmar in SharedLibraries: "Shared libs also completely eliminates the need for the “split objs” hack that GHC uses to reduce the size of statically linked programs. This should make our link times a bit quicker." -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9832#comment:12 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9832: Get rid of PERL dependency of `ghc-split` -------------------------------------+------------------------------------- Reporter: hvr | Owner: Phyx- Type: task | Status: new Priority: high | Milestone: 7.12.1 Component: Driver | Version: Resolution: | Keywords: perl Operating System: Windows | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by Phyx-): I'm not quite sure the problem being solved here is either, but I don't think it's binary size. since the perl distribution (looking at GHC 7.10.1) comes down to a `perl.exe` and a `perl56.dll` which together with the perl `GHC-split` file comes to a smaller since than the Haskell `GHC- split`. My assumption has been that the problem being addressed is to lower the amount of "other" languages used by the compiler/tools. The actual implementation isn't that big, what ended up being big is the fact that I needed to use a perl regular expression library unless I rewrite it to using something else, like parsec maybe... Two things could be done: First I could instead of having the `pcre` source inlined I could depend on the availability of `libpcre` being available on the build platform. This would remove the 30,000 lines of C but introduce an extra dependency. Secondly I can strip the `pcre` Haskell bindings down to the bare essentials that I actually use. This would be a very small subset of the current code. But would also mean that if you want to extend this and need any features I didn't need before from the lib the code would have to be re-introduced. Also when doing this I found a few lines of dead code and functions that I could never get the compiler to generate code to enter. Which makes me wonder if those parts are still needed to begin with. But in general, I agree, this may be more trouble then it's worth. Especially if the original goal was to reduce binary size. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9832#comment:14 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9832: Get rid of PERL dependency of `ghc-split` -------------------------------------+------------------------------------- Reporter: hvr | Owner: Phyx- Type: task | Status: new Priority: high | Milestone: 7.12.1 Component: Driver | Version: Resolution: | Keywords: perl Operating System: Windows | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #8405 | Differential Revisions: -------------------------------------+------------------------------------- Changes (by thomie): * related: => #8405 Comment: See also #8405, where @olsner is trying to get rid of the splitter altogether. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9832#comment:16 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9832: Get rid of PERL dependency of `ghc-split` ---------------------------------+---------------------------------------- Reporter: hvr | Owner: Phyx- Type: task | Status: new Priority: high | Milestone: 8.0.1 Component: Driver | Version: Resolution: | Keywords: perl Operating System: Windows | Architecture: Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #8405 | Differential Rev(s): Wiki Page: | ---------------------------------+---------------------------------------- Comment (by Phyx-): As I understood `--gc-sections` doesn't currently work on Windows due to a binutils bug. So we can't actually use it yet. Also I'm not sure the linker script issue has been solved, it's just been turned off because `--gc-sections` doesn't work anyway. So the PERL script is still there AFAIK, afterwards this issue can probably be about actually removing the script and perl bindist? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9832#comment:19 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9832: Get rid of PERL dependency of `ghc-split` ---------------------------------+---------------------------------------- Reporter: hvr | Owner: Phyx- Type: task | Status: new Priority: normal | Milestone: 8.2.1 Component: Driver | Version: Resolution: | Keywords: perl Operating System: Windows | Architecture: Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #8405 | Differential Rev(s): Wiki Page: | ---------------------------------+---------------------------------------- Changes (by RyanGlScott): * cc: RyanGlScott (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9832#comment:21 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9832: Get rid of PERL dependency of `ghc-split` ---------------------------------+---------------------------------------- Reporter: hvr | Owner: Phyx- Type: task | Status: new Priority: normal | Milestone: 8.2.1 Component: Driver | Version: Resolution: | Keywords: perl Operating System: Windows | Architecture: Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #8405 | Differential Rev(s): Wiki Page: | ---------------------------------+---------------------------------------- Comment (by Phyx-): I think we might just deprecate `ghc-split` as a whole for Windows no @bgamari? With the updated toolchain we should be able to get `function- sections` working on Windows as well. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9832#comment:23 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9832: Get rid of PERL dependency of `ghc-split` ---------------------------------+---------------------------------------- Reporter: hvr | Owner: Phyx- Type: task | Status: new Priority: normal | Milestone: 8.2.1 Component: Driver | Version: Resolution: | Keywords: perl Operating System: Windows | Architecture: Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #8405 | Differential Rev(s): Wiki Page: | ---------------------------------+---------------------------------------- Comment (by bgamari): Yes, I think killing off `ghc-split` would be a step in the right direction. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9832#comment:25 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9832: Get rid of PERL dependency of `ghc-split` ---------------------------------+---------------------------------------- Reporter: hvr | Owner: Phyx- Type: task | Status: new Priority: normal | Milestone: 8.2.1 Component: Driver | Version: Resolution: | Keywords: perl Operating System: Windows | Architecture: Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #8405 | Differential Rev(s): Wiki Page: | ---------------------------------+---------------------------------------- Comment (by Phyx-): Indeed. I'm working on the PE support. There is some experimental code in binutils but this is virtually untested code. It was added by copying the elf implementation and modifying it. The ones who implemented it did not have access to a Windows machine apparently so it was committed as experimental so people could provide feedback. But very little useful feedback was provided afaik. tl;dr: We will probably be the ones putting this through extensive testing. And I do expect some issues. So I have reservations dropping `split-objs` for `8.2` unless we get this tested enough. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9832#comment:27 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

Replying to [comment:24 dobenour]:

...

I think we can deprecate it on other systems too, except (possibly) OS X (which I believe bundles Perl with the OS).

I think GHC on OS X doesn't even support `-split-objs`, rather it uses

#9832: Get rid of PERL dependency of `ghc-split` ---------------------------------+---------------------------------------- Reporter: hvr | Owner: Phyx- Type: task | Status: new Priority: normal | Milestone: 8.2.1 Component: Driver | Version: Resolution: | Keywords: perl Operating System: Windows | Architecture: Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #8405 | Differential Rev(s): Wiki Page: | ---------------------------------+---------------------------------------- Comment (by dobenour): Replying to [comment:26 olsner]: the builtin subsections and `-dead_strip` linker flag, so that wouldn't be a blocker.

It's just Windows support for split sections that we might want to wait

for before we start ripping stuff out. In that case, the Darwin support in the splitter is dead code. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9832#comment:29 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

Replying to [comment:24 dobenour]:

...

I think we can deprecate it on other systems too, except (possibly) OS X (which I believe bundles Perl with the OS).

I think GHC on OS X doesn't even support `-split-objs`, rather it uses

#9832: Get rid of PERL dependency of `ghc-split` ---------------------------------+---------------------------------------- Reporter: hvr | Owner: dobenour Type: task | Status: new Priority: normal | Milestone: 8.2.1 Component: Driver | Version: Resolution: | Keywords: perl Operating System: Windows | Architecture: Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #8405 | Differential Rev(s): D2768 Wiki Page: | ---------------------------------+---------------------------------------- Comment (by dobenour): Replying to [comment:26 olsner]: the builtin subsections and `-dead_strip` linker flag, so that wouldn't be a blocker.

It's just Windows support for split sections that we might want to wait

for before we start ripping stuff out. GHC does support `-split-objs` on OS X, but I can't think of a good reason to use it. I also can't think of a good reason to use `-split-objs` when `-split-sections` works. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9832#comment:31 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9832: Get rid of PERL dependency of `ghc-split` ---------------------------------+---------------------------------------- Reporter: hvr | Owner: dobenour Type: task | Status: patch Priority: normal | Milestone: 8.2.1 Component: Driver | Version: Resolution: | Keywords: perl Operating System: Windows | Architecture: Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #8405 | Differential Rev(s): Phab:D2768 Wiki Page: | ---------------------------------+---------------------------------------- Changes (by bgamari): * status: new => patch * differential: D2768 => Phab:D2768 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9832#comment:33 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9832: Get rid of PERL dependency of `ghc-split` ---------------------------------+---------------------------------------- Reporter: hvr | Owner: dobenour Type: task | Status: patch Priority: normal | Milestone: 8.6.1 Component: Driver | Version: Resolution: | Keywords: perl Operating System: Windows | Architecture: Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #8405 | Differential Rev(s): Phab:D2768 Wiki Page: | ---------------------------------+---------------------------------------- Changes (by bgamari): * milestone: 8.4.1 => 8.6.1 Comment: As mentioned by Tamar in Phab:D2768, this won't happen for 8.4. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9832#comment:35 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9832: Get rid of PERL dependency of `ghc-split` ---------------------------------+---------------------------------------- Reporter: hvr | Owner: dobenour Type: task | Status: closed Priority: normal | Milestone: 8.10.1 Component: Driver | Version: Resolution: fixed | Keywords: perl Operating System: Windows | Architecture: Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #8405 | Differential Rev(s): Phab:D2768 Wiki Page: | ---------------------------------+---------------------------------------- Changes (by bgamari): * status: patch => closed * resolution: => fixed * milestone: 8.8.1 => 8.10.1 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9832#comment:38 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler