#14298: Let Template Haskell dynamically add something with which to link -------------------------------------+------------------------------------- Reporter: harpocrates | Owner: (none) Type: feature request | Status: new Priority: normal | Milestone: Component: Template Haskell | Version: 8.2.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Description changed by harpocrates: Old description:

As of today, Template Haskell supports emitting foreign files (for C compiler languages) via `addForeignFile`. In doing so, GHC takes on the work of compiling these, linking against them, and then cleaning up any other files.

This makes packages like `inline-c` ([https://hackage.haskell.org/package/inline-c]) possible, where you can write C snippets in quasiquotes and these can interact with your Haskell code. The user doesn't need to pass extra options to GHC (except to enable the right language extensions), and they don't have to see any of the intermediate generated artifacts.

Unfortunately, that breaks down for non C compiler languages. It would be nice for TH to also support directly adding a static library, since then one could

* use TH's `runIO` to generate static libraries by calling out to whatever other compilers * add the contents of those libraries via TH * delete the temporary files created in the process (again using `runIO`) * have GHC statically link against the content (from the second bullet point)

I'm not sure what the API for this could be, but maybe adding a `StaticLibrary` constructor to the `ForeignSrcLang` data type (so one could use `addForeignFile :: ForeignSrcLang -> String -> Q ()`)?

New description: As of today, Template Haskell supports emitting foreign files (for C compiler languages) via `addForeignFile`. In doing so, GHC takes on the work of compiling these, linking them, and then cleaning up any other files. This makes packages like `inline-c` ([https://hackage.haskell.org/package/inline-c]) possible, where you can write C snippets in quasiquotes and these can interact with your Haskell code. The user doesn't need to pass extra options to GHC (except to enable the right language extensions), and they don't have to see any of the intermediate generated artifacts. Unfortunately, that breaks down for non C compiler languages. It would be nice for TH to also support directly adding something to pass to the linker, since then one could * use TH's `runIO` to generate libraries or object files by calling out to whatever other compilers * add those via TH * have GHC statically link against the content (from the second bullet point) I'm not sure what the API for this could be, but maybe 1. add a `LangLinkable` constructor to the `ForeignSrcLang` data type 2. add a `qAddForeignFilePath :: ForeignSrcLang -> FilePath -> m ()` method to `Quasi m` -- -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14298#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#14298: Let Template Haskell dynamically add something with which to link -------------------------------------+------------------------------------- Reporter: harpocrates | Owner: (none) Type: feature request | Status: new Priority: normal | Milestone: Component: Template Haskell | Version: 8.2.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by goldfire): TH is a bit of a gray area. Non-TH language features go through the [https://github.com/ghc-proposals/ghc-proposals ghc-proposals] process, but TH features tend not to. I would recommend posting to ghc-devs. That will reach the people that care deeply about TH and would be able to evaluate this idea. (I care deeply about TH, but I have a very weak understanding of how the FFI works, and so I don't have an informed opinion here.) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14298#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#14298: Let Template Haskell dynamically add something with which to link -------------------------------------+------------------------------------- Reporter: harpocrates | Owner: (none) Type: feature request | Status: new Priority: normal | Milestone: Component: Template Haskell | Version: 8.2.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by harpocrates): Let me reply to those two points in reverse order. :)

It seems like in generate this feature would need support from external tools (e.g. cabal) to work properly.

I'm not sure that is the case. TH already has support for adding in C source files, compiling them, and linking them. This feature request is requesting a way to hook into ''just'' the linking part of the pipeline that was built in https://phabricator.haskell.org/D3280 (instead of also running a C compiler). I don't think cabal factors into the existing `qAddForeignFile`, so I don't see why it should factor into my proposed `qAddForeignFilePath`.

I'm a bit skeptical of this; what should happen when GHC is invoked in single-shot mode (e.g. `-c`)?

I'd expect to see the same thing that would happen if GHC where invoked in single-shot mode on something involving `addForeignFile`, that is to say: `cannot find object file './Support.dyn_o'` where `Support` is the module containing the TH. That said, my understanding of `-c` is that it shouldn't work on any module that uses TH from another module. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14298#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#14298: Let Template Haskell dynamically add something with which to link -------------------------------------+------------------------------------- Reporter: harpocrates | Owner: (none) Type: feature request | Status: new Priority: normal | Milestone: Component: Template Haskell | Version: 8.2.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari):

That said, my understanding of `-c` is that it shouldn't work on any module that uses TH from another module.

That isn't quite true. TH will work perfectly well in single-shot mode. However, as usual, the user must guarantee that any needed modules are compiled for whatever way GHC will want to load to evaluate splices. For instance, {{{ $ pwd /opt/exp/ghc/ghc-landing/testsuite/tests/th $ ghc -c TH_NestedSplices_Lib.hs -dynamic-too -fforce-recomp $ ghc -c TH_NestedSplices.hs -fforce-recomp }}} It's important that this behavior be preserved since Cabal is slowly moving towards using single-shot mode to improve module-level build parallelism.

About the API, I would prefer: {{{ addForeignObject :: ByteString -> Q () }}}

The interface in comment:6 sounds plausible. The worry that I have with just allowing the user to specify an object file path as in the original proposal is that it makes it seem as though they can safely `addForeignObject` the same object file from multiple modules. However, this is not safe as they will encounter conflicting symbol errors when they attempt to link the final object. Of course, we could in principle document our way out of this issue. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14298#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#14298: Let Template Haskell dynamically add something with which to link -------------------------------------+------------------------------------- Reporter: harpocrates | Owner: (none) Type: feature request | Status: new Priority: normal | Milestone: Component: Template Haskell | Version: 8.2.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by hsyl20):

It's worth noting that `bytestring` isn't currently a dependency of template-haskell though.

Then maybe we should avoid `bytestring` with something like: {{{#!hs addForeignObjectFromMem :: Word -> Ptr () -> Q () }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14298#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#14298: Let Template Haskell dynamically add something with which to link -------------------------------------+------------------------------------- Reporter: harpocrates | Owner: (none) Type: feature request | Status: new Priority: normal | Milestone: Component: Template Haskell | Version: 8.2.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4064 Wiki Page: | -------------------------------------+------------------------------------- Changes (by tmcdonell): * cc: tmcdonell (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14298#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#14298: Let Template Haskell dynamically add something with which to link -------------------------------------+------------------------------------- Reporter: harpocrates | Owner: (none) Type: feature request | Status: new Priority: normal | Milestone: 8.6.1 Component: Template Haskell | Version: 8.2.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4064 Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * milestone: => 8.6.1 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14298#comment:13 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#14298: Let Template Haskell dynamically add something with which to link -------------------------------------+------------------------------------- Reporter: harpocrates | Owner: (none) Type: feature request | Status: new Priority: normal | Milestone: 8.8.1 Component: Template Haskell | Version: 8.2.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4064 Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): Indeed it is. Thanks for noticing! -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14298#comment:16 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler