Luite's "out of process TH" work is amazing. If someone had suggested it to me I'd have said "it sounds entirely impractical", but he showed that I would have been quite wrong. I'm open to fixing up GHC, preferably via some plugin variations, to support this. (I would prefer not to add a whole batch of new dependencies.) If a little sub-group would like to: * Articulate their preferred design (from a user point of view) on a wiki page * Sketch the implementation plan * Build a patch and get consensus on all of that, let's go for it. I guess that means at least Luite and Moritz; I'm not sure who else is keen here. The ball is in your court. Thank you! Simon | -----Original Message----- | From: ghc-devs [mailto:ghc-devs-bounces@haskell.org] On Behalf Of | Moritz Angermann | Sent: 03 December 2014 21:24 | To: ghc-devs | Subject: Out Of Process Template Haskell | | Hi, | | as you may or may not know, template haskell is not available to cross | compilers. And therefore cross compiler are devoid of an important | feature of haskell. | | We are currently building ghc cross compiler as stage1 compiler with | a foreign target. | This means that we use a stage0 compiler (the one that is used to | build ghc), to build the ghc that will run on the host and produce | binaries for the target (e.g. running on x86_64, building for arm). | Template Haskell and GHCi are enabled only in stage2. To build a | stage2 compiler we use the stage1 compiler. Hence the stage2 compiler | runs on the target of the stage1 compiler. A motivating example could | be ghc running on an Intel based Mac OS X targeting iOS (ARM). A | *theoretical* stage2 compiler could have GHCi and Template Haskell, | but would run on the iOS Device, which is neat but probably not a | practical solution. Hence a compiler running on a more powerful host, | producing executables for a foreign target can be a very good solution | when the host is much more powerful. (Ref. [1]). In [2] we can read | the following: | | > Stage 1 does not support interactive execution (GHCi) and Template | > Haskell. The reason being that when running byte code we must | > dynamically link the packages, and only in stage 2 and later can we | > guarantee that the packages we dynamically link are compatible with | those that GHC was built against (because they are the very same | packages). | | One of the prominent cross compilers is ghcjs, which admittedly is | build a little different than the above cross compiler would be. | ghcjs shares the same template haskell issue though. ghcjs--to my | knowledge--followed a few different routes to solve this. And finally | came up with the *out-of-process-template-haskell* solution. | | The Out Of Process Template Haskell approach works by having a | *runner* on the target that waits for the compiler to send the | template haskell splices, and compiles those splices on the target, | whilte querying the *master* ghc for potential lookups during the | splice compilation. The result is then shipped back to the host to | integrate into the produced binary for the target. Hence allowing the | more powerful host to do the bulk of the compilation and using the | *runner* as a *slave compiler* on the target. | | | .-- host -. | .- target -. | | | -- ( encounters splice and sends it to runner ) ---> | | | | | | | | | | | ghc | .<---- ( compiles and queries ghc for lookups ) ----. | | runner | | | master | '----------- ( responds to to queries ) ----------->' | | slave | | | | | | | | | | <- ( receives the compiled splice from the runner ) - | | | | '---------' | '----------' | | Fig. 1: ghc - th runner communication. | | | I have implemented the *out-of-process-template-haskell* approach | with the help from luite for ghc as a plugin for stage2 *non*-cross | compiler[3] using dynamic libraries and a tcp transport by adapting | the code from ghcjs to run on the host. | Obviously for a regular stage2 ghc, which has GHCi and Template | Haskell, this doesn't really buy anyone anything yet. Also a plugin | is impossible to load into a stage1 compiler yet, as the plugin code | is wrapped in #ifdef GHCI sections. | | SIDENOTE: To allow the out-of-process-template-haskell, the plugin | api had to be | adapted minimally to allow plugins to install hooks. | Patches for 7.8[4] | and 7.10[5] are readily available. | | | Clearly the *out-of-process-template-haskell* could be integrated | into the ghc tree by hard wiring it in, where the plugin did the | wiring ad-hoc previous. This would imply that ghc would have to | integrate all dependencies which *out-of-process-template-haskell* | would bring with it. | | There are now three options discussed on #ghc so far: | a) Full integration in ghc, as described above. | b) Enabling the plugin interface in stage1 compiler [See Note 1 for | details]. | This is probably only interesting for cross-compiler, where stage1 | is the final | stage on the host. [See Note 2 for an example] | c) As proposed by luite: integrate a thin layer in ghc, that uses | pipes to communicate | with an external program on the same host that ghc is running on. | That external | program will then communicate with the *runner* on the target. | | The ultimate goal would be a multi-target compiler, that could | produce a stage2 compiler that runs on the host but can produce | binaries for multiple targets of which the host it's running on is | one. | | I am in favor of extending the plugin api and making it available to | stage1 compiler, because I see much potential in the plugin api. | Which--with the noted patches [4][5]-- allows you to install hooks, | and hence allow you to hook into the complete compiler pipeline and | potential other parts as well. It also makes developing plugins | easier, because they can be developed outside of the ghc source tree. | And only be used with cross compilers if we can load plugins in stage1 | compilers. | | Cheers, | Moritz | | | [1]: https://ghc.haskell.org/trac/ghc/wiki/CrossCompilation | [2]: | https://ghc.haskell.org/trac/ghc/wiki/Building/Architecture/Idiom/Stag | es | [3]: https://github.com/angerman/oopth | [4]: https://gist.github.com/angerman/7db11c24f8935c73fcf5 | [5]: https://phabricator.haskell.org/D535 | Note 1: where the plugin will have to be compiled with a stage0 | compiler, and may even | require a stage0 compiler that is as new as the stage1 | compiler one wants to build, | because the compiler being built has to be binary compatible | with the plugin. | Note 2: An example would be building 7.10 on the host for the host | with stage0=ghc7.8, | stage1=ghc7.10(built with ghc7.8), stage2=ghc7.10(built with | stage1), *and then* | building a ghc cross compiler for arm with stage0'=stage2 and | stage1'(ghc7.10 targeting | ARM built with ghc7.10 on the host). And allowing this newly | built cross compiler | (stage1') to accept -fplugin for plugins built with stage2 | (the same that built stage1') | _______________________________________________ | ghc-devs mailing list | ghc-devs@haskell.org | http://www.haskell.org/mailman/listinfo/ghc-devs