Re: Require -fexternal-interpreter support for future TH changes?

How's this? https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/ExternalInterprete... I don't want to go into too much detail in the wiki, because details are more likely to stay current if they're in Notes in the code. There are already a few Notes (e.g. https://phabricator.haskell.org/diffusion/GHC/browse/master/compiler/ghci/GH...) but if anything is confusing I'll happily add more Notes. Cheers Simon On 22 June 2016 at 11:50, Simon Marlow wrote:

On 22 June 2016 at 11:37, Simon Peyton Jones wrote:

...

I’m ok with this. *It would certainly be great not to support TWO mechanisms indefinitely*.

What are the disadvantages to committing to this path? Would anyone even notice?

Yes, people who are making changes to TH will need to ensure that their changes work with -fexternal-interpreter. In some cases that might mean extra work, e.g. if we do TemplateHaskell/Introspective https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective then potentially the whole of HsSyn needs to be binary serializable. That worries me quite a lot - THSyn is big but tractable, Generic deriving handled the generation of the binary instances easily enough, but HsSyn is another matter entirely.

...

There are a lot of moving parts to the implementation, and I for one am utterly ignorant of how it all works. I would love to see an implementation overview, either somewhere in the code or on a wiki page. Things like:

· How, when, and where in the compiler is the separate process started?

· How do the compiler and server communicate? Unix pipes? Is it the same on Windows and Unix?

· What is serialised, when, and how? For example, GHC has some TH code to run. Do we send a syntax tree? Or compile to bytecode and send that? Or what?

· How are external references managed. E.g. if the code to be run refers to ‘map’ I’m sure we don’t serialise the code for ‘map’.

I’m sure there is a lot more. E.g the [wiki:RemoteGHCi wiki page] refers to “a library implementing a message type…” but I don’t know what that library is called or where it lives.

Yes, we should really have a page in the commentary with an overview of the pieces and the main implementation strategy. I'll write one.

Cheers Simon

...

Thanks

Simon

*From:* ghc-devs [mailto:ghc-devs-bounces@haskell.org] *On Behalf Of *Simon Marlow *Sent:* 22 June 2016 09:51 *To:* ghc-devs@haskell.org *Subject:* Require -fexternal-interpreter support for future TH changes?

*Background*

A few months ago I added -fexternal-interpreter to GHC:

- docs: http://downloads.haskell.org/~ghc/latest/docs/html/users_guide/ghci.html#ghc... https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fdownloads.haskell.org%2f~ghc%2flatest%2fdocs%2fhtml%2fusers_guide%2fghci.html%23ghc-flag--fexternal-interpreter&data=01%7C01%7Csimonpj%40064d.mgd.microsoft.com%7C5ca29759b5c9483e03fb08d39a7a60ea%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=6t6FA3nUkP5FOi%2fqCBSr3GxMH2rwWNk89je1qBH9GfI%3d - wiki, rationale: https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi

When -fexternal-interpreter is used, GHC runs interpreted code in a separate subprocess, and communicates with it using binary messages over a pipe.

-fexternal-interpreter currently implements all of TH, quasi-quoting, annotations, and all the GHCi features except for some features of the debugger. It is also now implemented on Windows, thanks to Tamar Christina.

*Proposal*

I'd like to propose that going forward we commit to maintaining full support for -fexternal-interpreter, with a view to making it the default.

Why?

- -fexternal-interpreter will be a prerequisite for GHCJS support, so maintaining full support for TH in -fexternal-interpreter will ensure that everything that works with GHC works with GHCJS. - We will be able to make simplifications in GHC and the build system once -fexternal-interpreter is the default, because when compiling with -prof or -dynamic we won't have to compile things twice any more. - Ultimately we don't want to have two ways of doing everything, because that's harder to maintain.

How?

- I'll make all the TH and quasi-quoting tests run with and without -fexternal-interpreter, so it will break validate if one of these fails.

*Why now?*

There are some TH changes in the pipeline that will need special attention to work with -fexternal-interpreter. e.g. https://phabricator.haskell.org/D2286 and https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective, so I'd like to raise it now so we can keep the issue in mind.

Cheers

Simon