How's this? https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/ExternalInterprete... I don't want to go into too much detail in the wiki, because details are more likely to stay current if they're in Notes in the code. There are already a few Notes (e.g. https://phabricator.haskell.org/diffusion/GHC/browse/master/compiler/ghci/GH...) but if anything is confusing I'll happily add more Notes. Cheers Simon On 22 June 2016 at 11:50, Simon Marlow wrote:

On 22 June 2016 at 11:37, Simon Peyton Jones wrote:

...

I’m ok with this. *It would certainly be great not to support TWO mechanisms indefinitely*.

What are the disadvantages to committing to this path? Would anyone even notice?

Yes, people who are making changes to TH will need to ensure that their changes work with -fexternal-interpreter. In some cases that might mean extra work, e.g. if we do TemplateHaskell/Introspective https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective then potentially the whole of HsSyn needs to be binary serializable. That worries me quite a lot - THSyn is big but tractable, Generic deriving handled the generation of the binary instances easily enough, but HsSyn is another matter entirely.

...

There are a lot of moving parts to the implementation, and I for one am utterly ignorant of how it all works. I would love to see an implementation overview, either somewhere in the code or on a wiki page. Things like:

· How, when, and where in the compiler is the separate process started?

· How do the compiler and server communicate? Unix pipes? Is it the same on Windows and Unix?

· What is serialised, when, and how? For example, GHC has some TH code to run. Do we send a syntax tree? Or compile to bytecode and send that? Or what?

· How are external references managed. E.g. if the code to be run refers to ‘map’ I’m sure we don’t serialise the code for ‘map’.

I’m sure there is a lot more. E.g the [wiki:RemoteGHCi wiki page] refers to “a library implementing a message type…” but I don’t know what that library is called or where it lives.

Yes, we should really have a page in the commentary with an overview of the pieces and the main implementation strategy. I'll write one.

Cheers Simon

...

Thanks

Simon

*From:* ghc-devs [mailto:ghc-devs-bounces@haskell.org] *On Behalf Of *Simon Marlow *Sent:* 22 June 2016 09:51 *To:* ghc-devs@haskell.org *Subject:* Require -fexternal-interpreter support for future TH changes?

*Background*

A few months ago I added -fexternal-interpreter to GHC:

- docs: http://downloads.haskell.org/~ghc/latest/docs/html/users_guide/ghci.html#ghc... https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fdownloads.haskell.org%2f~ghc%2flatest%2fdocs%2fhtml%2fusers_guide%2fghci.html%23ghc-flag--fexternal-interpreter&data=01%7C01%7Csimonpj%40064d.mgd.microsoft.com%7C5ca29759b5c9483e03fb08d39a7a60ea%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=6t6FA3nUkP5FOi%2fqCBSr3GxMH2rwWNk89je1qBH9GfI%3d - wiki, rationale: https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi

When -fexternal-interpreter is used, GHC runs interpreted code in a separate subprocess, and communicates with it using binary messages over a pipe.

-fexternal-interpreter currently implements all of TH, quasi-quoting, annotations, and all the GHCi features except for some features of the debugger. It is also now implemented on Windows, thanks to Tamar Christina.

*Proposal*

I'd like to propose that going forward we commit to maintaining full support for -fexternal-interpreter, with a view to making it the default.

Why?

- -fexternal-interpreter will be a prerequisite for GHCJS support, so maintaining full support for TH in -fexternal-interpreter will ensure that everything that works with GHC works with GHCJS. - We will be able to make simplifications in GHC and the build system once -fexternal-interpreter is the default, because when compiling with -prof or -dynamic we won't have to compile things twice any more. - Ultimately we don't want to have two ways of doing everything, because that's harder to maintain.

How?

- I'll make all the TH and quasi-quoting tests run with and without -fexternal-interpreter, so it will break validate if one of these fails.

*Why now?*

There are some TH changes in the pipeline that will need special attention to work with -fexternal-interpreter. e.g. https://phabricator.haskell.org/D2286 and https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective, so I'd like to raise it now so we can keep the issue in mind.

Cheers

Simon

It seems there are nice advantages to this approach, and so I'm fine requiring support from TH. But here is a comment I just posted on D2286 requesting more comments in the code: --- I've just spent some time staring at GHCi/TH.hs and friends, and it's starting to make sense. Would it be possible to add some comments/Notes there? It would be great to know what is run on the client (GHC, right?) and what on the server (TH, right?). Also, why do we need a RemoteRef of an IORef? This, at first, strikes me as odd. I'm sure it makes great sense once I understand all the other moving parts, but it would be helpful to have a primer. I also find it a bit confusing that the Message type contains both messages to the server and messages from the server. For example, I would expect StartTH and Reify to live in different datatypes, because the two should never mix. Or am I horribly mistaken? --- Before we require -fexternal-interpreter (which seems to be the endgoal here), has anyone looked at performance implications? All the marshaling and unmarshaling has to take a toll. Richard On Jun 22, 2016, at 8:27 AM, Simon Peyton Jones via ghc-devs wrote:

It’s a great start, thanks

Simon

From: Simon Marlow [mailto:marlowsd@gmail.com] Sent: 22 June 2016 12:32 To: Simon Peyton Jones Cc: ghc-devs@haskell.org Subject: Re: Require -fexternal-interpreter support for future TH changes?

How's this? https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/ExternalInterprete...

I don't want to go into too much detail in the wiki, because details are more likely to stay current if they're in Notes in the code. There are already a few Notes (e.g.https://phabricator.haskell.org/diffusion/GHC/browse/master/compiler/ghci/GH...) but if anything is confusing I'll happily add more Notes.

Cheers

Simon

On 22 June 2016 at 11:50, Simon Marlow wrote:

On 22 June 2016 at 11:37, Simon Peyton Jones wrote:

I’m ok with this. It would certainly be great not to support TWO mechanisms indefinitely.

What are the disadvantages to committing to this path? Would anyone even notice?

Yes, people who are making changes to TH will need to ensure that their changes work with -fexternal-interpreter. In some cases that might mean extra work, e.g. if we do TemplateHaskell/Introspective then potentially the whole of HsSyn needs to be binary serializable. That worries me quite a lot - THSyn is big but tractable, Generic deriving handled the generation of the binary instances easily enough, but HsSyn is another matter entirely.

There are a lot of moving parts to the implementation, and I for one am utterly ignorant of how it all works. I would love to see an implementation overview, either somewhere in the code or on a wiki page. Things like: · How, when, and where in the compiler is the separate process started?

· How do the compiler and server communicate? Unix pipes? Is it the same on Windows and Unix?

· What is serialised, when, and how? For example, GHC has some TH code to run. Do we send a syntax tree? Or compile to bytecode and send that? Or what?

· How are external references managed. E.g. if the code to be run refers to ‘map’ I’m sure we don’t serialise the code for ‘map’.

I’m sure there is a lot more. E.g the [wiki:RemoteGHCi wiki page] refers to “a library implementing a message type…” but I don’t know what that library is called or where it lives.

Yes, we should really have a page in the commentary with an overview of the pieces and the main implementation strategy. I'll write one.

Cheers Simon

Thanks

Simon

From: ghc-devs [mailto:ghc-devs-bounces@haskell.org] On Behalf Of Simon Marlow Sent: 22 June 2016 09:51 To: ghc-devs@haskell.org Subject: Require -fexternal-interpreter support for future TH changes?

Background

A few months ago I added -fexternal-interpreter to GHC:

docs: http://downloads.haskell.org/~ghc/latest/docs/html/users_guide/ghci.html#ghc... wiki, rationale: https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi When -fexternal-interpreter is used, GHC runs interpreted code in a separate subprocess, and communicates with it using binary messages over a pipe.

-fexternal-interpreter currently implements all of TH, quasi-quoting, annotations, and all the GHCi features except for some features of the debugger. It is also now implemented on Windows, thanks to Tamar Christina.

Proposal

I'd like to propose that going forward we commit to maintaining full support for -fexternal-interpreter, with a view to making it the default.

Why? -fexternal-interpreter will be a prerequisite for GHCJS support, so maintaining full support for TH in -fexternal-interpreter will ensure that everything that works with GHC works with GHCJS. We will be able to make simplifications in GHC and the build system once -fexternal-interpreter is the default, because when compiling with -prof or -dynamic we won't have to compile things twice any more. Ultimately we don't want to have two ways of doing everything, because that's harder to maintain. How? I'll make all the TH and quasi-quoting tests run with and without -fexternal-interpreter, so it will break validate if one of these fails. Why now?

There are some TH changes in the pipeline that will need special attention to work with -fexternal-interpreter. e.g.https://phabricator.haskell.org/D2286 and https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective, so I'd like to raise it now so we can keep the issue in mind.

Cheers

Simon

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

On Jun 22, 2016, at 10:38 AM, Simon Marlow wrote:

Thanks for the feedback Richard. It's helpful to know which parts of the implementation you found non-obvious. I will write some more Notes, but first:

I think what would help most is a place that describes the little state machine that's going on here: the startTH / runTH / finishTH cycle, with possible callbacks in between. All RPC calls are blocking, yes? That certainly simplifies the matter. In the state machine, I think it would help to see clearly what causes the state transition (e.g. splice encountered during renaming, reify call made during splicing) and where the code is located (both source file/function and client vs server). There's just a bunch of moving parts here spread across a number of files and two processes.

* On why we need a RemoteRef of an IORef - because the IORef lives on the server, so GHC has a RemoteRef to it. Did you see Note [External GHCi pointers] in compiler/ghci/GHCi.hs? Does that help?

Somewhat. So the server is sending a pointer (the IORef) to the client? Presumably this pointer is only usable on the server, so the client just ends up sending it back at some point. From the client's point of view, it's just getting an opaque key that only the server can decode. Perhaps this would all become clearer with the overall map.

Should I xref it from somewhere else?

Perhaps from libraries/ghci/GHCi/RemoteTypes.hs, on the declaration of RemoteRef. Thanks! Richard

One other thing: On 22 June 2016 at 14:36, Richard Eisenberg wrote:

It seems there are nice advantages to this approach, and so I'm fine requiring support from TH. But here is a comment I just posted on D2286 requesting more comments in the code:

--- I've just spent some time staring at GHCi/TH.hs and friends, and it's starting to make sense. Would it be possible to add some comments/Notes there? It would be great to know what is run on the client (GHC, right?) and what on the server (TH, right?). Also, why do we need a RemoteRef of an IORef? This, at first, strikes me as odd. I'm sure it makes great sense once I understand all the other moving parts, but it would be helpful to have a primer. I also find it a bit confusing that the Message type contains both messages to the server and messages from the server. For example, I would expect StartTH and Reify to live in different datatypes, because the two should never mix. Or am I horribly mistaken? ---

Before we require -fexternal-interpreter (which seems to be the endgoal here), has anyone looked at performance implications? All the marshaling and unmarshaling has to take a toll.

Yes, I've looked at performance and done a fair bit of optimisation of the current implementation, but mainly for GHCi, not so much for TH yet. For GHCi I think it was <10% of compile time, mainly in shipping the byte code over the pipe. I'll need to do some measurements of TH performance before making it the default. We can probably claw back quite a lot of perf by switching from binary to something else (binary-cbor or store). Cheers Simon

Richard

On Jun 22, 2016, at 8:27 AM, Simon Peyton Jones via ghc-devs < ghc-devs@haskell.org> wrote:

It’s a great start, thanks

Simon

*From:* Simon Marlow [mailto:marlowsd@gmail.com] *Sent:* 22 June 2016 12:32 *To:* Simon Peyton Jones *Cc:* ghc-devs@haskell.org *Subject:* Re: Require -fexternal-interpreter support for future TH changes?

How's this? https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/ExternalInterprete...

I don't want to go into too much detail in the wiki, because details are more likely to stay current if they're in Notes in the code. There are already a few Notes (e.g. https://phabricator.haskell.org/diffusion/GHC/browse/master/compiler/ghci/GH...) but if anything is confusing I'll happily add more Notes.

Cheers

Simon

On 22 June 2016 at 11:50, Simon Marlow wrote:

On 22 June 2016 at 11:37, Simon Peyton Jones wrote:

I’m ok with this. *It would certainly be great not to support TWO mechanisms indefinitely*.

What are the disadvantages to committing to this path? Would anyone even notice?

Yes, people who are making changes to TH will need to ensure that their changes work with -fexternal-interpreter. In some cases that might mean extra work, e.g. if we do TemplateHaskell/Introspective https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective then potentially the whole of HsSyn needs to be binary serializable. That worries me quite a lot - THSyn is big but tractable, Generic deriving handled the generation of the binary instances easily enough, but HsSyn is another matter entirely.

There are a lot of moving parts to the implementation, and I for one am utterly ignorant of how it all works. I would love to see an implementation overview, either somewhere in the code or on a wiki page. Things like:

· How, when, and where in the compiler is the separate process started?

· How do the compiler and server communicate? Unix pipes? Is it the same on Windows and Unix?

· What is serialised, when, and how? For example, GHC has some TH code to run. Do we send a syntax tree? Or compile to bytecode and send that? Or what?

· How are external references managed. E.g. if the code to be run refers to ‘map’ I’m sure we don’t serialise the code for ‘map’. I’m sure there is a lot more. E.g the [wiki:RemoteGHCi wiki page] refers to “a library implementing a message type…” but I don’t know what that library is called or where it lives.

Yes, we should really have a page in the commentary with an overview of the pieces and the main implementation strategy. I'll write one. Cheers Simon

Thanks

Simon

*From:* ghc-devs [mailto:ghc-devs-bounces@haskell.org] *On Behalf Of *Simon Marlow *Sent:* 22 June 2016 09:51 *To:* ghc-devs@haskell.org *Subject:* Require -fexternal-interpreter support for future TH changes?

*Background*

A few months ago I added -fexternal-interpreter to GHC:

- docs: http://downloads.haskell.org/~ghc/latest/docs/html/users_guide/ghci.html#ghc... https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fdownloads.haskell.org%2f~ghc%2flatest%2fdocs%2fhtml%2fusers_guide%2fghci.html%23ghc-flag--fexternal-interpreter&data=01%7C01%7Csimonpj%40064d.mgd.microsoft.com%7C5ca29759b5c9483e03fb08d39a7a60ea%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=6t6FA3nUkP5FOi%2fqCBSr3GxMH2rwWNk89je1qBH9GfI%3d - wiki, rationale: https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi

When -fexternal-interpreter is used, GHC runs interpreted code in a separate subprocess, and communicates with it using binary messages over a pipe.

-fexternal-interpreter currently implements all of TH, quasi-quoting, annotations, and all the GHCi features except for some features of the debugger. It is also now implemented on Windows, thanks to Tamar Christina.

*Proposal*

I'd like to propose that going forward we commit to maintaining full support for -fexternal-interpreter, with a view to making it the default. Why?

- -fexternal-interpreter will be a prerequisite for GHCJS support, so maintaining full support for TH in -fexternal-interpreter will ensure that everything that works with GHC works with GHCJS. - We will be able to make simplifications in GHC and the build system once -fexternal-interpreter is the default, because when compiling with -prof or -dynamic we won't have to compile things twice any more. - Ultimately we don't want to have two ways of doing everything, because that's harder to maintain.

How?

- I'll make all the TH and quasi-quoting tests run with and without -fexternal-interpreter, so it will break validate if one of these fails.

*Why now?*

There are some TH changes in the pipeline that will need special attention to work with -fexternal-interpreter. e.g. https://phabricator.haskell.org/D2286 and https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective, so I'd like to raise it now so we can keep the issue in mind.

Cheers

Simon

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Yes - I wasn't planning on dropping support for the in-process interpreter, for that reason. I haven't thought about plugins very much, but aren't they always compiled code? We don't need an in-process interpreter for those, but we do need in-process linking. Maybe that doesn't simplify things much. Cheers Simon On 22 June 2016 at 15:40, Edward Z. Yang wrote:

Hello Simon,

I have no exception to having it be default and dropping the special case support for building profiled/dynamic so that TH works. But I don't think support for loading code in-process for GHC should be dropped, c.f. Manuel's email https://mail.haskell.org/pipermail/ghc-devs/2015-November/010491.html and also the necessity to run code in-process for typechecking plugins, etc.

Edward

Excerpts from Simon Marlow's message of 2016-06-22 04:51:12 -0400:

...

*Background*

A few months ago I added -fexternal-interpreter to GHC:

- docs:

...

- wiki, rationale: https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi

When -fexternal-interpreter is used, GHC runs interpreted code in a separate subprocess, and communicates with it using binary messages over a pipe.

-fexternal-interpreter currently implements all of TH, quasi-quoting, annotations, and all the GHCi features except for some features of the debugger. It is also now implemented on Windows, thanks to Tamar Christina.

*Proposal*

I'd like to propose that going forward we commit to maintaining full support for -fexternal-interpreter, with a view to making it the default.

Why?

- -fexternal-interpreter will be a prerequisite for GHCJS support, so maintaining full support for TH in -fexternal-interpreter will ensure

http://downloads.haskell.org/~ghc/latest/docs/html/users_guide/ghci.html#ghc... that

...

everything that works with GHC works with GHCJS. - We will be able to make simplifications in GHC and the build system once -fexternal-interpreter is the default, because when compiling with -prof or -dynamic we won't have to compile things twice any more. - Ultimately we don't want to have two ways of doing everything, because that's harder to maintain.

How?

- I'll make all the TH and quasi-quoting tests run with and without -fexternal-interpreter, so it will break validate if one of these fails.

*Why now?*

There are some TH changes in the pipeline that will need special attention to work with -fexternal-interpreter. e.g. https://phabricator.haskell.org/D2286 and https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective, so I'd like to raise it now so we can keep the issue in mind.

Cheers

Simon