Including remote-iserv upstream?

Hi Simon, devs, As part of my work to get TH working when cross-compiling to iOS, I've developed remote-iserv [1] (not yet on hackage), a set of libraries for letting GHC communicate with an external interpreter that may be on another machine. So far, there are only three additions of note on top of what the ghci library offers: 1. The remote-iserv protocol has facilities for the host sending archives and object files the target doesn't have (dynlibs not yet implemented but there's no reason they can't be). This works by having the server send back a Bool after a loadObj or loadArchive indicating whether it needs the object sent, and then just reading it off the Pipe. 2. The remote-iserv lib abstracts over how the Pipe it communicates over is obtained. One could imagine e.g. an ssh-based implementation that just uses stdin and stdout* for the communication, the implementation I've actually tested on is a TCP server advertised over bonjour. 3. There is a protocol version included to address forwards compatibility concerns. As the library currently stands, it works for my use case. However, there would be a number of benefits if it were included with ghc (and used for local iserv as well): 1. Reduced code duplication (the server side copies iserv/src/Main.hs pretty heavily) 2. Reduced overhead keeping up to date with iserv protocol changes 3. No need for an extra client-side process, GHC can just open the Pipe itself 4. Proper library distribution in the cross-compiling case. The client needs to be linked with the ghci lib built by the stage0 compiler, as it runs on the build machine, while the server needs to be linked with the ghci lib built by the stage1 compiler. With a distribution created by 'make install', we only get the ghci lib for the target. Currently, I'm working around this by just using the ghci lib of the bootstrap compiler, which in my case is built from the same source checkout, but of course this isn't correct. If these libs were upstream, we'd only need one version of the client lib exposed and one version of the server lib exposed and could have them be for the build machine and the target, respectively 5. Better haskell hackers than I invested in the code ;) Thoughts on this? Would this be welcome upstream in some form? Thanks, Shea * Note that, in the general case, having the server process's stdio be the same as the compiler's (as we have in the local-iserv case) is not possible. Future work includes adding something to the protocol to allow forwarding stdio over the protocol pipe, to make GHCi usable without watching the *server*'s console. [1]: https://github.com/obsidiansystems/remote-iserv

As a matter of interest, are you making use of `createIservProcessHook` which allows you to set the FDs to be used when communicating with the iserve process?

No, as it currently exists it has to create a ProcessHandle, and I have
to layer some stuff on top of the iserv protocol anyway, so it wouldn't
really help much. I just use -pgmi to point to the client executable.
~Shea
Alan & Kim Zimmerman
As a matter of interest, are you making use of `createIservProcessHook` which allows you to set the FDs to be used when communicating with the iserve process? _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Absolutely, let's get this code upstream. Just put it up on Phabricator
and I'll be happy to review.
I recall that we wanted to split up the ghci lib into modules that are
compiled with stage0 (the client) and modules compiled with stage1 (the
server). Is that a part of your plans? I think it would be a good cleanup.
Cheers
Simon
On 14 January 2017 at 15:34, Shea Levy
Hi Simon, devs,
As part of my work to get TH working when cross-compiling to iOS, I've developed remote-iserv [1] (not yet on hackage), a set of libraries for letting GHC communicate with an external interpreter that may be on another machine. So far, there are only three additions of note on top of what the ghci library offers:
1. The remote-iserv protocol has facilities for the host sending archives and object files the target doesn't have (dynlibs not yet implemented but there's no reason they can't be). This works by having the server send back a Bool after a loadObj or loadArchive indicating whether it needs the object sent, and then just reading it off the Pipe. 2. The remote-iserv lib abstracts over how the Pipe it communicates over is obtained. One could imagine e.g. an ssh-based implementation that just uses stdin and stdout* for the communication, the implementation I've actually tested on is a TCP server advertised over bonjour. 3. There is a protocol version included to address forwards compatibility concerns.
As the library currently stands, it works for my use case. However, there would be a number of benefits if it were included with ghc (and used for local iserv as well):
1. Reduced code duplication (the server side copies iserv/src/Main.hs pretty heavily) 2. Reduced overhead keeping up to date with iserv protocol changes 3. No need for an extra client-side process, GHC can just open the Pipe itself 4. Proper library distribution in the cross-compiling case. The client needs to be linked with the ghci lib built by the stage0 compiler, as it runs on the build machine, while the server needs to be linked with the ghci lib built by the stage1 compiler. With a distribution created by 'make install', we only get the ghci lib for the target. Currently, I'm working around this by just using the ghci lib of the bootstrap compiler, which in my case is built from the same source checkout, but of course this isn't correct. If these libs were upstream, we'd only need one version of the client lib exposed and one version of the server lib exposed and could have them be for the build machine and the target, respectively 5. Better haskell hackers than I invested in the code ;)
Thoughts on this? Would this be welcome upstream in some form?
Thanks, Shea
* Note that, in the general case, having the server process's stdio be the same as the compiler's (as we have in the local-iserv case) is not possible. Future work includes adding something to the protocol to allow forwarding stdio over the protocol pipe, to make GHCi usable without watching the *server*'s console.

OK, will do, thanks!
Simon Marlow
Absolutely, let's get this code upstream. Just put it up on Phabricator and I'll be happy to review.
I recall that we wanted to split up the ghci lib into modules that are compiled with stage0 (the client) and modules compiled with stage1 (the server). Is that a part of your plans? I think it would be a good cleanup.
Cheers Simon
On 14 January 2017 at 15:34, Shea Levy
wrote: Hi Simon, devs,
As part of my work to get TH working when cross-compiling to iOS, I've developed remote-iserv [1] (not yet on hackage), a set of libraries for letting GHC communicate with an external interpreter that may be on another machine. So far, there are only three additions of note on top of what the ghci library offers:
1. The remote-iserv protocol has facilities for the host sending archives and object files the target doesn't have (dynlibs not yet implemented but there's no reason they can't be). This works by having the server send back a Bool after a loadObj or loadArchive indicating whether it needs the object sent, and then just reading it off the Pipe. 2. The remote-iserv lib abstracts over how the Pipe it communicates over is obtained. One could imagine e.g. an ssh-based implementation that just uses stdin and stdout* for the communication, the implementation I've actually tested on is a TCP server advertised over bonjour. 3. There is a protocol version included to address forwards compatibility concerns.
As the library currently stands, it works for my use case. However, there would be a number of benefits if it were included with ghc (and used for local iserv as well):
1. Reduced code duplication (the server side copies iserv/src/Main.hs pretty heavily) 2. Reduced overhead keeping up to date with iserv protocol changes 3. No need for an extra client-side process, GHC can just open the Pipe itself 4. Proper library distribution in the cross-compiling case. The client needs to be linked with the ghci lib built by the stage0 compiler, as it runs on the build machine, while the server needs to be linked with the ghci lib built by the stage1 compiler. With a distribution created by 'make install', we only get the ghci lib for the target. Currently, I'm working around this by just using the ghci lib of the bootstrap compiler, which in my case is built from the same source checkout, but of course this isn't correct. If these libs were upstream, we'd only need one version of the client lib exposed and one version of the server lib exposed and could have them be for the build machine and the target, respectively 5. Better haskell hackers than I invested in the code ;)
Thoughts on this? Would this be welcome upstream in some form?
Thanks, Shea
* Note that, in the general case, having the server process's stdio be the same as the compiler's (as we have in the local-iserv case) is not possible. Future work includes adding something to the protocol to allow forwarding stdio over the protocol pipe, to make GHCi usable without watching the *server*'s console.
participants (3)
-
Alan & Kim Zimmerman
-
Shea Levy
-
Simon Marlow