Cross-compiling Template Haskell via -fexternal-interpreter and IPC

newer
RE: Trees that Grow in the hsSyn...

Alberto Valverde

6 Jul 2016 6 Jul '16

9 a.m.

Hello, I'm trying to put together a GHC 8.0.1 cross-compiler with Template Haskell support. Initially to target Windows (32bits) from a Linux host but a similar procedure should enable to target other platforms too. I'd like to contribute the patches back so I'm asking for advice on how to implement it in order to increase the chances of them being accepted. I've managed to get a working stage 1 cross-compiler with some patches which correctly builds all stage1 libs and GHC + stage 2 compiler and ghc-iserv.exe. However, compiling TH by using wine and ghc-iserver.exe fails because the file descriptor ids that GHC passes as arguments to "wine ghc-iserv.exe" don't make sense in the emulated windows world. I've hacked around this to test the feasibility of the approach by using stdin/stdout instead of creating new pipes and, surprisingly, managed to cross-compile a simple Template Haskell program. I'm considering using a socket for communicating between both processes as a more permanent solution but this would incur in a dependency on the "network" package. Would this be acceptable? Named pipes have also crossed my mind but I'm not sure how well they're supported by wine. Thanks for your attention. Alberto P.S: Code is in the "cross-mingw-hacks" branches of these repositories and is based on the ghc-8.0.1-release tag: https://github.com/albertov/ghc https://github.com/albertov/hsc2hs A Docker image script here that builds it is here: https://github.com/albertov/ghc-cross-compiler-windows-x86

Attachments:

attachment.html (text/html — 2.1 KB)

Show replies by date

Ben Gamari

6 Jul 6 Jul

1:14 p.m.

Explicitly CCing Simon Marlow to ensure he sees this. Alberto Valverde writes:

...

Hello,

I'm trying to put together a GHC 8.0.1 cross-compiler with Template Haskell support. Initially to target Windows (32bits) from a Linux host but a similar procedure should enable to target other platforms too. I'd like to contribute the patches back so I'm asking for advice on how to implement it in order to increase the chances of them being accepted.

I've managed to get a working stage 1 cross-compiler with some patches which correctly builds all stage1 libs and GHC + stage 2 compiler and ghc-iserv.exe. However, compiling TH by using wine and ghc-iserver.exe fails because the file descriptor ids that GHC passes as arguments to "wine ghc-iserv.exe" don't make sense in the emulated windows world.

Ahh, right. Out of curiosity what toolchain are you using to build your stage 1 cross compiler?

...

I've hacked around this to test the feasibility of the approach by using stdin/stdout instead of creating new pipes and, surprisingly, managed to cross-compile a simple Template Haskell program.

Well done!

...

I'm considering using a socket for communicating between both processes as a more permanent solution but this would incur in a dependency on the "network" package. Would this be acceptable?

It would be nice if we could avoid it; GHC depending upon a library has the very unfortunate effect that any user code also needing to link against GHC or one of its libraries now has no choice in the version of that library. We go to some lengths to try to keep the dependency footprint of GHC small for this reason.

...

Named pipes have also crossed my mind but I'm not sure how well they're supported by wine.

It would be great if there were some way we could make this work with named pipes. Not only does it side-step the dependency issue, but it feels like the right way forward. Cheers, - Ben

Alberto Valverde

8:01 p.m.

Ben, Phyx, On Wed, Jul 6, 2016 at 7:14 PM, Ben Gamari wrote:

...

(...) Ahh, right. Out of curiosity what toolchain are you using to build your stage 1 cross compiler?

I'm using an MXE (http://mxe.cc) environment in a Debian Jessie Docker image which has gcc-4.9.3. The build env is here: https://github.com/albertov/ghc-cross-compiler-windows-x86 On Wed, Jul 6, 2016 at 7:14 PM, Ben Gamari wrote:

...

It would be nice if we could avoid it; GHC depending upon a library has the very unfortunate effect that any user code also needing to link against GHC or one of its libraries now has no choice in the version of that library. We go to some lengths to try to keep the dependency footprint of GHC small for this reason.

(...)

...

...
Named pipes have also crossed my mind but I'm not sure how well they're supported by wine.

It would be great if there were some way we could make this work with named pipes. Not only does it side-step the dependency issue, but it feels like the right way forward.

On Wed, Jul 6, 2016 at 8:05 PM, Phyx wrote:

...

Named pipes would solve your issue, Just don't forget to set the proper ACL on the pipes.

It does indeed work! However, I've broken GHC on a Windows host until I implement the equivalent to Posix.createNamedPipe since it seems there's no binding for CreateNamedPipe that I can find in the tree. I'll look into it tomorrow. I've pushed the changes so far to https://github.com/albertov/ghc/tree/cross-external-interpreter. Thanks Alberto

...

Cheers,

- Ben

Moritz Angermann

8:57 p.m.

Great work Alberto, I’m in favor of adding some form of network layer, as there are scenarios where you have to run the th compilation process on a different machine. This would be the case for iOS for example. When I toyed with this ~2years ago, trying to port the out of process th solution from ghcjs, I tried to use GHC’s plugin interface, to load the module that would allow ghc to communicate with the runner on the device. This in principle allows to have more dependencies on the plugin and not force them into ghc. At the same time it requires the installation of some additional hooks into the plugin system. I guess one could also come up with a tiny proxy that pretends to be the iserv endpoint and would forward anything over a network layer; this again could probably work outside of ghc. Cheers, Moritz

...

On Jul 7, 2016, at 8:01 AM, Alberto Valverde wrote:

Ben, Phyx,

On Wed, Jul 6, 2016 at 7:14 PM, Ben Gamari wrote: (...) Ahh, right. Out of curiosity what toolchain are you using to build your stage 1 cross compiler?

I'm using an MXE (http://mxe.cc) environment in a Debian Jessie Docker image which has gcc-4.9.3. The build env is here: https://github.com/albertov/ghc-cross-compiler-windows-x86

On Wed, Jul 6, 2016 at 7:14 PM, Ben Gamari wrote: It would be nice if we could avoid it; GHC depending upon a library has the very unfortunate effect that any user code also needing to link against GHC or one of its libraries now has no choice in the version of that library. We go to some lengths to try to keep the dependency footprint of GHC small for this reason. (...)

...
Named pipes have also crossed my mind but I'm not sure how well they're supported by wine.

It would be great if there were some way we could make this work with named pipes. Not only does it side-step the dependency issue, but it feels like the right way forward.

On Wed, Jul 6, 2016 at 8:05 PM, Phyx wrote: Named pipes would solve your issue, Just don't forget to set the proper ACL on the pipes.

It does indeed work! However, I've broken GHC on a Windows host until I implement the equivalent to Posix.createNamedPipe since it seems there's no binding for CreateNamedPipe that I can find in the tree. I'll look into it tomorrow.

I've pushed the changes so far to https://github.com/albertov/ghc/tree/cross-external-interpreter.

Thanks Alberto

Cheers,

- Ben

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Alberto Valverde

7 Jul 7 Jul

6:36 a.m.

On Thu, Jul 7, 2016 at 2:57 AM, Moritz Angermann wrote:

...

Great work Alberto,

Thanks :)

...

I’m in favor of adding some form of network layer, as there are scenarios where you have to run the th compilation process on a different machine. This would be the case for iOS for example.

When I toyed with this ~2years ago, trying to port the out of process th solution from ghcjs, I tried to use GHC’s plugin interface, to load the module that would allow ghc to communicate with the runner on the device.

This in principle allows to have more dependencies on the plugin and not force them into ghc. At the same time it requires the installation of some additional hooks into the plugin system.

I guess one could also come up with a tiny proxy that pretends to be the iserv endpoint and would forward anything over a network layer; this again could probably work outside of ghc.

I'm pretty sure this can be done by adding networking capabilities to just iserv without adding networking capabilities to GHC itself. However, I'm not sure if this could cause a linking problem if GHC requests iserv to load a different version of the "network" library than the one it was compiled against. Cheers Alberto

Simon Marlow

3:30 a.m.

On 6 July 2016 at 18:14, Ben Gamari wrote:

...

Alberto Valverde writes:

...
I've hacked around this to test the feasibility of the approach by using stdin/stdout instead of creating new pipes and, surprisingly, managed to cross-compile a simple Template Haskell program.

Well done!

Indeed, nice hacking :)

...

...
I'm considering using a socket for communicating between both processes as a more permanent solution but this would incur in a dependency on the "network" package. Would this be acceptable?

It would be nice if we could avoid it; GHC depending upon a library has the very unfortunate effect that any user code also needing to link against GHC or one of its libraries now has no choice in the version of that library. We go to some lengths to try to keep the dependency footprint of GHC small for this reason.

...
Named pipes have also crossed my mind but I'm not sure how well they're supported by wine.

It would be great if there were some way we could make this work with named pipes. Not only does it side-step the dependency issue, but it feels like the right way forward.

I agree, named pipes are probably a better plan, perhaps a better solution overall than the way we currently pass FD numbers on the command line. Do named pipes work work as expected through wine? We would have to be careful to clean them up again afterwards. Cheers, Simon

Alberto Valverde

6:37 a.m.

On Thu, Jul 7, 2016 at 9:30 AM, Simon Marlow wrote:

...

On 6 July 2016 at 18:14, Ben Gamari wrote:

...
It would be great if there were some way we could make this work with named pipes. Not only does it side-step the dependency issue, but it feels like the right way forward.

I agree, named pipes are probably a better plan, perhaps a better solution overall than the way we currently pass FD numbers on the command line. Do named pipes work work as expected through wine? We would have to be careful to clean them up again afterwards.

I've implemented IPC with named pipes and it appeared to work through wine but now I'm investigating an issue which causes GHC to "freeze" when talking to the external interpreter when it is not running on a TTY (ie: in a build process) Cheers, Alberto

Ben Gamari

26 Aug 26 Aug

10:56 a.m.

Alberto Valverde writes:

...

On Thu, Jul 7, 2016 at 9:30 AM, Simon Marlow wrote:

...
I agree, named pipes are probably a better plan, perhaps a better solution overall than the way we currently pass FD numbers on the command line. Do named pipes work work as expected through wine? We would have to be careful to clean them up again afterwards.

I've implemented IPC with named pipes and it appeared to work through wine but now I'm investigating an issue which causes GHC to "freeze" when talking to the external interpreter when it is not running on a TTY (ie: in a build process)

Hi Alterto, What ever happened to this? Is there any way I can be of assistance? It would be great to get this merged. Cheers, - Ben

Alberto Valverde

27 May 27 May

4:29 a.m.

Hi Ben, I've just found this email from almost a year ago. Sorry for ignoring it until now. Given the date it was sent I think I missed it when it was fresh while I was on vacation. Regarding the question, I have abandoned the work I was doing on getting cross-compilation for Windows to work since my employer decided a Windows build for the app we were developing could wait some time and tasked me we something else. However, I see that Moritz is working on cross-compilation right now so I'll try get some time to see if I can get his approach to work for our use-case and give some feedback or help in any way. Alberto On Fri, Aug 26, 2016 at 4:56 PM, Ben Gamari wrote:

...

Alberto Valverde writes:

...
On Thu, Jul 7, 2016 at 9:30 AM, Simon Marlow wrote:

...
I agree, named pipes are probably a better plan, perhaps a better solution overall than the way we currently pass FD numbers on the command line. Do named pipes work work as expected through wine? We would have to be careful to clean them up again afterwards.

I've implemented IPC with named pipes and it appeared to work through wine but now I'm investigating an issue which causes GHC to "freeze" when talking to the external interpreter when it is not running on a TTY (ie: in a build process)

Hi Alterto,

What ever happened to this? Is there any way I can be of assistance? It would be great to get this merged.

Cheers,

- Ben

Moritz Angermann

4:51 a.m.

Hi Alberto, let me know if I can be of help. As you likely saw I’ve been writing this all up on https://medium.com/@zw3rk. With the outstanding diffs[1], this should hopefully just work. As I haven’t used windows or know the linker on windows, I’m not perfectly sure about the implications. It could *just work* though. Maybe you can even use the GHCSlave’s Slave.hs[2] for the Raspberry Pi for windows, as it doesn’t encode any Raspberry Pi specifics. All you would need is to run GHCSlave on wine / or a windows system with some form of network interface, so that the iserv-proxy and GHCSlave can communicate. Cheers, Moritz [1]: you can just use the my-ghc branch from https://github.com/zw3rk/ghc [2]: https://github.com/zw3rk/ghc-slave/blob/master/RPi/Slave.hs

...

On May 27, 2017, at 4:29 PM, Alberto Valverde wrote:

Hi Ben,

I've just found this email from almost a year ago. Sorry for ignoring it until now. Given the date it was sent I think I missed it when it was fresh while I was on vacation.

Regarding the question, I have abandoned the work I was doing on getting cross-compilation for Windows to work since my employer decided a Windows build for the app we were developing could wait some time and tasked me we something else.

However, I see that Moritz is working on cross-compilation right now so I'll try get some time to see if I can get his approach to work for our use-case and give some feedback or help in any way.

Alberto

On Fri, Aug 26, 2016 at 4:56 PM, Ben Gamari wrote: Alberto Valverde writes:

...
On Thu, Jul 7, 2016 at 9:30 AM, Simon Marlow wrote:

...
I agree, named pipes are probably a better plan, perhaps a better solution overall than the way we currently pass FD numbers on the command line. Do named pipes work work as expected through wine? We would have to be careful to clean them up again afterwards.

I've implemented IPC with named pipes and it appeared to work through wine but now I'm investigating an issue which causes GHC to "freeze" when talking to the external interpreter when it is not running on a TTY (ie: in a build process)

Hi Alterto,

What ever happened to this? Is there any way I can be of assistance? It would be great to get this merged.

Cheers,

- Ben

Alberto Valverde

30 May 30 May

6:16 a.m.

Hi Moritz, Sounds like it should be pretty straightforward to run GHCSlave on wine, so it might just work indeed. I'll build a nix environment for my app using your GHC branch and report back ASAP. However, what might block me won't be related to the cross-compilation per-se but to the migration to post 8.2.1 ghc since I'm still slowly making progress with all the dependencies but not there yet. Cheers, Alberto On Sat, May 27, 2017 at 10:51 AM, Moritz Angermann wrote:

...

Hi Alberto,

let me know if I can be of help. As you likely saw I’ve been writing this all up on https://medium.com/@zw3rk. With the outstanding diffs[1], this should hopefully just work. As I haven’t used windows or know the linker on windows, I’m not perfectly sure about the implications. It could *just work* though. Maybe you can even use the GHCSlave’s Slave.hs[2] for the Raspberry Pi for windows, as it doesn’t encode any Raspberry Pi specifics.

All you would need is to run GHCSlave on wine / or a windows system with some form of network interface, so that the iserv-proxy and GHCSlave can communicate.

Cheers, Moritz

[1]: you can just use the my-ghc branch from https://github.com/zw3rk/ghc [2]: https://github.com/zw3rk/ghc-slave/blob/master/RPi/Slave.hs

...
On May 27, 2017, at 4:29 PM, Alberto Valverde wrote:

Hi Ben,

I've just found this email from almost a year ago. Sorry for ignoring it until now. Given the date it was sent I think I missed it when it was fresh while I was on vacation.

Regarding the question, I have abandoned the work I was doing on getting cross-compilation for Windows to work since my employer decided a Windows build for the app we were developing could wait some time and tasked me we something else.

However, I see that Moritz is working on cross-compilation right now so I'll try get some time to see if I can get his approach to work for our use-case and give some feedback or help in any way.

Alberto

On Fri, Aug 26, 2016 at 4:56 PM, Ben Gamari wrote: Alberto Valverde writes:

...
On Thu, Jul 7, 2016 at 9:30 AM, Simon Marlow wrote:

...
I agree, named pipes are probably a better plan, perhaps a better solution overall than the way we currently pass FD numbers on the command line. Do named pipes work work as expected through wine? We would have to be careful to clean them up again afterwards.

I've implemented IPC with named pipes and it appeared to work through wine but now I'm investigating an issue which causes GHC to "freeze" when talking to the external interpreter when it is not running on a TTY (ie: in a build process)

Hi Alterto,

What ever happened to this? Is there any way I can be of assistance? It would be great to get this merged.

Cheers,

- Ben

Phyx

6 Jul 6 Jul

2:05 p.m.

...

However, compiling TH by using wine and ghc-iserver.exe fails because the file descriptor ids that GHC passes as arguments to "wine ghc-iserv.exe" don't make sense in the emulated windows world.

Just as a note, On Windows we don't pass FDs (As in the posix file descriptor) to the child process. It passes down Windows File Handles. This requires the Windows process inheritance/security model to be implemented in wine and I know nothing about Wine. But just wanted to clarify. Named pipes would solve your issue, Just don't forget to set the proper ACL on the pipes. On Wed, Jul 6, 2016 at 2:00 PM, Alberto Valverde wrote:

...

Hello,

I'm trying to put together a GHC 8.0.1 cross-compiler with Template Haskell support. Initially to target Windows (32bits) from a Linux host but a similar procedure should enable to target other platforms too. I'd like to contribute the patches back so I'm asking for advice on how to implement it in order to increase the chances of them being accepted.

I've managed to get a working stage 1 cross-compiler with some patches which correctly builds all stage1 libs and GHC + stage 2 compiler and ghc-iserv.exe. However, compiling TH by using wine and ghc-iserver.exe fails because the file descriptor ids that GHC passes as arguments to "wine ghc-iserv.exe" don't make sense in the emulated windows world.

I've hacked around this to test the feasibility of the approach by using stdin/stdout instead of creating new pipes and, surprisingly, managed to cross-compile a simple Template Haskell program.

I'm considering using a socket for communicating between both processes as a more permanent solution but this would incur in a dependency on the "network" package. Would this be acceptable?

Named pipes have also crossed my mind but I'm not sure how well they're supported by wine.

Thanks for your attention. Alberto

P.S: Code is in the "cross-mingw-hacks" branches of these repositories and is based on the ghc-8.0.1-release tag:

https://github.com/albertov/ghc https://github.com/albertov/hsc2hs

A Docker image script here that builds it is here:

https://github.com/albertov/ghc-cross-compiler-windows-x86

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

2912

Age (days ago)

3240

Last active (days ago)

List overview

Download

11 comments

6 participants

participants (6)

Alberto Valverde
Ben Gamari
Ben Gamari
Moritz Angermann
Phyx
Simon Marlow

Cross-compiling Template Haskell via -fexternal-interpreter and IPC

tags

participants (6)