Making (useful subsets of) bytecode portable between targets

Shea Levy

21 Nov 2016 21 Nov '16

7:50 a.m.

Hi all, I'm interested in implementing a general solution for TH during cross-compilation, which if my naive lack-of-understanding is correct will broadly involve the following three tasks: 1. Make the generation of byte code, or at least the subset needed for useful TH use, target-independent 2. Teach the external-interpreter to launch a build-native iserv when cross-compiling 3. Teach cabal to compile dependencies and modules for the build and target when cross-compiling and TH is used Of course, due to the generality of TH there will be code you can write that would be different in this approach from what you would get with a fully native compilation (e.g. due to GHC conditional compilation in the TH functions or FFI that does runtime host introspection), but since requiring a target device at build time is in many cases impractical (do you want to hook up an iPhone to your build farm?) and in some cases probably impossible (for targets without the resources to run a full GHC even if they can run GHC-compiled code) I think this is a reasonable restriction to require. My questions to the list are: * Is 1 above a pipe dream, keeping in mind that we are assuming the availability of build-native versions of all dependencies and already-compiled modules? * Any pointers for getting started with 1? * Anything I'm missing? Thanks, Shea

Attachments:

signature.asc (application/pgp-signature — 800 bytes)

Show replies by date

Ben Gamari

21 Nov 21 Nov

4:06 p.m.

CCing Luite who has done this sort of thing in the context of GHCJS. Shea Levy writes:

...

Hi all,

I'm interested in implementing a general solution for TH during cross-compilation, which if my naive lack-of-understanding is correct will broadly involve the following three tasks:

1. Make the generation of byte code, or at least the subset needed for useful TH use, target-independent

The bytecode machine itself (defined by ByteCodeInstr) is reasonably target-independent. However, I suspect primops will be problematic since they can appear in BCOs and are quite target dependent (namely due to word size). I honestly don't know how much of a problem this will be in practice; I tried loading a few example programs into ghci -ddump-bcos and didn't see any PUSH_PRIMOP instructions, so perhaps this won't be as problematic as I expect; it is worrisome though. In general it's hard to identify "the subset needed for useful TH use" but the fact that I can load the gitrev package (a fairly non-trivial use of TH) into GHCi with no apparently platform dependent BCOs being produced is promising.

...

2. Teach the external-interpreter to launch a build-native iserv when cross-compiling

For this you'll want to have a careful look through the Binary instances for the things that are sent over the wire to iserv. You'll find these occur in libraries/ghci/*.hs. The first thing to look for is places where the system word-size may leak into the serialized representation (e.g. instances where Ints and Words are being used). Many of these can probably just be narrowed into 32-bit representations (e.g. in ResolvedBCO, where I think it's fair to assume that four-billion bytecode instructions is an unreasonable BCO size). Don't forget to consider the platform dependence of the "primitive" encodings like the Binary instance for ByteString [1]; you'll need to take care to replace these with platform-independent encodings. [1] https://hackage.haskell.org/package/binary-0.8.4.1/docs/src/Data.Binary.Clas...

...

3. Teach cabal to compile dependencies and modules for the build and target when cross-compiling and TH is used

You'll need to speak with the Cabal folks about this. However, it's probably safe to ignore this for now; there's a lot of work to done before you'll be in a position to start thinking about this.

...

Of course, due to the generality of TH there will be code you can write that would be different in this approach from what you would get with a fully native compilation (e.g. due to GHC conditional compilation in the TH functions or FFI that does runtime host introspection), but since requiring a target device at build time is in many cases impractical (do you want to hook up an iPhone to your build farm?) and in some cases probably impossible (for targets without the resources to run a full GHC even if they can run GHC-compiled code) I think this is a reasonable restriction to require.

My questions to the list are:

* Is 1 above a pipe dream, keeping in mind that we are assuming the availability of build-native versions of all dependencies and already-compiled modules?

This sounds quite feasible to me.

...

* Any pointers for getting started with 1?

See above. The first thing to do is to rid the Binary instances of platform dependent encodings. This will be a bit of a game of whack-a-mole, but it shouldn't be that hard. Don't hesitate to ask if you get stuck! Cheers, - Ben

Edward Z. Yang

24 Nov 24 Nov

10:59 p.m.

Excerpts from Ben Gamari's message of 2016-11-21 16:06:14 -0500:

...

...
3. Teach cabal to compile dependencies and modules for the build and target when cross-compiling and TH is used

You'll need to speak with the Cabal folks about this. However, it's probably safe to ignore this for now; there's a lot of work to done before you'll be in a position to start thinking about this.

Some related work that has been done for just compiling Setup scripts with the host system: https://github.com/haskell/cabal/issues/1493 and https://github.com/haskell/cabal/issues/4047 is where we have been going with it. John Ericson is in charge of the effort, I've CC'ed him to this list. Edward

Simon Marlow

23 Nov 23 Nov

6:50 a.m.

On 21 November 2016 at 12:50, Shea Levy wrote:

...

Hi all,

I'm interested in implementing a general solution for TH during cross-compilation, which if my naive lack-of-understanding is correct will broadly involve the following three tasks:

1. Make the generation of byte code, or at least the subset needed for useful TH use, target-independent 2. Teach the external-interpreter to launch a build-native iserv when cross-compiling 3. Teach cabal to compile dependencies and modules for the build and target when cross-compiling and TH is used

The external interpreter was aimed at being able to run target code at compile time. I think what you have in mind is running *host* code at compile-time instead. There are a number of tricky things here. - You have to build everything for the host as well as the target. All the libraries that GHC builds will need to be built in two flavours, and the external build tools Cabal/Stack (as you mentioned) need to know about this. - GHC itself will need to be able to generate code for multiple platforms. We've done some work in this direction, but I don't think it's complete. Maybe it's not all that far off, but there are still some platform #ifdefs in the code. - As you point out, the code we compile for the host platform will not necessarily be the same as the target code, due to #ifdefs and other platform differences. - There is a point where the two platforms meet, when we're compiling a module for the target platform that uses TH. Every assumption that we made about the target platform (from reading .hi files) must also be true of the host platform, because we're about to run that code. So the .hi files for the host and target must be exactly the same. One way this could easily go wrong is if you use some API that's only available on the target platform in some TH code. GHC won't know that anything is wrong, but we'll get a link error (or worse, a crash) when trying to run the TH code. What this comes down to is that GHC doesn't have a clean separation between compile-time execution and runtime execution, it assumes that they're the same. For a (slightly opinionated) perspective on this, see http://blog.ezyang.com/2016/07/what-template-haskell-gets-wrong-and-racket-g... . So what I'm saying is: you could probably make this work, for some slightly flaky and brittle value of "work". Doing it properly is also pretty hard in the context of GHC, though, because it's a fairly fundamental upheaval. My question would be: are you *sure* you can't run target code at compile time? Not even with an iphone simulator? Cheers Simon Of course, due to the generality of TH there will be code you can write

...

that would be different in this approach from what you would get with a fully native compilation (e.g. due to GHC conditional compilation in the TH functions or FFI that does runtime host introspection), but since requiring a target device at build time is in many cases impractical (do you want to hook up an iPhone to your build farm?) and in some cases probably impossible (for targets without the resources to run a full GHC even if they can run GHC-compiled code) I think this is a reasonable restriction to require.

My questions to the list are:

* Is 1 above a pipe dream, keeping in mind that we are assuming the availability of build-native versions of all dependencies and already-compiled modules? * Any pointers for getting started with 1? * Anything I'm missing?

Thanks, Shea

Moritz Angermann

9:05 p.m.

...

On Nov 23, 2016, at 7:50 PM, Simon Marlow wrote:

[…]

My question would be: are you *sure* you can't run target code at compile time? Not even with an iphone simulator?

This should be possible. However for proper development one would need to run on the device (iPhone, iPad, …) for armv7 or arm64, as the Simulator is i386 or x86_64. There is a bit of additional engineering required here to get the shipping of code from ghc to the runner on the target required (e.g. via network). As executing and controlling applications on the actual hardware is limited, I guess a custom ghc-runner application would have to be manually started on the device, which could trivially be discovered using bonjour/zeroconf (or just giving ghc the host:port information). In general though, the runner does not have to obey all the restrictions apple puts onto app-store distributed apps, as I expect that everyone could build and install the runner themselves when intending to do iOS development with ghc. cheers, moritz

Manuel M T Chakravarty

11:42 p.m.

Sorry, but I don’t think running on the device is practical. How do you want to do CI, for example? Manuel

...

Moritz Angermann :

...
On Nov 23, 2016, at 7:50 PM, Simon Marlow wrote:

[…]

My question would be: are you *sure* you can't run target code at compile time? Not even with an iphone simulator?

This should be possible. However for proper development one would need to run on the device (iPhone, iPad, …) for armv7 or arm64, as the Simulator is i386 or x86_64.

There is a bit of additional engineering required here to get the shipping of code from ghc to the runner on the target required (e.g. via network). As executing and controlling applications on the actual hardware is limited, I guess a custom ghc-runner application would have to be manually started on the device, which could trivially be discovered using bonjour/zeroconf (or just giving ghc the host:port information).

In general though, the runner does not have to obey all the restrictions apple puts onto app-store distributed apps, as I expect that everyone could build and install the runner themselves when intending to do iOS development with ghc.

cheers, moritz _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Moritz Angermann

11:51 p.m.

It's certainly far from ideal, but for CI, what obstacles are there besides needing a runner accessible from cross compiling machine? E.g. Start the runner app on an iPhone plugged in into a USB power source and leave it there? Sent from my iPhone

...

On 24 Nov 2016, at 12:42 PM, Manuel M T Chakravarty wrote:

Sorry, but I don’t think running on the device is practical. How do you want to do CI, for example?

Manuel

...
Moritz Angermann :

...
On Nov 23, 2016, at 7:50 PM, Simon Marlow wrote:

[…]

My question would be: are you *sure* you can't run target code at compile time? Not even with an iphone simulator?

This should be possible. However for proper development one would need to run on the device (iPhone, iPad, …) for armv7 or arm64, as the Simulator is i386 or x86_64.

There is a bit of additional engineering required here to get the shipping of code from ghc to the runner on the target required (e.g. via network). As executing and controlling applications on the actual hardware is limited, I guess a custom ghc-runner application would have to be manually started on the device, which could trivially be discovered using bonjour/zeroconf (or just giving ghc the host:port information).

In general though, the runner does not have to obey all the restrictions apple puts onto app-store distributed apps, as I expect that everyone could build and install the runner themselves when intending to do iOS development with ghc.

cheers, moritz _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Moritz Angermann

11:55 p.m.

Hi, just to also note: that this completely ignores any host / target IO actions that TH might want to run. cheers, moritz

...

On Nov 24, 2016, at 12:51 PM, Moritz Angermann wrote:

It's certainly far from ideal, but for CI, what obstacles are there besides needing a runner accessible from cross compiling machine?

E.g. Start the runner app on an iPhone plugged in into a USB power source and leave it there?

Sent from my iPhone

...
On 24 Nov 2016, at 12:42 PM, Manuel M T Chakravarty wrote:

Sorry, but I don’t think running on the device is practical. How do you want to do CI, for example?

Manuel

...
Moritz Angermann :

...
On Nov 23, 2016, at 7:50 PM, Simon Marlow wrote:

[…]

My question would be: are you *sure* you can't run target code at compile time? Not even with an iphone simulator?

This should be possible. However for proper development one would need to run on the device (iPhone, iPad, …) for armv7 or arm64, as the Simulator is i386 or x86_64.

There is a bit of additional engineering required here to get the shipping of code from ghc to the runner on the target required (e.g. via network). As executing and controlling applications on the actual hardware is limited, I guess a custom ghc-runner application would have to be manually started on the device, which could trivially be discovered using bonjour/zeroconf (or just giving ghc the host:port information).

In general though, the runner does not have to obey all the restrictions apple puts onto app-store distributed apps, as I expect that everyone could build and install the runner themselves when intending to do iOS development with ghc.

cheers, moritz _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Manuel M T Chakravarty

24 Nov 24 Nov

12:38 a.m.

If you use Travis CI or such, do you really want to have a runner accessible from an arbitrary host on the Internet?

...

Moritz Angermann :

It's certainly far from ideal, but for CI, what obstacles are there besides needing a runner accessible from cross compiling machine?

E.g. Start the runner app on an iPhone plugged in into a USB power source and leave it there?

Sent from my iPhone

...
On 24 Nov 2016, at 12:42 PM, Manuel M T Chakravarty wrote:

Sorry, but I don’t think running on the device is practical. How do you want to do CI, for example?

Manuel

...
Moritz Angermann :

...
On Nov 23, 2016, at 7:50 PM, Simon Marlow wrote:

[…]

My question would be: are you *sure* you can't run target code at compile time? Not even with an iphone simulator?

This should be possible. However for proper development one would need to run on the device (iPhone, iPad, …) for armv7 or arm64, as the Simulator is i386 or x86_64.

There is a bit of additional engineering required here to get the shipping of code from ghc to the runner on the target required (e.g. via network). As executing and controlling applications on the actual hardware is limited, I guess a custom ghc-runner application would have to be manually started on the device, which could trivially be discovered using bonjour/zeroconf (or just giving ghc the host:port information).

In general though, the runner does not have to obey all the restrictions apple puts onto app-store distributed apps, as I expect that everyone could build and install the runner themselves when intending to do iOS development with ghc.

cheers, moritz _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Edward Z. Yang

11:01 p.m.

At least for Travis, you can generate a private key that only Travis has access to, and use this to authenticate access to the runner. See https://docs.travis-ci.com/user/encryption-keys/ Edward Excerpts from Manuel M T Chakravarty's message of 2016-11-24 16:38:34 +1100:

...

If you use Travis CI or such, do you really want to have a runner accessible from an arbitrary host on the Internet?

...
Moritz Angermann :

It's certainly far from ideal, but for CI, what obstacles are there besides needing a runner accessible from cross compiling machine?

E.g. Start the runner app on an iPhone plugged in into a USB power source and leave it there?

Sent from my iPhone

...
On 24 Nov 2016, at 12:42 PM, Manuel M T Chakravarty wrote:

Sorry, but I don’t think running on the device is practical. How do you want to do CI, for example?

Manuel

...
Moritz Angermann :

...
On Nov 23, 2016, at 7:50 PM, Simon Marlow wrote:

[…]

My question would be: are you *sure* you can't run target code at compile time? Not even with an iphone simulator?

This should be possible. However for proper development one would need to run on the device (iPhone, iPad, …) for armv7 or arm64, as the Simulator is i386 or x86_64.

There is a bit of additional engineering required here to get the shipping of code from ghc to the runner on the target required (e.g. via network). As executing and controlling applications on the actual hardware is limited, I guess a custom ghc-runner application would have to be manually started on the device, which could trivially be discovered using bonjour/zeroconf (or just giving ghc the host:port information).

In general though, the runner does not have to obey all the restrictions apple puts onto app-store distributed apps, as I expect that everyone could build and install the runner themselves when intending to do iOS development with ghc.

cheers, moritz _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Manuel M T Chakravarty

25 Nov 25 Nov

1:38 a.m.

Ok, I am not saying that it is technical impossible. I am saying that it is *impractical*. Imagine Travis CI needing to run stuff on my phone that is attached to my Mac (if we are lucky), which is behind NAT somewhere in Australia. Running stuff in the simulator during a build would be pretty awkward, but running it on the device is not practical. Manuel PS: BTW, shipping binary code to the device means it has to be code signed using a distribution profile of a registered developer. That is one thing if Xcode does all the magic behind the scenes, but do you really want to make that part of the GHC build process?

...

Edward Z. Yang :

At least for Travis, you can generate a private key that only Travis has access to, and use this to authenticate access to the runner. See https://docs.travis-ci.com/user/encryption-keys/

Edward

Excerpts from Manuel M T Chakravarty's message of 2016-11-24 16:38:34 +1100:

...
If you use Travis CI or such, do you really want to have a runner accessible from an arbitrary host on the Internet?

...
Moritz Angermann :

It's certainly far from ideal, but for CI, what obstacles are there besides needing a runner accessible from cross compiling machine?

E.g. Start the runner app on an iPhone plugged in into a USB power source and leave it there?

Sent from my iPhone

...
On 24 Nov 2016, at 12:42 PM, Manuel M T Chakravarty wrote:

Sorry, but I don’t think running on the device is practical. How do you want to do CI, for example?

Manuel

...
Moritz Angermann :

...
On Nov 23, 2016, at 7:50 PM, Simon Marlow wrote:

[…]

My question would be: are you *sure* you can't run target code at compile time? Not even with an iphone simulator?

This should be possible. However for proper development one would need to run on the device (iPhone, iPad, …) for armv7 or arm64, as the Simulator is i386 or x86_64.

There is a bit of additional engineering required here to get the shipping of code from ghc to the runner on the target required (e.g. via network). As executing and controlling applications on the actual hardware is limited, I guess a custom ghc-runner application would have to be manually started on the device, which could trivially be discovered using bonjour/zeroconf (or just giving ghc the host:port information).

In general though, the runner does not have to obey all the restrictions apple puts onto app-store distributed apps, as I expect that everyone could build and install the runner themselves when intending to do iOS development with ghc.

cheers, moritz _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Moritz Angermann

2:23 a.m.

Aren’t we taking this a bit too far off topic? I feared as much when I wrote my initial reply. Please excuse. I agree that ghc + runner is not an optimal, and maybe even for some tasks (iOS) a pretty convoluted solution. This is only if we follow the proven solution to TH that luite in ghcjs pioneered, and which later found it’s way into ghc through iserv. If someone proposes a solution to TH that does not require a runner and allows the TH to be fully evaluated on the host with no need to evaluate on the target for cross compilation, that would be great! If the runner would just require the same architecture, maybe qemu would be a solution that would not require a device running? Then again I’m not sure how that would work with TH that directly or indirectly accesses libraries only available on iOS for example. Please don’t get me wrong. IMO ghc without TH is quite crippled, and therefore so is a cross compiling ghc. From the solutions I saw to this problem (zeroth, evil-splicer, and the ghcjs runner approach), the ghcjs runner approach, seems to me at least as the most promising, that would work for the largest subset of TH. To get this back on topic, if we have a architecture independent interpretable bytecode, for ghc, could we sidestep the runner solution altogether and have TH for the target be evaluated on the host? Is this what Shea initially wanted to go after? cheers, moritz

...

On Nov 25, 2016, at 2:38 PM, Manuel M T Chakravarty wrote:

Ok, I am not saying that it is technical impossible. I am saying that it is *impractical*.

Imagine Travis CI needing to run stuff on my phone that is attached to my Mac (if we are lucky), which is behind NAT somewhere in Australia.

Running stuff in the simulator during a build would be pretty awkward, but running it on the device is not practical.

Manuel

PS: BTW, shipping binary code to the device means it has to be code signed using a distribution profile of a registered developer. That is one thing if Xcode does all the magic behind the scenes, but do you really want to make that part of the GHC build process?

...
Edward Z. Yang :

At least for Travis, you can generate a private key that only Travis has access to, and use this to authenticate access to the runner. See https://docs.travis-ci.com/user/encryption-keys/

Edward

Excerpts from Manuel M T Chakravarty's message of 2016-11-24 16:38:34 +1100:

...
If you use Travis CI or such, do you really want to have a runner accessible from an arbitrary host on the Internet?

...
Moritz Angermann :

It's certainly far from ideal, but for CI, what obstacles are there besides needing a runner accessible from cross compiling machine?

E.g. Start the runner app on an iPhone plugged in into a USB power source and leave it there?

Sent from my iPhone

...
On 24 Nov 2016, at 12:42 PM, Manuel M T Chakravarty wrote:

Sorry, but I don’t think running on the device is practical. How do you want to do CI, for example?

Manuel

...
Moritz Angermann :

> On Nov 23, 2016, at 7:50 PM, Simon Marlow wrote: > > […] > > My question would be: are you *sure* you can't run target code at compile time? Not even with an iphone simulator?

This should be possible. However for proper development one would need to run on the device (iPhone, iPad, …) for armv7 or arm64, as the Simulator is i386 or x86_64.

There is a bit of additional engineering required here to get the shipping of code from ghc to the runner on the target required (e.g. via network). As executing and controlling applications on the actual hardware is limited, I guess a custom ghc-runner application would have to be manually started on the device, which could trivially be discovered using bonjour/zeroconf (or just giving ghc the host:port information).

In general though, the runner does not have to obey all the restrictions apple puts onto app-store distributed apps, as I expect that everyone could build and install the runner themselves when intending to do iOS development with ghc.

cheers, moritz _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Manuel M T Chakravarty

4:49 a.m.

I totally agree that GHC with TH is crippled and that this is a major restriction off the cross-compilation story. All I am saying is that I see a device runner (and to a degree a simulator runner) not as a solution to this *important* problem. Architecture independent interpretable byte code seems a much more attractive avenue for me. (Sorry if I didn’t make this clear.) Manuel

...

Moritz Angermann :

Aren’t we taking this a bit too far off topic? I feared as much when I wrote my initial reply. Please excuse.

I agree that ghc + runner is not an optimal, and maybe even for some tasks (iOS) a pretty convoluted solution.

This is only if we follow the proven solution to TH that luite in ghcjs pioneered, and which later found it’s way into ghc through iserv. If someone proposes a solution to TH that does not require a runner and allows the TH to be fully evaluated on the host with no need to evaluate on the target for cross compilation, that would be great!

If the runner would just require the same architecture, maybe qemu would be a solution that would not require a device running? Then again I’m not sure how that would work with TH that directly or indirectly accesses libraries only available on iOS for example.

Please don’t get me wrong. IMO ghc without TH is quite crippled, and therefore so is a cross compiling ghc. From the solutions I saw to this problem (zeroth, evil-splicer, and the ghcjs runner approach), the ghcjs runner approach, seems to me at least as the most promising, that would work for the largest subset of TH.

To get this back on topic, if we have a architecture independent interpretable bytecode, for ghc, could we sidestep the runner solution altogether and have TH for the target be evaluated on the host? Is this what Shea initially wanted to go after?

cheers, moritz

...
On Nov 25, 2016, at 2:38 PM, Manuel M T Chakravarty wrote:

Ok, I am not saying that it is technical impossible. I am saying that it is *impractical*.

Imagine Travis CI needing to run stuff on my phone that is attached to my Mac (if we are lucky), which is behind NAT somewhere in Australia.

Running stuff in the simulator during a build would be pretty awkward, but running it on the device is not practical.

Manuel

PS: BTW, shipping binary code to the device means it has to be code signed using a distribution profile of a registered developer. That is one thing if Xcode does all the magic behind the scenes, but do you really want to make that part of the GHC build process?

...
Edward Z. Yang :

At least for Travis, you can generate a private key that only Travis has access to, and use this to authenticate access to the runner. See https://docs.travis-ci.com/user/encryption-keys/

Edward

Excerpts from Manuel M T Chakravarty's message of 2016-11-24 16:38:34 +1100:

...
If you use Travis CI or such, do you really want to have a runner accessible from an arbitrary host on the Internet?

...
Moritz Angermann :

It's certainly far from ideal, but for CI, what obstacles are there besides needing a runner accessible from cross compiling machine?

E.g. Start the runner app on an iPhone plugged in into a USB power source and leave it there?

Sent from my iPhone

...
On 24 Nov 2016, at 12:42 PM, Manuel M T Chakravarty wrote:

Sorry, but I don’t think running on the device is practical. How do you want to do CI, for example?

Manuel

> Moritz Angermann : > > >> On Nov 23, 2016, at 7:50 PM, Simon Marlow wrote: >> >> […] >> >> My question would be: are you *sure* you can't run target code at compile time? Not even with an iphone simulator? > > This should be possible. However for proper development one would need to run on the > device (iPhone, iPad, …) for armv7 or arm64, as the Simulator is i386 or x86_64. > > There is a bit of additional engineering required here to get the shipping of > code from ghc to the runner on the target required (e.g. via network). As executing > and controlling applications on the actual hardware is limited, I guess a custom > ghc-runner application would have to be manually started on the device, which could > trivially be discovered using bonjour/zeroconf (or just giving ghc the host:port information). > > In general though, the runner does not have to obey all the restrictions apple puts > onto app-store distributed apps, as I expect that everyone could build and install > the runner themselves when intending to do iOS development with ghc. > > cheers, > moritz > _______________________________________________ > ghc-devs mailing list > ghc-devs@haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Simon Marlow

6:11 a.m.

On 25 November 2016 at 07:23, Moritz Angermann wrote: [snip]

...

To get this back on topic, if we have a architecture independent interpretable bytecode, for ghc, could we sidestep the runner solution altogether and have TH for the target be evaluated on the host? Is this what Shea initially wanted to go after?

Yes, but architecture-independent bytecode is the least of the problems. Doing this properly is a really big change. We basically have two worlds: first, the compile-time world. In this world, we need all the packages and modules of the current package built for the host platform. Secondly, we need the runtime world, with all the packages and modules of the current package cross-compiled for the target platform. When compiling a module that uses TH, we need to - compile it as if we were compiling for the host platform, reading .hi files from the host world - run the TH code in the host world - restart the compilation, using the .hi files from the target world, and the results of running the splices But even this isn't going to be enough. What if your code imports some modules that are only available on the runtime platform? iOS APIs, for example. The right thing is to have a clean separation between runtime imports and compile-time imports. Perhaps we just annotate some imports to say they aren't needed at compile-time for running the TH code. but then we also need compile-time vs. runtime build-depends in our .cabal files, and so on. This is just off the top of my head, I'm sure there are more complexities I haven't thought about. Its a big project, but ultimately we do have to tackle it, because it's the right thing to do. Anyone interested in working on this? Maybe start a new proposal to flesh out the details. Cheers Simon

...

cheers, moritz

...
On Nov 25, 2016, at 2:38 PM, Manuel M T Chakravarty < chak@justtesting.org> wrote:

Ok, I am not saying that it is technical impossible. I am saying that it is *impractical*.

Imagine Travis CI needing to run stuff on my phone that is attached to my Mac (if we are lucky), which is behind NAT somewhere in Australia.

Running stuff in the simulator during a build would be pretty awkward, but running it on the device is not practical.

Manuel

PS: BTW, shipping binary code to the device means it has to be code signed using a distribution profile of a registered developer. That is one thing if Xcode does all the magic behind the scenes, but do you really want to make that part of the GHC build process?

...
Edward Z. Yang :

At least for Travis, you can generate a private key that only Travis has access to, and use this to authenticate access to the runner. See https://docs.travis-ci.com/user/encryption-keys/

Edward

...
If you use Travis CI or such, do you really want to have a runner accessible from an arbitrary host on the Internet?

...
Moritz Angermann :

It's certainly far from ideal, but for CI, what obstacles are there besides needing a runner accessible from cross compiling machine?

E.g. Start the runner app on an iPhone plugged in into a USB power

...
...
Sent from my iPhone

...
On 24 Nov 2016, at 12:42 PM, Manuel M T Chakravarty <

chak@justtesting.org> wrote:

...
...
Sorry, but I don’t think running on the device is practical. How do

you want to do CI, for example?

...
Manuel

> Moritz Angermann : > > >> On Nov 23, 2016, at 7:50 PM, Simon Marlow

wrote:

...
>> >> […] >> >> My question would be: are you *sure* you can't run target code at compile time? Not even with an iphone simulator? > > This should be possible. However for proper development one would need to run on the > device (iPhone, iPad, …) for armv7 or arm64, as the Simulator is i386 or x86_64. > > There is a bit of additional engineering required here to get the shipping of > code from ghc to the runner on the target required (e.g. via network). As executing > and controlling applications on the actual hardware is limited, I guess a custom > ghc-runner application would have to be manually started on the device, which could > trivially be discovered using bonjour/zeroconf (or just giving ghc

Excerpts from Manuel M T Chakravarty's message of 2016-11-24 16:38:34 +1100: source and leave it there? the host:port information).

...
...
...
> > In general though, the runner does not have to obey all the restrictions apple puts > onto app-store distributed apps, as I expect that everyone could build and install > the runner themselves when intending to do iOS development with ghc. > > cheers, > moritz > _______________________________________________ > ghc-devs mailing list > ghc-devs@haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Shea Levy

10:30 a.m.

Simon Marlow writes:

...

The right thing is to have a clean separation between runtime imports and compile-time imports. Perhaps we just annotate some imports to say they aren't needed at compile-time for running the TH code. but then we also need compile-time vs. runtime build-depends in our .cabal files, and so on.

Yes, I was just looking into this yesterday. We already have something similar for plugins, though of course the TH story is much more involved (in part because GHC has to compile haskell code, not just load and run a pre-existing object file).

...

Its a big project, but ultimately we do have to tackle it, because it's the right thing to do. Anyone interested in working on this? Maybe start a new proposal to flesh out the details.

Yeah, I'm going to at least try. Don't know where formal proposals go, but what I've got so far (definitely subject to change!): 1. Add a separate package database like we have for plugins for TH imports, defaulting to the normal package database when empty 2. Use the TH package db for code to be *run*, but resolve all reifications etc. against the normal target scope. 3. Add {-# TH #-}-annotated imports to specify modules to import compile-time code from (if a module *also* has definitions to be reified, it needs to be imported twice) 3a. If a module has any {-# TH #-} imports, enforce that *all* compile-time executed code is pulled in via {-# TH #-} imports. 3b. Optionally add a warning for TH-using code that doesn't use {-# TH #-}, as the first start of a migration path to enforcing compiletime/runtime separation across the ecosystem. 3c. Not sure what the specific difficulties are that require the staging restrictions, but possibly annotating same-module definitions with {-# TH -} might make this easier? 4. Teach cabal how to add packages to the TH database 4a. Not sure how all this works, but probably this is the place to e.g. request the non-profiled version of a package if ghc is non-profiled and the target code is profiled 5. Modify the build process of the cross-compiler to build a stage-2 compiler that builds code for the target *and* host (at least, enough of the host to build BCOs), and runs that host-targeted code in the interpreter. 5a. This will require some way to distinguish stage-2-as-target-native-compiler VS stage-2-as-hybrid-compiler.

Moritz Angermann

10:35 a.m.

There is https://github.com/ghc-proposals/ghc-proposals Sent from my iPhone

...

On 25 Nov 2016, at 11:30 PM, Shea Levy wrote:

Simon Marlow writes:

...
The right thing is to have a clean separation between runtime imports and compile-time imports. Perhaps we just annotate some imports to say they aren't needed at compile-time for running the TH code. but then we also need compile-time vs. runtime build-depends in our .cabal files, and so on.

Yes, I was just looking into this yesterday. We already have something similar for plugins, though of course the TH story is much more involved (in part because GHC has to compile haskell code, not just load and run a pre-existing object file).

...
Its a big project, but ultimately we do have to tackle it, because it's the right thing to do. Anyone interested in working on this? Maybe start a new proposal to flesh out the details.

Yeah, I'm going to at least try. Don't know where formal proposals go, but what I've got so far (definitely subject to change!):

1. Add a separate package database like we have for plugins for TH imports, defaulting to the normal package database when empty 2. Use the TH package db for code to be *run*, but resolve all reifications etc. against the normal target scope. 3. Add {-# TH #-}-annotated imports to specify modules to import compile-time code from (if a module *also* has definitions to be reified, it needs to be imported twice) 3a. If a module has any {-# TH #-} imports, enforce that *all* compile-time executed code is pulled in via {-# TH #-} imports. 3b. Optionally add a warning for TH-using code that doesn't use {-# TH #-}, as the first start of a migration path to enforcing compiletime/runtime separation across the ecosystem. 3c. Not sure what the specific difficulties are that require the staging restrictions, but possibly annotating same-module definitions with {-# TH -} might make this easier? 4. Teach cabal how to add packages to the TH database 4a. Not sure how all this works, but probably this is the place to e.g. request the non-profiled version of a package if ghc is non-profiled and the target code is profiled 5. Modify the build process of the cross-compiler to build a stage-2 compiler that builds code for the target *and* host (at least, enough of the host to build BCOs), and runs that host-targeted code in the interpreter. 5a. This will require some way to distinguish stage-2-as-target-native-compiler VS stage-2-as-hybrid-compiler.

Edward Z. Yang

10:45 a.m.

Excerpts from Shea Levy's message of 2016-11-25 10:30:25 -0500:

...

...
The right thing is to have a clean separation between runtime imports and compile-time imports. Perhaps we just annotate some imports to say they aren't needed at compile-time for running the TH code. but then we also need compile-time vs. runtime build-depends in our .cabal files, and so on.

Yes, I was just looking into this yesterday. We already have something similar for plugins, though of course the TH story is much more involved (in part because GHC has to compile haskell code, not just load and run a pre-existing object file).

Actually, we don't really have this for plugins either. It is true that you can load a plugin independently of the imports of the module it is compiling, and that you can setup an independent package database for plugins, but the interfaces of the plugin are still all dumped in the same interface database that GHC uses for compile-time interfaces as well. It's delicate and has caused bugs before (https://ghc.haskell.org/trac/ghc/ticket/10420) Edward

MarLinn

1:04 p.m.

On 2016-11-25 12:11, Simon Marlow wrote:

...

We basically have two worlds: first, the compile-time world. In this world, we need all the packages and modules of the current package built for the host platform. Secondly, we need the runtime world, with all the packages and modules of the current package cross-compiled for the target platform.

Maybe this separation and the preceding discussion of the two possible solutions suggests a usable approach to the architecture of this future system? First, let me reframe the "runner" idea. In a real-world environment, this seems like a viable solution either with two separate machines or with a VM nested in the main build machine. In both cases, we would need two parts of the compiler, communicating over customized channels. The cross-compiler approach is more or less just a variation on this with far less overhead. So why not build an architecture that supports both solutions? I practice, this would mean we need a tightly defined, but flexible API between at least two "architecture plugins" and one controller that could run on either side. To me, this sounds more like a build system than a mere compiler. And I'm okay with that, but I don't think GHC+Cabal alone can and should shoulder the complexity. There are nice, working build-systems out there that could take over the role of the controller, so all GHC and Cabal would have to offer are parsing, modularized steps, and nice hooks. In other words, /a //kind of meta-language to describe compiler deployments/ – and Haskell is great for describing languages. Here's yet another idea I'd like to add, although it is rather silly. The idea of a meta-language that describes a conversion structure seems very close to what Pandoc is doing for documents. And while Pandoc's architecture and history make it a bit static, GHC can still learn from it. Maybe, someday, there could even be a bigger, even more over-arching build language that describes the program, the documentation, and the deployment processes of the whole system? Cheers, MarLinn

3098

Age (days ago)

3102

Last active (days ago)

List overview

Download

17 comments

7 participants

participants (7)

Ben Gamari
Edward Z. Yang
Manuel M T Chakravarty
MarLinn
Moritz Angermann
Shea Levy
Simon Marlow