[Sorry for the multiple reposts - couldn't quite figure out which
email address doesn't get refused by the list..]

Hi Carter,

thank you for the good points you raise. I'll try and address each of
them as best I can below.

> 0) I think you could actually implement this proposal as a userland library,
> at least as you've described it. Have you tried doing so?

Indeed, this could be done without touching the compiler at all. We
thought long and hard about a path that would ultimately make an
extension either unnecessary, or at any rate very small. At this
point, the only thing that we are proposing to add to the compiler is
the syntactic form "static e". Contrary to the presentation in the
paper, the 'unstatic' function can be implemented entirely as library
code and does not need to be a primop. Moreover, we do not need to
piece together any kind of global remote table at compile time or link
time, because we're piggy backing on that already constructed by the
system linker.

The `static e` form could as well be a piece of Template Haskell, but
making it a proper extension means that the compiler can enforce more
invariants and be a bit more helpful to the user. In particular,
detecting situations where symbolic references cannot be generated
because e.g. the imported packages were not compiled as dynamic linked
libraries. Or seamlessly supporting calling `static f` on an idenfier
`f` that is not exported by the module.

> 1) what does this accomplish that can not be accomplished by having various
> nodes agree on a DSL, and sending ASTs to each other?
>      1a) in fact, I'd argue (and some others agree, and i'll admit my
> opinions have been shaped by those more expert than me) that the sending a
> wee AST you can interpret on the other side is much SAFER than "sending a
> function symbol thats hard coded hopefully into both programs in a way that
> it means the same thing".

I very much subscribe to the idea of defining small DSL's for
exchanging code between nodes. And this proposal is compatible with
that idea.

One thing that might not have been so clear in the original email is
that we are proposing here to introduce just *one such DSL*. It's just
that it's a trivial one whose grammar only contains linker symbol
names.

As it happens, distributed-static today already supports two such
DSL's: a DSL of labels, which are arbitrary string names for
functions, and a small language for composing Static values together.
There is a patch lying around by Edsko proposing to add a third "DSL":
one that allows nodes to trade arbitrary Haskell strings that are then
eval'ed on the other end by the 'plugins' package.

As Facundo explains at the end of his email, the notion of a "static"
value ought to be a more general one than was first envisioned in the
paper: a static value is any closed denotation, denoted in any of a
choice of multiple small languages, some of which ship standard with
distributed-static. The user can define his own DSL for shipping code
around.

This is why we propose to make Static into a class. Each DSL is
generated by one datatype. Each such datatype has a Static instance.
If you would like to ship an AST around the cluster, you can make the
datatype for that AST an instance of Static, with 'unstatic' being
defined as an interpreter for your AST.

Concretely:

data HsExpr = ...

instance Static HsExpr where
  unstatic e = Hs.interpret e

> I've had many educational conversations with

... ?

> 2) how does it provide more type safety than the current TH based approach?
> (I've seen Tim and others hit very very gnarly bugs in cloud haskell based
> upon the "magic static values" approach).

The type safety of the current TH approach is reasonable I think. One
potential problem comes from managing dynamically typed values in the
remote table, which must be coerced to the right type and use the
right decoders if you don't use TH. With the approach we propose,
there is no remote table, so I guess this should help eliminate a
source of bugs.

> 3) this proposal requires changes to linking etc that would really make it
> useful only on systems and deployments that only have Template Haskell AND
> Dynamic linking.  (and also rules out any context where it'd be nice to
> deploy a static app or say, use CH in ios! )

I don't know about iOS. And it's very likely that there are contexts
in which this extension doesn't work. But as I said above, you are
always free to define your own DSL's that cover the particular use
case that you have in mind. The nice thing with this particular DSL is
that it requires little to no TH to generate label names, which can
always be a source of bugs, especially when you forget to include them
in the global remote table (which is something that TH doesn't and
can't help you with).

Furthermore, it was my understanding that GHC is heading towards a
world of "dynamic linkable by default", and it is by now something
that is supported on most platforms by GHC. See e.g.

https://ghc.haskell.org/trac/ghc/wiki/DynamicGhcPrograms

There are fairly good solutions to deploy self contained dynamically
linked apps these days, e.g. Docker. And in any case, with a few extra
flags we can still do away with the dynamic linking requirement on
some (all?) platforms.

> to repeat: have you considered defining an AST type + interpreter for the
> computations you want to send around, and doing that? I think its a much
> simpler, safer, easier, flexible and PORTABLE approach, though one current
> CH doesn't do (though the folks working on CH seem to be receptive to
> switching to such a strategy if someone validates it)

We have, and it's an option with different tradeoffs. Both solutions
could gainfully live side by side and are in fact complementary. I
contend that the solution described by Facundo has the advantage of
eliminating much of the syntactic overhead associated with sending
references to (higher-order) values across the cluster. We have more
ideas specific to distributed-process which we can discuss in a
separate thread to reduce the syntactic overhead even further, to
practically nothing.

Best,

Mathieu