Hi all, sorry if it's a FAQ or I'm simply too dumb to look in the right places. What I want to do is to send Haskell values over a network to another Haskell process. Or save a Haskell value on disk and read it in in a later incarnation of the same pogram, or with a different program. Is there a good way to do this? Or, even better: a good way to find the right documentation for this sort of thing?

Hal Daume III wrote:

I think the standard way would be to use Show and Read and then send/write/rcv/read strings from the network/disk.

What sent me first into deep confusion is that I found all of {Text,GHC}.{Read,Show} first, and the Read classes marked as "nonportable GHC extensions". Quite disheartening :-( Well, then I found the Prelude definition (though it's entirely unclear how they relate to the Text and GHC versions). Hmm... I can write out functions using the "Show (a -> b)" instance, but there's no matching "Read (a -> b)". I do understand the technical difficulties, so I'm not really disappointed... but I had hoped I could handle all the code upgrade issues from within the language.

Or you could use the Binary class to get binary representations, but these (sometimes, depending on which implementation you use) have endianness issues if you're transmitting between different architectures.

I didn't find Binary in my GHC 6.0.1 distro. Is it a library that I have to install as an add-on? In all, I'm finding it rather difficult to find my way in the documentation (just as a data point for the documentation folks). To make the critique helpful instead of just a critique, here are a few ideas of what might have helped me (YMMV): 1. My main problem was that I have no good idea of all the installed libraries and the names defined in them (remember I'm a complete newbie to doing anything with a concrete Haskell installation). If somebody tells me to look for "Show", I have to sequentially open the indexes of all installed packages, and since I'm a newbie, I don't even know where to look first. I'd like to see a tool that gives me a unified overview of all installed packages. The tool should extend the "index.html" file in the ../doc/html directory so that it includes an entry for each installed library (probably copied from some "manifest.ihtml" file in the package directories), and it should create/extend the Haddockish doc-index-*.html files to be synopses of the doc-index-*.html files from the installed packages. If would be best if that tool were run automatically during package installation :-) (particularly for newbies who are fully occupied with learning the language and don't have much energy or time left for learning tools). 2. Remove the split between "type/class index" and "function/constructor index" in the Haddockish doc-index.html file. For newbies like me, the split means we have to visit two places instead of one. An index is for giving the reader all the usages of a word regardless of semantics. The doc-index-*.html file can still give kind (terminology?) information, e.g. in an extra column that says "function", "constructor", "type", or "class", respectively. 3. Some links in the indexes are bold, others aren't. I have no idea what they are - they /might/ differentiate between kinds, but if they do, I have no means to test this hypothesis. No, it seems that the normal ones are modules and the bold ones are uses, but I'm unable to determine what qualifies a module for being listed in the index for a given keyword, the Pool data type most certainly doesn't even appear in the Foreign module that's in the link list for Pool... Well, this all is getting me thoroughly confused, any clarification would be appreciated. I've got no idea how practical these ideas are, so regard them at their actual value :-) Regards, Jo

Hi, Hal Daume III wrote:

...

Hmm... I can write out functions using the "Show (a -> b)" instance, but there's no matching "Read (a -> b)".

Show (a -> b) is a bogus instances -- you won't actually be able to use it for marshalling functions.

Well, marshalling functions (or storing-restoring some internal forms of them) might be especially nice... This would mean I can declare and compile a function on my side of a network connection (for example), and then send it to the other end for evaluation, and then get the result. Like a database request. Is this something absolutely impossible in Haskell and by what reason? Just because of strong typing (forgive my stupidity ;)? Or are there some deeper theoretical limitations? -- Dmitry M. Golubovsky South Lyon, MI

Dimitry Golubovsky wrote:

Well, marshalling functions (or storing-restoring some internal forms of them) might be especially nice... This would mean I can declare and compile a function on my side of a network connection (for example), and then send it to the other end for evaluation, and then get the result. Like a database request.

I've got a different and (IMHO) more interesting application: upgrading code on the other side of a network. Application startup would be downloading updates from the server, then loading all the code from disk.

Is this something absolutely impossible in Haskell and by what reason? Just because of strong typing (forgive my stupidity ;)? Or are there some deeper theoretical limitations?

No theoretical limitations but practical ones. AFAIK, dynamic code loading has issues with garbage collection and global optimization - it's certainly doable and some languages do it, but it's nontrivial and certainly a lot of work. I'd like to have it anyway :-) But I'm not going to hold my breath for it; it's probably more prudent to try things with the currently available mechanisms (that is, Show/Read and Binary). Regards, Jo

Alastair Reid wrote:

...

Is [marshaling functions] something absolutely impossible in Haskell and by what reason? Just because of strong typing (forgive my stupidity ;)? Or are there some deeper theoretical limitations?

The big theoretical issue is whether it would provide an Eq or Show instance for -> by the backdoor. Careful API design could avoid the worst of this.

What's the problem with a Show instance for ->? (Or, put another way: what makes the current Show instance in the Prelude harmless compared to a fuller implementation?) I'm not sure whether having an Eq instance for -> would be so bad. General equality satisfies two constraints: 1. For all functions (including field access functions and whatnot), "equal" inputs will return "equal" outputs. 2. True generality is the finest partitioning that satisfies (1). (1) is a recursion (and a massive one), (2) specifies we want the least fixed point of the recursion. Also, it's (2) that makes equality undecidable. (1) is essential and cannot be changed. Would changing (2) be dangerous? What are the dangers? For example, we could define -> equality to be source code equality, modulo a well-defined set of semantics-preserving transformations (such as translation to core language and renaming local variables - nothing fancy, I'd consider f x = x to be different from f = \x x).

A semi-theoretical issue is to do with sharing of values, libraries and types. A simple function like '\ x -> x' is fairly easy to write out because it doesn't refer to any other objects but what about:

\ x -> head [ y | (x',y) <- table, x == x' ]

This refers to an object 'table' that will have to be accessible from the receiver. Choices are to access the object over the network, to copy the object or to move the object and access it remotely from the sender.

The default semantics for marshalling such an object would be to copy 'table' along with the function. Essentially it's the responsibility of the programmer to make sure that he isn't inadvertently exporting the entire state of the computation. (Actually sometimes that's exactly what's wanted: dump out the entire state of the computation, say, as a checkpoint to safeguard against potential crashes.)

Remote access requires a network connection and suddenly adding two numbers can raise a TCP/IP exception.

Right, but, in principle, adding two numbers can generate a stack overflow or an out-of-memory error. There's nothing new here, except for failure probabilities.

Copying has consequences for mutable datatypes (a useful but non-standard extension of Haskell), for foreign datatypes (Ptr, etc.) and for interfaces to foreign functions.

Indeed. These cannot be marshalled in any useful sense. (Which makes me more than a bit suspicious about the Fd instances of Show and Read, BTW.)

Copying also has consequences for space and time usage: each copy uses more space

That can be controlled by the programmer (provided that sharing is preserved over marshalling).

and laziness is lost.

Not at all. Simply marshall the thunks instead of the evaluated values. In other words, instances of marshalling functions must be non-strict. Actually I didn't find strictness annotations on the Show instances in the library docs. Is this an omission, intentionally left implementation-dependent, or are they indeed lazy?

If an object is copied:

1) Is it evaluated first?

It shouldn't. Haskell is a lazy language, and forcing evaluation would make any objects that contain an infinite list deep down in their bowels potentially non-marshallable.

2) Can it contain mutable objects? 3) Can it contain Foreign pointers?

Not by my book, at least not by default. If would probably be possible to create instances that handle specific cases, e.g. if the Foreign pointer is just an implementation vehicle for an FtpFile object, then there might be specific marshalling code that marshals just the URL and rebuilds the connection on the receiving side. This also exemplifies the standard problem: Most foreign functions can fail, even unpredictably, so it's probably better to send just the specifications for building an equivalent FFI object across the network (which is exactly what the IO monad is good for: it contains action specifications, not the actions themselves).

4) Can it contain functions?

Of course - this is what got this thread started in the first place :-)

5) Is sharing preserved within the object? e.g., does let x=1; y=(x,x,x) in (y,y,y) get sent as 3 objects or as 13 objects?

Semantically, this is irrelevant. However, solutions to the next point will usually preserve sharing, so this is less of an issue.

6) Are cyclic objects ok? e.g., can 'let x=1:x in x' be sent?

They should be. Actually, Show and Read should be able to handle this. ... unfortunately, a quick test in GHCi revealed that "show" goes into an endless loop when fed with the above expression. Obviously, I'm expecting too much here. Maybe some of my Mozart experience is showing through here - the Mozart/Oz Inspector tool will show circular values with no problems, any references that form a cycle are printed as a marker. I.e. the above would look something like A=1 : A when printed, 'let x = 1 : 2 : x' would print as A=1 : 2 : A etc.

If it wasn't for mutable objects, foreign types and foreign functions, copying objects would be a straightforward but tedious issue.

Agreed.

Some of these issues can be avoided if objects are evaluated first because then the type can be used to determine whether mutable objects or foreign types are reachable.

Sometimes the evaluated value will be shorter than its unevaluated version, sometimes it's the reverse. Let the programmer decide what makes more sense. Since the programmer can always force evaluation as needed but cannot "unevaluate" something once it was executed, the system shouldn't force evaluation. Particularly since evaluation may not always terminate (transmitting a value doesn't mean it will be evaluated on the other side after all).

\ x -> lookup x table

As well as referring to 'table', this also refers to a standard library function. We can either copy 'lookup' over or we can use the function of the same name on the receiver.

We can send a hash code of the function and have the receiver request the full function if it finds that the versions doesn't match. Actually, hash codes would be a better identification than qualified function names. Primitive functions would need name equality - there's no good way to compare them in any other useful way.

Copying the function over can get expensive because lots of Haskell libraries depend on other Haskell libraries and especially if we end up copying it over multiple times.

A function need never be transferred more than once. (Keep them in a hash table with weak references, so that they can be garbage collected if they aren't in use anymore.)

Using the function with the same name runs the risk that the name is the same but the function itself is different.

Indeed.

Avoiding the overhead of sharing functions more than once is probably easy enough. Create a signature for each function by hashing all the code in the function and in every function and object it depends on. Don't transmit functions which the receiver already has. If storing the function to a file, have the store operation omit functions listed in a table of signatures. e.g., you might provide signatures for all the standard libraries in ghc 6.02 on windows98. (Since most GHC operations are the same on all platforms and in all recent releases, this ought to match many functions in ghc 5.04.)

foo :: T -> T

If the receiving program contains a type 'T' and the sending program contains a type 'T', we might have to check that they are the same type so we have to choose an appropriate equality test for types. Is structural equality enough or too strong or do we also want to insist that any data hiding property the original program had is preserved?

Types can be compared just like functions. I believe that the issues are just the same as with equality, but I may be wrong here.

A small corner of this question concerns portability of type representations between architectures with different word sizes and different endianness.

Representation is not a serious problem (though getting it under control can be some work). What's more of a problem are implementation-dependent types, such as that integer type that's guaranteed to have at least such-and-so many binary digits. The shift functions assume a one's-complement representation of integers, which is the case on most modern processors but doesn't hold once things are ported to an MVS mainframe. (Oh, and Unicode function names are going to be very problematic if you want to compile them on a 7-bit ASCII or EBCDIC machine, but the only consequence that I can see for marshalling is that functions should never be identified by name during marshalling, they should be identified via hash code.) Regards, Jo

Joachim Durchholz writes:

What sent me first into deep confusion is that I found all of {Text,GHC}.{Read,Show} first, and the Read classes marked as "nonportable GHC extensions".

Then you will surely love the Foreign.* hierarchy, most notably Foreign.Storable. If you want to do binary I/O within "standard" Haskell, this is the place to look at. I found [1] to be a pretty good introduction to the subject. Just for the sake of providing you with an alternative to read / show: Take a look at HaXml [2]. This package provides means to read and write Haskell data structures as XML files, what is about as platform independent as you can get. Peter [1] http://www.cse.unsw.edu.au/~chak/haskell/ffi/ [2] http://www.cs.york.ac.uk/fp/HaXml/index.html

Glynn Clements wrote:

The problem with Storable is that the representation is architecture-specific. It isn't suitable for implementing a "commodity" protocol which can be used for communication between different architectures.

Just endianness issues, or are there deeper differences? If it's just endianness, it should be relatively easy to get that sorted out, either via wrappers around the networking functions, or by extending Storable itself (well, it's probably a bit too early for me to try this, but there's a good chance that it will be at least one of generally useful and instructive *g*). Regards, Jo

On Tue, 28 Oct 2003 17:09:36 +0100 Joachim Durchholz wrote:

In all, I'm finding it rather difficult to find my way in the documentation (just as a data point for the documentation folks). To make the critique helpful instead of just a critique, here are a few ideas of what might have helped me (YMMV):

Yes, I agree. The best way to learn seems to be to just sequentially read through all the libraries, keeping everything in the back of your mind. When there is no text, look at the type signature and name and try to guess what it does. When there is something odd, ask yourself why whoever wrote GHC would want something like that. There is some stuff that uses technical words like Monoid or Unboxed, so if you don't know what those mean, look up those concepts in google (either web or comp.lang.functional) and read random posts, pages, and academic papers about them. Finally the design of some stuff (esp monadaic things like STRef) is complicated and won't make sense until you try to use them in a sample program. At least that was/is my experience. :) It's not a totally unpleasant way of learning a language, but quite different than with most languages. Also it doesn't work if you're in hurry... -- Ben Escoto