Hello, With the support of Tweag I/O, Mathieu and I have been assembling a design proposal for the language extension for Static values that will take Cloud Haskell a big step forward in usability. Please, find the proposal inlined below. We are looking forward to discuss its feasibility and features with the community. Best, Facundo -- In these notes we discuss a design of the language extension proposed in [1] for Cloud Haskell. That is, support from the compiler to produce labels that can be used to identify Haskell top-level bindings across processes in a network. Static values ========= Following [1], the extension consists of a new syntactic form `static e`, along with a type constructor `StaticRef` and a function unstatic :: StaticRef a -> a The idea is that values of type `StaticRef a` uniquely identify a value that can be referred to by a global name rather than serialized over the network between processes that are instances of a single binary, because all such processes share the same top-level bindings. Generating static references ==================== We start by introducing global names. A `GlobalName` is a symbol bound in the top-level environment. It is much like global names in Template Haskell, but `GlobalNames` always refer to terms, and they include a package version. data GlobalName = GlobalName PkgName PkgVersion ModName OccName `GlobalNames` can be used as references to static values. newtype StaticRef a = StaticRef GlobalName `StaticRef a` is to `GlobalName` what `Ptr a` is to `Addr#`: a wrapper with a phantom type parameter that keeps track of the type of the value that is referenced. The special form static e is an expression of type `StaticRef a` where `e :: a` is a closed expression (meaning any free variables in `e` are bound in the top-level environment). If `e` is an identifier, `static e` just refers to it. Otherwise, the compiler needs to introduce a new top-level binding with a fresh name and the expression used as right-hand side, and the static reference would point to this top-level binding instead. Looking up static references ==================== `unstatic` is implemented as a function which finds a top-level value from the `GlobalName`, otherwise it raises an exception. It crucially relies on leveraging the system’s dynamic linker, so out-of-the-box only works with dynamically linked binaries (but see below). `unstatic` proceeds as follows: * Determines the name of the shared library from the package name and the package version. * Determines the symbol of the value by Z-Encoding the package name, the module name and the value name. * Uses the system’s dynamic linker interface to obtain the address of the symbol. * Converts the symbol to a haskell value with `GHC.Prim.addrToAny#` In principle, only symbols in shared libraries can be found. However, the dynamic linker is able to find symbols in modules that are linked statically if GHC is fed with the option -optl-Wl,--export-dynamic. A future enhancement could be to have GHC warn the user when modules using the extension are linked statically and this option is not used during linking. GHC only defines symbols for exported definitions in modules. So unstatic won’t be able to find the private bindings of a module. For this sake, the implementation of static should in addition ensure that the bindings it gets will appear in the symbol table when they are not exported by their defining modules. Template Haskell support ================== The static keyword needs to be made available in Template Haskell so the distributed-static package can benefit from this language extension. Rationale ======= We want the language extension to meet the following requirements: 1. It must be a practical alternative to the remoteTable functions in the distributed-static package. 2. It must not change the build scheme used for Haskell programs. A collection of .o files produced from Haskell source code should still be possible to link with the system linking tools. 3. It must not restrict all communicating processes using the extension to be launched from the same binary. 4. It must not significantly increase the binary size. (1) is addressed by replacing remote tables with the symbol tables produced by the compiler. Additionally, Template Haskell support is included so that the existing distributed-static package can be adapted and extended to include this extension. (2) is addressed by choosing a scheme which does not require the linker to perform any extension-specific procedure to collect the static values in various modules. There’s a trade off here though, since symbols in statically linked modules cannot be accessed unless -optl-Wl,--export-dynamic is supplied during linking. (3) is addressed by allowing programs to exchange static values for any bindings found in the modules they share. (4) is addressed by reusing the symbol tables produced by the compiler in object files rather than creating separate remote tables. About the need for using different binaries ============================== While using distributed-process we found some use cases for supporting communicating closures between multiple binaries. One of these use cases involved a distributed application and a monitoring tool. The monitoring tool would need to link in some graphics libraries to display information on the screen, none of which were required by the monitored application. Conversely, the monitored application would link in some modules that the monitoring application didn’t need. Crucially, both applications are fairly loosely coupled, even if they both need to exchange static values about bindings in some modules they shared. An analogous use case involved the distributed application and a control application that would be used to change dynamic settings of the former. Further Work ========== As the application depends on shared libraries, now a tool to collect these libraries would be required so they can be distributed together with the executable binary when deploying a Cloud Haskell application in a cluster. We won’t delve further into this problem. Another possible line of work is extending this approach so a process can pull shared objects from a remote peer, when this remote peer sends a static value that is defined in a shared object not available to the process. Integration with distributed-static ======================= The package distributed-static could either adopt this extension as the only implementation of static values, or it could support many notions of static references, say by using a type class to overload `unstatic`. class Static st s | s -> st where unstatic :: st -> s a -> Either String a where the class parameter `st` is provided for backwards compatibility with the existing scheme to provide context-dependent information. The extension we present here does not depend on this parameter, so `()` could be used for the `StaticRef` instance. instance Static () StaticRef where ... References ======== [1] Jeff Epstein, Andrew P. Black, and Simon Peyton-Jones. Towards Haskell in the cloud. SIGPLAN Not., 46(12):118–129, September 2011. ISSN 0362-1340.