This is a herculean effort due, among other things, to Spark’s heavy reliance on function serialization. Not to say you can’t do it, but it’s going to be a huge amount of effort to build and maintain it in any way that provides feature parity with any JVM language on Spark, and of course you’re going to be going Haskell -> C -> JVM, and JVM -> C -> Haskell for everything, which entails its own set of delightful pains.
If you’re not too picky about the ML, I’d suggest a better path is contributing to PureScript on the JVM (either that project or another one). PureScript’s semantics map cleanly to the JVM’s and very good native interop is possible; in addition, the PureScript compiler is designed for compiling to different backends (i.e. it has no runtime and will necessarily utilize the target’s native runtime environment).
One could ease Spark integration by targeting TASTY or generating functional code compatible with Scala / Java 8 runtimes.
--
John A. De Goes
Follow me @jdegoes
On July 14, 2015 at 9:13:05 PM, Vasili I. Galchin (vigalchin@gmail.com) wrote:
I am looking to suppprt Haskell (OCaml ??) for apache.spark.org. Currently there is support for programming language "R" which has functional features (not as rich as Haskell, OCaml, et. al.) and does lazy evaluation. I am going to drop back and look at the R->Scala/Java implementation BUT I will also read your link. Any more.suggestions are gratefully welcome.
Kind thx,
Vasili
On Tuesday, July 14, 2015, Jason Dagit