Picking an architecture for a Haskell web app

Folks, I have finished an alpha version of my EasyLanguage [1] to C# compiler and need to deploy it on Amazon EC2/S3. The compiler web interface is very simple: paste EL code, get back C# code or the same EL code with the error highlighted. I view the site as more than just a compiler, though, and want a blog, forums, etc. Last but not least, I want to be able to update portions of the site without shutting it down. My choice seems to boil down to HAppS [2], HOPE [3], or a combination of Ruby/Rails with Lambdabot. At first glance, HAppS lacks publishing (my ultimate need) but promises an interface to S3 sometime in the future. My understanding is that it's ready but is not being released. HAppS is heavy on features that I may not make use of. I had a hard time understanding the HPaste [4] code written using HAppS and I don't fancy using XSLT as my HTML template language. HOPE has a publishing framework (blog, RSS, comments). I could write the forums or just use Google Groups. I like the module system and the FCGI interface should let me couple HOPE with Lighty [5]. The HOPE code base looks clean and simple and I'm guessing that adding Lambdabot-style dynamic loading of modules would be straightforward. Overall, HOPE has everything that I need, without unnecessary extras. It does not have an interface to Amazon S3, though, and I would need that to rewrite the storage bits to use S3 instead of a regular database. Lambdabot by itself doesn't offer much apart from robust dynamic loading of Haskell code. Its features have been heavily tested, though, which gives me a warm feeling. I'm thinking that the same hot- loading technology that Lambdabot uses could be coupled with either HAppS or HOPE. Lambdabot has a FCGI which should make mixing it with HOPE particularly easy. Going with Ruby and Rails I will have everything I need and more. The compiler could run in a separate server (think Lambdabot) but I would need to manage two different platforms. Is my assessment correct? Did I miss anything? What approach do you favor? Thanks, Joel P.S. I have the same compiler written in OCaml which is what I started with. Developing it has been a relative breeze but deploying on the web was a tale of frustration. I gave up and finished the Haskell version. [1] http://lambda-the-ultimate.org/node/2201 [2] http://happs.org/HAppS/README.html [3] http://hope.bringert.net/about [4] http://hpaste.org [5] http://www.lighttpd.net/ -- http://wagerlabs.com/

From: haskell-cafe-bounces@haskell.org [mailto:haskell-cafe-bounces@haskell.org] On Behalf Of Joel Reymont
I have finished an alpha version of my EasyLanguage [1] to C# compiler and need to deploy it on Amazon EC2/S3.
My choice seems to boil down to HAppS [2], HOPE [3], or a combination of Ruby/Rails with Lambdabot.
Lambdabot by itself doesn't offer much apart from robust dynamic loading of Haskell code
Is my assessment correct? Did I miss anything?
Depends. Did you leave out WASH intentionally? http://www.informatik.uni-freiburg.de/~thiemann/WASH/ Note that Adam Peacock recently built a web-app with WASH, so there's a real live example to eyeball: http://www.haskell.org/pipermail/haskell-cafe/2007-April/024127.html http://article.gmane.org/gmane.comp.lang.haskell.cafe/21159 Regarding Lambdabot, if dynamic loading is all you're after then you'd be better off learning how to use hs-plugins and rolling your own. It's pretty simple. For dynamic-loading-application design ideas, I suggest reading this Yi paper: http://www.cse.unsw.edu.au/~dons/papers/SC05.html Alistair ***************************************************************** Confidentiality Note: The information contained in this message, and any attachments, may contain confidential and/or privileged material. It is intended solely for the person(s) or entity to which it is addressed. Any review, retransmission, dissemination, or taking of any action in reliance upon this information by persons or entities other than the intended recipient(s) is prohibited. If you received this in error, please contact the sender and delete the material from any computer. *****************************************************************

On May 10, 2007, at 10:01 AM, Bayley, Alistair wrote:
Depends. Did you leave out WASH intentionally? http://www.informatik.uni-freiburg.de/~thiemann/WASH/
Nope, I forgot about it but looked at the Hemp app this morning. Thanks, Joel -- http://wagerlabs.com/

Regarding Lambdabot, if dynamic loading is all you're after then you'd be better off learning how to use hs-plugins and rolling your own. It's pretty simple. For dynamic-loading-application design ideas, I suggest reading this Yi paper: http://www.cse.unsw.edu.au/~dons/papers/SC05.html
hs-plugins? Mmm... what an interesting concept. I've crawled all over the webpage, and I can't see any documentation anywhere to this effect, but presumably all this dynamic goodness only works if GHC is installed, right? BTW... Does lambdabot seriously take every expression you type and compile it to a binary executable, dynamically link it, and execute it? I was expecting it to just run a simple source-level interpreter...

On May 10, 2007, at 11:37 , Andrew Coppin wrote:
I've crawled all over the webpage, and I can't see any documentation anywhere to this effect, but presumably all this dynamic goodness only works if GHC is installed, right?
Yes, it's GHC-specific.
BTW... Does lambdabot seriously take every expression you type and compile it to a binary executable, dynamically link it, and execute it? I was expecting it to just run a simple source-level interpreter...
There ain't no such thing, and if it were it'd be painful at best. Future versions of lambdabot *might* use ghc-api instead, but yes, it really does compile and link a program. I think that allows it to exercise tighter control over what is permitted to execute, i.e. hide everything capable of I/O actions. -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH

I've crawled all over the webpage, and I can't see any documentation anywhere to this effect, but presumably all this dynamic goodness only works if GHC is installed, right?
Yes, it's GHC-specific.
I can see it's GHC-specific, what I was asking is does the computer than runs the final program need to have GHC installed. Presumably is does if it's going to compile files on the fly. What about if it only loads *.o files that are already compiled? Is GHC still required? (Not that the answer worries me greatly; I'm just curiose.)
BTW... Does lambdabot seriously take every expression you type and compile it to a binary executable, dynamically link it, and execute it? I was expecting it to just run a simple source-level interpreter...
There ain't no such thing, and if it were it'd be painful at best. Future versions of lambdabot *might* use ghc-api instead, but yes, it really does compile and link a program. I think that allows it to exercise tighter control over what is permitted to execute, i.e. hide everything capable of I/O actions.
Wait a sec... Are you trying to tell me that it is *faster* to take the source, type check it, convert it to Core, perform 25,000 Core-to-Core transformations, convert Core to C, call GCC, link the result together, dynamically load it, execute it, extract the result and confirm that it type checks, and display it....... then it is to just directly execute an interpreted expression? I find that highly surprising. (I don't have the source any more, but a while back I wrote a short Tcl script that takes some function definitions and an expression and interactively step-evaluates it. Sure, it didn't support every syntactic construct that Haskell has, but it supported enough that a vast number of Haskell functions could be implemented. And evaluation was instantaneous. Even though the parser was written in Tcl.)

On May 10, 2007, at 12:14 , Andrew Coppin wrote:
I can see it's GHC-specific, what I was asking is does the computer than runs the final program need to have GHC installed. Presumably is does if it's going to compile files on the fly. What about if it only loads *.o files that are already compiled? Is GHC still required? (Not that the answer worries me greatly; I'm just curiose.)
You would need the object files and archives comprising the Haskell runtime, and you'd need to run a ghc link in verbose mode to figure out how to use them and write a script to do the link.
Wait a sec... Are you trying to tell me that it is *faster* to take the source, type check it, convert it to Core, perform 25,000 Core- to-Core transformations, convert Core to C, call GCC, link the result together, dynamically load it, execute it, extract the result and confirm that it type checks, and display it....... then it is to just directly execute an interpreted expression? I find that highly surprising.
It's pretty fast, actually. But your interactive expression still needs to do most of that, except it compiles cmm to bytecode instead of to C (old default; handed off to gcc afterward) or directly to asm (modern ghc). In particular, you are doing both core transformations and type checking no matter what --- and they take *far* more time than the compile and link. Truly. Plus, consider that people often throw extensions at lambdabot --- do you support even simple stuff like forall in your interpreter? Using ghc means you can use most of the ghc extensions. -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH

I can see it's GHC-specific, what I was asking is does the computer than runs the final program need to have GHC installed. Presumably is does if it's going to compile files on the fly. What about if it only loads *.o files that are already compiled? Is GHC still required? (Not that the answer worries me greatly; I'm just curiose.)
You would need the object files and archives comprising the Haskell runtime, and you'd need to run a ghc link in verbose mode to figure out how to use them and write a script to do the link.
Uh... right. So basically you need GHC. ;-)
Wait a sec... Are you trying to tell me that it is *faster* to take the source, type check it, convert it to Core, perform 25,000 Core-to-Core transformations, convert Core to C, call GCC, link the result together, dynamically load it, execute it, extract the result and confirm that it type checks, and display it....... then it is to just directly execute an interpreted expression? I find that highly surprising.
It's pretty fast, actually. But your interactive expression still needs to do most of that, except it compiles cmm to bytecode instead of to C (old default; handed off to gcc afterward) or directly to asm (modern ghc). In particular, you are doing both core transformations and type checking no matter what --- and they take *far* more time than the compile and link. Truly.
Interesting. On my system, the link stage typically takes far longer than compilation. (Compilation takes 4 - 5 seconds per module, linking takes another 15 seconds or so. Depending on how big the program is. Use -O2 to make the compilation stage take round about 30 seconds per module instead.)
Plus, consider that people often throw extensions at lambdabot --- do you support even simple stuff like forall in your interpreter? Using ghc means you can use most of the ghc extensions.
Ah, yes, well, I avoid everything that isn't in Haskell 98. ;-) (Mainly because 1. most of it makes no semblence of sense, and 2. why do we need any of this stuff anyway?) FWIW (again, I don't have the code any more) my interpreter had no type checker, but would gracefully handle runtime errors. (Typically if the code isn't well-typed, that means it hits some constructor the function isn't expecting, and dies that way.) Very useful and instructive for seeing how stuff works; I discovered first-hand why foldl' exists!

On May 10, 2007, at 12:52 , Andrew Coppin wrote:
Plus, consider that people often throw extensions at lambdabot --- do you support even simple stuff like forall in your interpreter? Using ghc means you can use most of the ghc extensions.
Ah, yes, well, I avoid everything that isn't in Haskell 98. ;-)
(Mainly because 1. most of it makes no semblence of sense, and 2. why do we need any of this stuff anyway?)
FWIW (again, I don't have the code any more) my interpreter had no type checker, but would gracefully handle runtime errors. (Typically if the code isn't well-typed, that means it hits some constructor the function isn't expecting, and dies that way.) Very useful and instructive for seeing how stuff works; I discovered first-hand why foldl' exists!
Quite a lot of what people throw at lambdabot in #haskell is intended to do type checking or type inference. Maybe you have different needs, and would be better served by something else --- but LB is what it is for good reason. -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH

On Thu, May 10, 2007 at 05:52:55PM +0100, Andrew Coppin wrote:
Wait a sec... Are you trying to tell me that it is *faster* to take the source, type check it, convert it to Core, perform 25,000 Core-to-Core transformations, convert Core to C, call GCC, link the result together, dynamically load it, execute it, extract the result and confirm that it type checks, and display it....... then it is to just directly execute an interpreted expression? I find that highly surprising.
It's pretty fast, actually. But your interactive expression still needs to do most of that, except it compiles cmm to bytecode instead of to C (old default; handed off to gcc afterward) or directly to asm (modern ghc). In particular, you are doing both core transformations and type checking no matter what --- and they take *far* more time than the compile and link. Truly.
Interesting. On my system, the link stage typically takes far longer than compilation. (Compilation takes 4 - 5 seconds per module, linking takes another 15 seconds or so. Depending on how big the program is. Use -O2 to make the compilation stage take round about 30 seconds per module instead.)
1. Lambdabot is running on a fairly fast machine - idle 2Ghz p4. 2. runplugs uses GHCi's custom dynamic linker which can load .o files directly, so the link phase simply doesn't exist. 3. most user expressions are very short, and the compiler proper runs in approximate linear time. 4. IRC lag is on the order of 2-10 seconds, so people don't notice the 1.5s it takes to compile. Also, they can't hear Don's fans. 5. Lambdabot uses -O2, which makes it an order of magnitude faster than GHCi on longer running expressions. Stefan

Hi
Wait a sec... Are you trying to tell me that it is *faster* to take the source, type check it, convert it to Core, perform 25,000 Core-to-Core transformations, convert Core to C, call GCC, link the result together, dynamically load it, execute it, extract the result and confirm that it type checks, and display it....... then it is to just directly execute an interpreted expression? I find that highly surprising.
No, you can do the GHCi trick, converting it to Core, perform a small number of Core-to-Core transformations, convert it to bytecode, interpret the bytecode. Compare this to the programmer time to implement directly executing an interpetted expression, and it starts to get complex. Also remember that evaluating an expression in Haskell is _really_ hard! The value depends on the types, there is loads of sugar, loads of libraries etc. A Haskell compiler is a very hard thing to write, an interepretter doesn't save much of that complexity. Thanks Neil

No, you can do the GHCi trick, converting it to Core, perform a small number of Core-to-Core transformations, convert it to bytecode, interpret the bytecode. Compare this to the programmer time to implement directly executing an interpetted expression, and it starts to get complex.
One of the things on the long list of questions I haven't got round to asking yet is "is there some library that lets you execute arbitrary Haskell code from a string?" Clearly GHCi manages to do it without difficulty. Apparently hs-plugins allows this too, but... surely this is a sledgehammer to crack a nut? A Haskell program is just a set of graph rewrite rules, after all...
Also remember that evaluating an expression in Haskell is _really_ hard!
Really? Looks pretty damn simple to me...
The value depends on the types, there is loads of sugar, loads of libraries etc. A Haskell compiler is a very hard thing to write, an interepretter doesn't save much of that complexity.
Oh, yeah, well, definitely the hardest part is *parsing* Haskell. In particular, the fact that layout exists makes this problem essentually unsolvable. If I could figure out how to parse Haskell, executing it wouldn't be too hard. (Unless you wanted it to actually be capable of running *arbitrary* programs - ones that use FFI, concurrency, mutable state, exceptions, graphics, etc. Or if you want it to be *efficient*. If you just want to be able to run pure functional code and have it produce a correct result, that's quite easy.)

Hi Andrew,
Also remember that evaluating an expression in Haskell is _really_ hard!
Really? Looks pretty damn simple to me...
In that case I throw down the challenge of writing an interpetter that takes a Haskell syntax tree and evaluates it :) What is the value of show [] ? Remember that show ([] :: String) /= show ([] :: [Bool), so I can't see how you can drop the types and keep the semantics. I would love to write a Haskell interpretter which didn't deal with types (crashing at runtime for ill-typed is fine) - unfortunately I couldn't make it work because of this. If you figured it out, please share!
Oh, yeah, well, definitely the hardest part is *parsing* Haskell. In particular, the fact that layout exists makes this problem essentually unsolvable.
I'd say the type checking was much harder. The problem of desugaring is broad, but in most cases isn't that complex. Thanks Neil

Also remember that evaluating an expression in Haskell is _really_ hard!
Really? Looks pretty damn simple to me...
In that case I throw down the challenge of writing an interpetter that takes a Haskell syntax tree and evaluates it :)
I'm currently in the process of attempting to write such a thing...
What is the value of show [] ?
That is indeed a difficult point. Since I want an interpreter so I can step through the code interactively, I was just going to make the interpreter *ask* me which implementation to call...
Oh, yeah, well, definitely the hardest part is *parsing* Haskell. In particular, the fact that layout exists makes this problem essentually unsolvable.
I'd say the type checking was much harder. The problem of desugaring is broad, but in most cases isn't that complex.
I did wonder how in the name of God the Haskell type inference thingy actually works. I mean, clearly it *should* be possible to infer the type - I can do it - but how on earth do you implement such a thing on a computer? And then I read The Fun of Programming... and I found out. ;-) On the other hand, parsing Haskell input is intractably hard. Whitespace actually matters, which makes the program to parse Haskell horrendusly complex. Several times I've been forced to give up due to the sheer complexity of it all.

On Thu, May 10, 2007 at 07:24:51PM +0100, Andrew Coppin wrote:
On the other hand, parsing Haskell input is intractably hard. Whitespace actually matters, which makes the program to parse Haskell horrendusly complex.
Do you know about the algorithm for converting Haskell source into a whitespace-insignificant sublanguage? It's in section 9.3 of http://haskell.org/onlinereport/syntax-iso.html Thanks Ian

Hi
I'm currently in the process of attempting to write such a thing...
Good luck! Haskell needs more implementations to retain its purity and compatibility.
What is the value of show [] ?
That is indeed a difficult point. Since I want an interpreter so I can step through the code interactively, I was just going to make the interpreter *ask* me which implementation to call...
Yuk! I couldn't think of a better answer, which is what stopped me from writing such an evaluator.
On the other hand, parsing Haskell input is intractably hard. Whitespace actually matters, which makes the program to parse Haskell horrendusly complex. Several times I've been forced to give up due to the sheer complexity of it all.
Why not just reuse the standard Haskell parser? Its in Hacle (google keyword: Hacle) amongst other places. It parses all of Haskell, and is pretty easy to use. Thanks Neil

On Thu, May 10, 2007 at 06:13:16PM +0100, Neil Mitchell wrote:
Also remember that evaluating an expression in Haskell is _really_ hard!
Really? Looks pretty damn simple to me...
In that case I throw down the challenge of writing an interpetter that takes a Haskell syntax tree and evaluates it :)
What is the value of show [] ? Remember that show ([] :: String) /= show ([] :: [Bool), so I can't see how you can drop the types and keep the semantics.
show [] by itself can't be evaluated even with type inference, though. A more convincing example, IMO, is something like class Foo a where foo :: a -> String instance Foo Bool where foo _ = "Bool" instance Foo Char where foo _ = "Char" x :: String x = foo (id $! (undefined :: Bool)) where to evaluate x you can't evaluate the argument to foo (in general it might not terminate) in order to find out what type this instance of foo needs to have.. Thanks Ian

Parsing a very close approximation to what Haskell specifies isn't that hard. You just need some mild interaction between the parser and lexer. Executing Haskell is more difficult, since you can't execute Haskell without first doing type checking. -- Lennart On Thu, 10 May 2007, Andrew Coppin wrote:
Date: Thu, 10 May 2007 17:59:38 +0100 From: Andrew Coppin
To: haskell-cafe@haskell.org Subject: Re: [Haskell-cafe] Picking an architecture for a Haskell web app No, you can do the GHCi trick, converting it to Core, perform a small number of Core-to-Core transformations, convert it to bytecode, interpret the bytecode. Compare this to the programmer time to implement directly executing an interpetted expression, and it starts to get complex.
One of the things on the long list of questions I haven't got round to asking yet is "is there some library that lets you execute arbitrary Haskell code from a string?" Clearly GHCi manages to do it without difficulty. Apparently hs-plugins allows this too, but... surely this is a sledgehammer to crack a nut? A Haskell program is just a set of graph rewrite rules, after all...
Also remember that evaluating an expression in Haskell is _really_ hard!
Really? Looks pretty damn simple to me...
The value depends on the types, there is loads of sugar, loads of libraries etc. A Haskell compiler is a very hard thing to write, an interepretter doesn't save much of that complexity.
Oh, yeah, well, definitely the hardest part is *parsing* Haskell. In particular, the fact that layout exists makes this problem essentually unsolvable. If I could figure out how to parse Haskell, executing it wouldn't be too hard. (Unless you wanted it to actually be capable of running *arbitrary* programs - ones that use FFI, concurrency, mutable state, exceptions, graphics, etc. Or if you want it to be *efficient*. If you just want to be able to run pure functional code and have it produce a correct result, that's quite easy.)
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
-- Lennart

On Fri, May 11, 2007 at 12:00:10AM +0200, Lennart Augustsson wrote:
Parsing a very close approximation to what Haskell specifies isn't that hard. You just need some mild interaction between the parser and lexer.
This algorithm can correctly de-whitespace haskell without any parser interaction. It works for all the code in the wild I tried it on. Only 50 lines long or so to boot. One can come up with pathological cases it doesn't get right, but one can do the same for ghc as well :) http://repetae.net/repos/getlaid/ John -- John Meacham - ⑆repetae.net⑆john⑈
participants (9)
-
Andrew Coppin
-
Bayley, Alistair
-
Brandon S. Allbery KF8NH
-
Ian Lynagh
-
Joel Reymont
-
John Meacham
-
Lennart Augustsson
-
Neil Mitchell
-
Stefan O'Rear