Hi Michael, Thanks for the information.

Essentailly, in a reverse HTTP, you'll have a standalone server running on some port that never talks to the outside world, and Apache will proxy all HTTP connections for your application to that server.

I hadn't heard of Reverse HTTP proxy before, but it seems to be the simplest option for me - as I'll want Hoogle to be able to run as a web server anyway. Taking a look it seems easy enough to configure in Apache, so seems the perfect answer.

There are at this point three main web frameworks in Haskell (that I'm aware of, someone correct me if I'm wrong): Yesod[1], Happstack[2] and Snap[3].

I've seen and looked at all the three main options. The one that seemed to appeal to me most was Wai+Warp - does that count as Yesod? One question on Wai - I see version 0.3.0 says "For that reason, this library includes Network.Wai.Enumerator" - but the .cabal file doesn't mention that file, and it's not included. Did you forget to edit the .cabal file? I think the use of LazyIO might appeal to me, as that is how Hoogle is currently structured, and I'd rather not switch to enumerators at this stage.

All of your desired features are easily implemented in Yesod except the timeout one.

I can implement timeout within my response easily enough, so I don't think that matters too much.

If you are interested in trying a migration to Yesod, just send me an email (either privately or on web-devel).

I think it's well documented enough that I'm not sure I'd have to. I've downloaded warp and will give it a try shortly. Thanks, Neil

Hi, I tried switch Hoogle to Wai and Warp today. The results were good, and once a few little things are cleared up I'm fairly certain that the the next Hoogle release will be based on them. As I wrote the code, I made some notes: 1) Why not add statusOK as an alias for status200? As someone who doesn't spend all day dealing with HTTP response codes, statusOK is far better documenting - similarly for all the other status values. I added statusOK as a local definition in Hoogle. 2) Changing from HTTP to Wai, it was a shame that I had to define all the header keys myself - could they be included - it would have better performance, and give static guarantees. I added locally: hdrContentType = mkCIByteString "Content-Type" hdrCacheControl = mkCIByteString "Cache-Control" 3) The documentation for ResponseFile doesn't say what happens if the file is not found. I am sure it must throw an exception, but for a very brief second I wondered if it might change my status code to 404 on my behalf. Could the documentation be clarified. 4) I defined a Wai handler that threw an error, and Warp did nothing. Should it? Or should I just be careful and include my own exception handling? I'm not really sure what the right thing to do is - silently eating the error isn't great, but bring down my server would be even less appreciated. 5) Hoogle is currently written as lazy IO + String. Converting this format to a response was easy with liftIO, LBS.pack and responseLBS. Converting back was rather tricky. I eventually came up with: responseFlatten :: Response -> IO (Status, ResponseHeaders, LBS.ByteString) responseFlatten r = responseEnumerator r $ \status headers -> let f res E.EOF = return (status, headers, Blaze.toLazyByteString res) f res (E.Chunks xs) = E.continue (f $ mconcat $ res:xs) in E.continue (f mempty) It's not too long, but it has a relatively high level of complexity. Is there something simpler I could have done? 6) I want responseFlatten for two purposes, the first of which is to output a CGI representation of the Response, which is easy with the above function. However, outputting a response in CGI format is trivial, and is probably the easiest way to debug/view the contents of a Response. Would it be worth adding showResponse :: Response -> IO LBS.ByteString to the Wai library, which produced the result representing how it would be sent in CGI format, and was also useful for debugging? 7) The second reason for using responseFlatten is that I want to write a function Response -> IO Response which modifies a response to rewrite "href='file://" to "href='/file/". I currently do this with responseFlatten, converting to a String, and then stepping through it Char by Char with isPrefixOf. Given the performance focus of this email list, I can imagine several of you are now crying. What would have been the Iterator/Builder approved way to do it? 8) The last point is the only point that is a blocker for Hoogle, and regards the Warp server, Windows and ghci. In ghci on Windows, running the network function accept doesn't terminate if you hit Ctrl+C. If you run Warp in the main thread ghci will never respond to you again. If you run Warp on a second thread then hitting Ctrl+C twice returns to the prompt, but the thread serving the web page is still running (and still serving web pages), and if you run the server again the old server continues to serve the pages instead. I develop Hoogle on Windows through ghci in server mode, so without a workaround it's fairly fatal. When writing Hoogle, I found that even though accept doesn't terminate on exceptions, if you do sClose on the socket it raises an exception in accept and does terminate. I experimented in a local copy of Warp and found the following worked very well: import Control.Concurrent import Control.Monad run :: Port -> Application -> IO () run port app = withSocketsDo $ do var <- newMVar Nothing let clean = modifyMVar_ var $ \s -> maybe (return ()) sClose s >> return Nothing forkIO $ bracket (listenOn $ PortNumber $ fromIntegral port) (const clean) (\s -> do modifyMVar_ var (\_ -> return $ Just s); serveConnections port app s) forever (threadDelay maxBound) `finally` clean If you hit Ctrl+C in ghci on Windows you get the message "Network.Socket.accept: failed (No error)" - but it terminates perfectly. That error could even be dropped silently if you reach the `finally` clause. Can this code be included in Warp? Thanks, Neil

Hi Michael,

...

1) Why not add statusOK as an alias for status200?

The main reason I stuck with numerical values was the redirects: there is a bit of ambiguity about the right name for 302, 303 and 307. (Yes, I know that the specs give a name for these, but in practice they are treated slightly different.) Frankly, I had always intended WAI to be something low-level that a framework would wrap around. It's a new phenomenon that there is interest in writing directly against WAI.

I think WAI could be increasingly used - its at a level of abstraction that I quite like. I certainly have other projects which I will probably switch over to WAI/Warp.

...

2) Changing from HTTP to Wai, it was a shame that I had to define all the header keys myself - could they be included - it would have better performance, and give static guarantees. I added locally:

hdrContentType = mkCIByteString "Content-Type" hdrCacheControl = mkCIByteString "Cache-Control"

Firstly, CIByteString is an instance of IsString, so you can just write "Content-Type".

My main concern is the static safety. I know about fromString, and could write "Content-Type", but I could equally write "Contont-Type" and never notice. I realise there is an issue with adding lots of headers, and there are always more headers, so I'm happy to leave the 4 or so I send defined locally.

...

4) I defined a Wai handler that threw an error, and Warp did nothing. Should it? Or should I just be careful and include my own exception handling? I'm not really sure what the right thing to do is - silently eating the error isn't great, but bring down my server would be even less appreciated.

There's already a request in place to allow you to provide a callback function for exceptions. However, I personally prefer having the application catch all exceptions (which is what Yesod does in practice).

I'm not overly fussed - in my custom web server I just printed exceptions to stdout, but that's probably not a great idea for a more general server.

You could probably write something simpler using the consume iteratee from the enumerator package. It's late at night here, so the following may or may not fire missiles:

   responseEnumerator r $ \s hs -> do        builders <- consume        return (s, hs, Blaze.toLazyByteString $ mconcat builders)

That looks much simpler. Is there any chance of adding some operation like responseFlatten to WAI? It nicely mirrors the responseLBS, and would make it very easy to transition from HTTP - although it's easy enough to write this function locally.

...

above function. However, outputting a response in CGI format is trivial, and is probably the easiest way to debug/view the contents of

I think such a function would make more sense for the wai-extra package. I already have a debug middleware over there, that kind of function could be a good addition.

wai-extra introduces the zlib-bindings dependency, so I wouldn't make use of it in wai-extra, given it's just for CGI output, and I'm not intending anyone to use CGI. But it would be a useful function to be around, and I may want to use the gzip middleware at some point anyway, in which case I would use it.

Builders. You'll probably be best off mconcat'ing your Builders into a single Builder, converting that to a lazy ByteString and performing the conversion there.

Thanks, with your far cleaner flattenResponse code I feel far less dirty switching back to a bytestring.

...

In ghci on Windows, running the network function accept doesn't terminate if you hit Ctrl+C. If you run Warp in the main thread ghci will never respond to you again.

I've also been using Warp for development purposes at work on my Windows machine, and have run into the Ctrl-C issue. I'll review this patch after some sleep. My initial reaction is to include it conditionally for Windows. My guess is that very few people (if any) are planning on running production websites using Windows/Warp, so adding something which enhanced usability at the expense of a few cycles would be a general win. However, for the speed-crazed masses serving from Linux (eg, you and me apparently), I'd like to avoid any extra code.

I'm a big fan of keeping Windows and Linux code as identical as possible, but am happy as long as it works. I can't imagine forking a single thread at startup would harm performance at all, but don't mind either way. Thanks, Neil

On Tue, Jan 25, 2011 at 3:14 AM, Jeremy Shaw wrote:

On Mon, Jan 24, 2011 at 4:38 PM, Neil Mitchell wrote:

...

I think WAI could be increasingly used - its at a level of abstraction that I quite like. I certainly have other projects which I will probably switch over to WAI/Warp.

I am curious to see how your feelings on this change or stay the same over time.

The basic WAI abstraction is very nice. You essentially supply a function, Request -> IO Response, that will be used to handle incoming Requests. It is very straight-forward and obvious what is going on. At first this is very attractive. It makes it very easy to get started.

But at the same time, it doesn't do much for you.. I wonder if you will ultimately end up recreating a higher level framework in your WAI based applications. For example, you might decide that passing the Request around explicitly is tiresome and stick it in a Reader monad. When looking up key/value pairs from the query string or post data, you might want more expressive combinators for validating the values and converting the string data to haskell types. You might find that sometimes you need to escape early and return an alternate page. You might realize that the Response is going to need some extra headers (like the Cookie field) but be in a spot in your code where the Response has not been created yet. So you will need a filter, or something, that can add the header later when the Response is created. And, of course, you need some way to look at the pieces of the url and map it to a handler.

These are the types of features (and more) that exist in the higher-level frameworks like Happstack, Snap, and Yesod. In theory, that is the value they add over just using WAI. It is likely that, in the future, happstack will not include it's own http backend, but be a layer on top of WAI+warp, hyena, or some other low-level WAI-like HTTP backend. So, I am curious if the value-added features we provide actual provide value :)  One measure of this will be if you end up creating similar adhoc functions and structure in your apps.

- jeremy

I personally agree with this sentiment to some extent: those who don't use a framework will be doomed to reinvent one. Nonetheless, I see two very compelling reasons for people to skip the frameworks entirely: * They are writing a site that involves very few of the features offered by a framework. For example, no routing, no persistence, no templating. In such a case, it's likely that a framework will simply get in their way. * Haskellers tend to be an "opinionated" bunch, so it's not surprising that any random Haskeller may disagree with the design decisions of specific frameworks. Can you honestly tell me that you think Happstack is that right tool for every site? I can definitely say that- as much as I like Yesod- there are use cases that it simply is not intended for. Even if I *personally* would likely still use Yesod for those, that's just because I'm so intimately familiar with the framework that I know all the right places to make modifications to get it to do exactly what I want. For someone coming from the outside, I think skipping the framework in those cases is the right decision. That's one of the main reasons I've designed Yesod and WAI the way I have. By splitting the project up into so many pieces (Hamlet, Persistent, authenticate, etc), users get to pick and choose exactly what they want. Sure, as a framework author I would prefer them using the whole stack, but as someone trying to push Haskell into more mainstream use[1], giving people choice here is best. It's very likely that Neil may end up reinventing a lot of the features of a framework in this Hoogle project. But I *still* think this is a good thing: * He's now providing valuable insight into how WAI can be improved for this new use case for which it was not designed. * If he is so inclined, he could release this "boilerplate" code into a separate package and, walla, we have a *new* Haskell framework, building on top of what we have already, that caters to a new audience. * And even if none of that happens, it probably took him less time to rewrite the features he needed than go through either the Happstack or Yesod tutorials. For someone developing lots of sites, the learning curve is IMO worth it; for a one-off project, I doubt it is. I don't like seeing a fractured community, but I think this is exactly the opposite effect: we're seeing new levels of integration of pre-existing tools than we saw before. Michael [1] Sorry to those offended by the thought of mainstream Haskell. :)

On Tue, Jan 25, 2011 at 7:14 PM, Jeremy Shaw wrote:

On Jan 24, 2011, at 11:54 PM, Michael Snoyman wrote:

* They are writing a site that involves very few of the features offered by a framework. For example, no routing, no persistence, no templating. In such a case, it's likely that a framework will simply get in their way.

Yes. I think that is a pretty small subset of web applications?

Yes, I agree.

* Haskellers tend to be an "opinionated" bunch, so it's not surprising that any random Haskeller may disagree with the design decisions of specific frameworks.

I think that many random Haskellers start by assuming their ideas are better than everyone else's and don't even look to see if they disagree with the decisions of specific frameworks :p

I also agree. Or was that poking fun at a specific framework? ;)

Can you honestly tell me that you think Happstack is that right tool for every site?

Not for every site. But possibly for every site that Neil is going to be working on. Many of his 'complaints' about WAI are things which are addressed by Happstack to some degree: 1. statusOK vs status200 - happstack offers combinators like, ok, notFound, seeOther, unauthorized, etc. and a function 'resp' for setting the response code explicitly.  2. predefined list of headers like "content-type" - happstack includes many constants an Handler.hs (they could be exposed better though)  3. what happens when using 'ResponseFile' but the file is not found - happstack's documentation says what happens when serveFile or serveDirectory does not find the file  4. exception handling -- the happstack docs show how to integrate exception catching and handling into your ServerPartT (though they could be more explicit about what happens to uncatch exceptions)

Let's be honest here: none of those are "serious" problems with WAI: docs need to be updated, function names are up for debate.

 5. happstack-server currently supports lazy IO natively and should be a good match

WAI also supports lazy IO (responseLBS).

 6. happstack has a showResponse function for debugging

True.

 7. the ServerPartT monad has a built-in FilterT monad which makes it easy to apply a function (Response -> Response) to modify the response.

In all honesty, this is an argument *against* frameworks, not for them: at a WAI level, we don't need a FilterT monad, since an application is just a function. (And I can't believe I'm giving arguments against frameworks here, I obviously think they're a good thing. I'm just trying to be fair.)

 8. killing the running background threads in GHCi is really tricky. You need to basically track all the forkIO calls and save the threadIds so that you can come back and kill them  later when you get Crtl+C. But, happstack does have support for doing that :) Also, the waitForTermination function that happstack provides also has support for Windows. So, given the fact that all the points Neil raised regarding WAI are addressed to some degree already in Happstack -- it seems like working with Happstack would not be a bad thing..

Personally: I think overall it *would* be a better approach. However, remember that to each his own. My goal is to enable *whatever* workflow someone wants to take. If there is a desire for a bare-metal web programming approach using WAI (or something else), I'd like to help facilitate it.

That's one of the main reasons I've designed Yesod and WAI the way I have. By splitting the project up into so many pieces (Hamlet, Persistent, authenticate, etc), users get to pick and choose exactly what they want.

Yes. Happstack is the same way.

Yes, and I'm a fan of that. I just wish that other frameworks (cough, cough) could take a similar approach.

It's very likely that Neil may end up reinventing a lot of the features of a framework in this Hoogle project. But I *still* think this is a good thing:

Based on my familiarity with Neil's previous work on things like catch, super-o, and other projects, I am interested to see what he comes up with as well. He has a tendency to address important issues that other people have swept under the rug.

* And even if none of that happens, it probably took him less time to rewrite the features he needed than go through either the Happstack or Yesod tutorials. For someone developing lots of sites, the learning curve is IMO worth it; for a one-off project, I doubt it is.

I highly doubt that. It can't take more than an hour or two to go through the happstack tutorial. What I do think is true is that:  1. writing code is satisfying and feels productive  2. reading documentation is boring and doesn't feel productive  3. most programmers assume their code is like a  beautiful flower, and that everyone else's is a putrid swamp of sewage. And hence they are happy to spend days reinventing something rather than spend an hour reading about an existing solution :p (This is not directed at Neil who actually asked for advice about what already existed. Though he may still feel that Happstack is a putrid swamp of sewage...)

Probably also very true. (Not the putrid swamp of sewage bit, the touchy-feely stuff.)

* If he is so inclined, he could release this "boilerplate" code into a separate package and, walla, we have a *new* Haskell framework, building on top of what we have already, that caters to a new audience.

...

type FrameworkT = ReaderT Request m a Then you'll realize you need some way to escape things early and return an error, so you might add the ErrorT monad transformer so you have a way to escape and return a Response. type FrameworkT = ReaderT Request (ErrorT Response m) a Then you'll realize you need some way to modify the Responses to do things

...

type FrameworkT = ReaderT Request (ErrorT Response (StateT (Response -> Response) m)) a

setFilter :: (Response -> Response) -> FrameworkT m a appendFilter :: (Response -> Response) -> FrameworkT m a And then you might decide that you want your FrameworkT monad to be an instance of MonadPlus, so you add in MaybeT: type FrameworkT = ReaderT Request (ErrorT Response (StateT (Response -> Response) (MaybeT m))) a And now you have created a new Framework monad that looks suspiciously like

Or, maybe it just reimplements what we already have with minor cosmetic differences, weaknesses compared to other frameworks due to have less real world usage, and a few improvements which could have been more easily added to existing frameworks. For example, let's consider what a framework is likely going to have for it's monad. For starters you are going to want easy access to the Request. So you'll have something like this: like add extra headers (for cookies, etc) or compress the Response, or whatever. So you might had a StateT that can hold filters to apply: the ones that already exist in other frameworks. Happstack, Snap, and Yesod all have variations on this basic theme. For example, in Yesod it appears you ultimately came up with,

type GHInner s m = ReaderT (HandlerData s m) ( MEitherT HandlerContents ( WriterT (Endo [Header]) ( StateT SessionMap ( -- session IO ))))

:p We don't really need anyone to reinvent another variation of this out of ignorance. They should either blatantly rip it off, or tell us why it is wrong and come up with something better :p

I do not disagree here either.

I don't like seeing a fractured community, but I think this is exactly the opposite effect: we're seeing new levels of integration of pre-existing tools than we saw before.

Oh ? Anyway, I am not against the development of new frameworks. But I think it would be better if people started new frameworks because they have discovered why the current ones are fundamentally broken, rather than just assuming they are and then reinventing the same thing :) But, in practice, people are going to keep reinventing the wheel, so I'll just keep my eye on them and steal the good parts ;) - jeremy p.s. Nothing in this message is intended to be a personal attack on anyone specific. It more reflects years of hearing people say, "Oh I didn't realize happstack already could do that."

For the record, I didn't realize that Happstack could actually serve web pages. I thought it could just create BBS games. :-D Michael

On Tue, Jan 25, 2011 at 8:29 PM, Chris Smith wrote:

On Tue, 2011-01-25 at 20:10 +0200, Michael Snoyman wrote:

...

...

That's one of the main reasons I've designed Yesod and WAI the way I have. By splitting the project up into so many pieces (Hamlet, Persistent, authenticate, etc), users get to pick and choose exactly what they want.

Yes. Happstack is the same way.

Yes, and I'm a fan of that. I just wish that other frameworks (cough, cough) could take a similar approach.

What parts of Snap would you like to see split out into separate projects?  Heist is completely independent of Snap.  And snap-server is already a separate package from snap-core that handles the low-level HTTP stuff, though perhaps the module boundary isn't exactly where and like you're accustomed to.  As of now, the Snap project doesn't really have an endorsed answer for persistence; if Snap ever decides that it needs its own new entry into persistence frameworks (which I don't yet see a reason for; there's really no need for all of us to reinvent everything!), I have no doubt it'll be split into its own package as well.

Granted, there is an umbrella mechanism for pulling these things together and making them interoperate in the same application -- that's the extensions mechanism.  But I'd consider it a flaw if an extension actually contained substantial independent ideas; the model Snap is following thus far is to provide functionality with a natural interface, and *also* an extension to help integrate it together.

Just my thoughts; not speaking for anyone else!

Sorry, you're right: Snap *has* done a good job of releasing a number of auxiliary packages: attoparsec-iteratee, Heist, the upcoming xmlhtml package. I should not have implied otherwise. Without reopening an old debate, I'm referring to the avoidance of WAI. Also, I've heard a number of people (yourself included) touting how good the routing mechanism is in Snap. I know routing is fairly central to a framework, so it may not be possible to separate out. But Happstack and Yesod are both able to use web-routes, it would be nice to see a similar concept from Snap. Looking back at the snap-core and snap-server packages, it *definitely* seems like I misspoke: most functionality that *could* be split off seems to already have been. My apologies. Michael

On Jan 25, 2011, at 12:29 PM, Chris Smith wrote:

On Tue, 2011-01-25 at 20:10 +0200, Michael Snoyman wrote:

...

...

That's one of the main reasons I've designed Yesod and WAI the way I have. By splitting the project up into so many pieces (Hamlet, Persistent, authenticate, etc), users get to pick and choose exactly what they want.

Yes. Happstack is the same way.

Yes, and I'm a fan of that. I just wish that other frameworks (cough, cough) could take a similar approach.

What parts of Snap would you like to see split out into separate projects? Heist is completely independent of Snap. And snap-server is already a separate package from snap-core that handles the low-level HTTP stuff, though perhaps the module boundary isn't exactly where and like you're accustomed to.

The splitting out of heist is perfect. That made it easy to create happstack-heist. The boundary on the spliting of snap-core and snap-server is not quite right in my opinion. It tries to use just two packages where we really want three. The snap-core layer is an odd mix of the low-level and high-level code with the middle part split out into snap-server. The base layer should be 'crackle' which just contains the low-level types like Request, Response, etc. And functions that work with just those low-level types. This would probably include low-level aspects of parsing and creating Cookies, etc. Do happstack, yesod, and snap really need their own special cookie parsers? I doubt it. On top of the 'crackle' layer you build things like snap-server. In fact, snap-server could be split into two libraries, snap-server- simple and snap-server-libev. Additionally you could have snap-server- fastcgi, snap-sever-cgi, etc. They will all export some function with a type like:

type Backend = Request -> Iteratee ByteString IO Response backend :: Conf -> Backend

On top of the 'crackle' layer you also build the high level snap- framework stuff which contains the other stuff that is current in snap- core such as the Snap monad, etc. Note that the snap-framework stuff does not build on top of the the snap-server stuff -- just the crackle layer. Now, when I build my snap-based application I import snap-framework (which likely re-exports the crackle stuff for me), and build my app on top of that. A vast majority of the application is server-backend agnostic. In my Main.hs is where I actually import the 'backend' function from one of the snap-server-* libraries. This should make it very easy too swap out one server backend for another. You basically just import a different 'server' function and drop it in (though each server backend may have a slightly different Conf type).

main = serve (backend backendConf) mySnapApp

Where 'serve' comes from snap-framework (not crackle):

serve :: Backend -> Snap () -> IO ()

Or something like that. The wider advantage of crackle+snap-server-* is that it makes it easier to: (1) add additional backends (2) use the code in other frameworks Happstack would be very interested in using the crackle layer + the snap-server-* components. While happstack can do that right now, it just doesn't feel very clean. When we import Snap.Types to get at the Request and Response types, it means we also have a bunch of extra crap we do not want like the Snap monad. With crackle+snap-server-* I can import just the pieces I will actually use. This is the exactly model that WAI offers. I can build happstack around the WAI package, and then get the warp server or whatever for the backend. Basically, crackle is WAI, but which a different name. I have heard numerous claims that it is too early to 'standardize' on WAI, though I am not clear what the practical differences are. The essence of WAI is this type: type Application = Request -> Iteratee B.ByteString IO Response Which is looks awfully similar to what Snap does. If you unpack the Snap monad enough, that is essentially what you end up with. The only difference is that you have the Request/Response types from Snap instead of from WAI. What makes the Snap version of the Request/Response types fundamentally incompatibly with and superior to WAI? - jeremy

On Tue, Jan 25, 2011 at 8:35 PM, Jeremy Shaw wrote:

On Jan 25, 2011, at 12:10 PM, Michael Snoyman wrote:

...

...

 7. the ServerPartT monad has a built-in FilterT monad which makes it easy to apply a function (Response -> Response) to modify the response.

In all honesty, this is an argument *against* frameworks, not for them: at a WAI level, we don't need a FilterT monad, since an application is just a function.

(And I can't believe I'm giving arguments against frameworks here, I obviously think they're a good thing. I'm just trying to be fair.)

Sure, you can do everything at the low-level. The idea is to add value. Ultimately happstack calls this function,

simpleHTTP'' :: (ToMessage b, Monad m, Functor m) => ServerPartT m b -> Request -> m Response

which is basically a function to convert a ServerPartT into a WAI-like, type Application = Request -> IO Response. :)

The value added by the FilterT monad is that it allows you to apply a transformation to the Response before you actually have created the Response.

For example, I may have a function that adds a session cookie. But I don't have to structure my code so that the function is run after I have generated the response. For example, lets say I have a function that sets a cookie and returns the response body as HTML:

myPart :: ServerPart Html myPart =        do addCookie someCookie             return $ <html>...</html>

I can't add the cookie directly to the Response yet, because this function returns Html not a Response. The cookie is added later after the Html is converted to a Response.

I assume the WriterT (Endo [Header]) thing in yesod serves a similar purpose ?

Yes, precisely. As I see it, the difference between the two approaches would be: * Yesod's is a little bit faster. And I mean a *little* bit. * Happstack's is more powerful, in that you can do more than simply append a header. Would you concur? Michael

Hi, I do appreciate the argument that I am likely to do lots of things that may benefit from a framework, and I have somewhat developed a framework for Hoogle. It has 3 layers, each calling the one below. :: IO () -- run a web server :: CmdLine -> Request -> IO Response -- deal with request/response, this layer is about 15 lines :: Database -> Query -> String -- the actual logic The first layer has been mostly replaced by WAI/Warp. The second layer is very small, and I don't think has any bits in it that could be abstracted. In the third layer the only feature I might make use of from a web framework is some templating library. Previously the third layer made use of trhsx, the XML in Haskell dialect, but then I switched back to just building strings. (I will probably change the String return to Builder at some point, or switch to a templating library.) Given that Hoogle does a lot more back end processing, and not so much serving of images/form submission/websity things etc.I think the use case for a framework would be limited - there just isn't enough code to get any gains - my .cabal file would probably grow more than my lines of code decreases :-) If in future Hoogle becomes more webby, gets more web features, I might grab more bits of web framework into the mix.

That's one of the main reasons I've designed Yesod and WAI the way I have. By splitting the project up into so many pieces (Hamlet, Persistent, authenticate, etc), users get to pick and choose exactly what they want. Sure, as a framework author I would prefer them using the whole stack, but as someone trying to push Haskell into more mainstream use[1], giving people choice here is best.

I am a big fan of the Yesod approach of lots of small components, that can be used separately, but also integrate nicely. If I ever go shopping for a template library I'll certainly be looking for something with similar properties. Thanks, Neil

Hi, On feature requests for Warp, I have only two outstanding requests: 1) A way of finding how long a response took to generate. 2) A way of finding out when exceptions happen. Can either of these be achieved with middleware, or Response transformers? I thought the timing one could probably be done by consuming all the Response output, and then as a final action at the end writing to an MVar or a Chan, which some other thread could read from to see the time elapsed. Is that possible? It seems feasible... I couldn't think of any way of displaying exceptions though, so perhaps allow runEx which takes an exception handler: runEx :: Port -> Application -> (Request -> SomeException -> IO ()) -> IO () Is there any other way to robustly get all exceptions? Regarding Jeremy's points about using Happstack, I originally considered Happs for use in Hoogle many years ago. Then I came to the conclusion that just installing Happs was too much pain (this was the days before cabal install), and I wouldn't get enough payoff. All the things you describe about setting cookies, transforming output etc. aren't things that Hoogle yet does, so I still think the payoff is still too small. If Hoogle gets more features, I'll evaluate once more. I certainly don't consider Happstack to be sewerage :-) Thanks, Neil

...

1) A way of finding how long a response took to generate. 2) A way of finding out when exceptions happen.

Can either of these be achieved with middleware, or Response transformers? I thought the timing one could probably be done by consuming all the Response output, and then as a final action at the end writing to an MVar or a Chan, which some other thread could read from to see the time elapsed. Is that possible? It seems feasible...

Are you talking about timing how long it takes to *generate* a response, or how long it takes to *send* a response? The latter would require changes to Warp which I do not want to make for the moment.

I want the generation time, but would accept send time as a rough approximation.

The former would require forcing evaluation of your response body, which will kill any lazy IO advantages you have. However, it should be implementable at the application/middleware level.

I wondered if it was possible to do so keeping the lazy IO, but if not it's not the end of the world.

See the other email I just sent; I added a function runOnException. However, I think runEx is a nicer name, so I'm going to steal it. As for the requested type signature (including a Request): it's not really feasible at the Warp level: the exceptions Warp will be catching are not necessarily part of a request, it could be in the middle of receiving a request, a timeout in between one response and the next request, etc.

Fair enough. Generally, just not losing exceptions will be appreciated - I don't expect exceptions to be common at all. Thanks, Neil

On Tue, Jan 25, 2011 at 9:33 AM, Bardur Arantsson wrote:

On 2011-01-25 02:14, Jeremy Shaw wrote:

...

On Mon, Jan 24, 2011 at 4:38 PM, Neil Mitchell  wrote:

...

I think WAI could be increasingly used - its at a level of abstraction that I quite like. I certainly have other projects which I will probably switch over to WAI/Warp.

I am curious to see how your feelings on this change or stay the same over time.

[--snip--]

...

These are the types of features (and more) that exist in the higher-level frameworks like Happstack, Snap, and Yesod. In theory, that is the value they add over just using WAI. It is likely that, in the future, happstack will not include it's own http backend, but be a layer on top of WAI+warp, hyena, or some other low-level WAI-like HTTP backend. So, I am curious if the value-added features we provide actual provide value :)  One measure of this will be if you end up creating similar adhoc functions and structure in your apps.

[--snip--]

I'm actually considering using WAI for two of my projects as well. I haven't had the time to actually try it, but I'm pretty sure I'll be doing it for at least one of these projects.

The first one is hums, a UPnP media server. I've essentially implemented the socket listening/accept/etc. bits myself and used the 'http' package for the protocol bits. In this particular case there is no need for sessions, cookies or anything more advanced than simple request/response cycle with the occasional "send file", so it seems WAI would be a great fit here. It would be nice to finally be rid of the String-based 'http' dependency as well.

The second one *is* a framework (just started, hopefully I'll actually finish it) for Rich Internet Applications a la Wicket, Vaadin and Echo3. Essentially you (as the framework user) program your application in a "magical" monad which turns widgets, components and other logic into a combination of browser JavaScript/Ajax calls and server-side logic for processing state updates. I think using a low-level API like WAI is also quite justified in this case simply to avoid huge dependencies. The only thing that I'm likely to have to "reimplement" is a little bit of routing (but that's a trivial amount) and some handing of cookies/sessions (though I may be able to avoid that in the short term). There will also be some autogenerated JavaScript and such, but that's such a niche thing that I wouldn't really expect much support from a general framework for such a thing anyway.

The thing I'd like to see is really more *framework-independent* code for things like session handling, cookie handling, etc. I really like the 'http' package for this reason. It does exactly one thing and (mostly) doesn't get in the way of whatever else you're trying to do.

Maybe some of this 'generic' stuff could be put in WAI middleware -- then at least Neil and I could share that code :).

I've been making a big push to get framework-agnostic code separated out into separate packages wherever possible. For your case you may want to look at the cookie and clientsession packages. Additionally, you may want to consider using blaze-html for the HTML generation, or even Hamlet. I'd be interested in hearing more about how you are planning on implementing this framework. Yesod already provides built-in support for widgets, though so far I have been very conservative about including direct Ajax features. For the most part, it is used for providing form controls that do validation. However, there's no reason it *couldn't* be used for Ajax. Following my normal sets of advice, I would recommend that any component that you build that is not intrinsically tied to your framework be released as a separate package so we can all benefit and share resources where relevant. And if you have questions/requests on the WAI side, just let me know. Michael

On Tue, Jan 25, 2011 at 8:53 PM, Bardur Arantsson wrote:

On 2011-01-25 09:49, Michael Snoyman wrote:

...

On Tue, Jan 25, 2011 at 9:33 AM, Bardur Arantsson  wrote:

...

On 2011-01-25 02:14, Jeremy Shaw wrote:

...

On Mon, Jan 24, 2011 at 4:38 PM, Neil Mitchell  wrote:

[--snip--] The thing I'd like to see is really more *framework-independent* code for things like session handling, cookie handling, etc. I really like the 'http' package for this reason. It does exactly one thing and (mostly) doesn't get in the way of whatever else you're trying to do.

Maybe some of this 'generic' stuff could be put in WAI middleware -- then at least Neil and I could share that code :).

I've been making a big push to get framework-agnostic code separated out into separate packages wherever possible. For your case you may want to look at the cookie and clientsession packages. Additionally, you may want to consider using blaze-html for the HTML generation, or even Hamlet.

I've already settled on blaze-html... a lot less fidgeting with string concatenation, etc. plus it seems to perform extremely well.

Hamlet isn't really useful for my purposes -- I really want *ALL* the client-side HTML/JS/etc. to be generated by widgets/components and my framework users shouldn't write any HTML directly. This HTML will also be injected into the page dynamically as the user interacts with the UI, so blaze-html is great.

For your use case, blaze-html *does* sound like the way to go. And I agree: it's an awesome package.

...

I'd be interested in hearing more about how you are planning on implementing this framework.

...

Yesod already provides built-in support

for widgets, though so far I have been very conservative about including direct Ajax features. For the most part, it is used for providing form controls that do validation. However, there's no reason it *couldn't* be used for Ajax.

I've actually implemented a little demo app in Yesod and have looked a little at the widget type/module, but as far as I can tell they don't go anywhere close to where I want to go. I don't have an actual design doc anywhere at the moment, so the only thing I can use to illustrate what I want to do is by way of pointing to Vaadin. You basically program as if you were programming an old-school graphical UI (Swing, SWT, GTK+, Tk, whatever) and the framework takes care of "translating" that to a browser application. Essentially the user should not have to write *ANY* html, JavaScript or CSS at all.

All the state transfer will happen through callbacks to the server and/or Comet-style push updates to the browser. I'm still investigating reasonable ways to implement proper state handling.

I could be wrong here, but I have the feeling that it *should* be possible to build up such a solution based on the what Yesod provides in widgets. Obviously, such a solution would add much more power than what Yesod provides, but I think it already provides enough of the low-level details to get what you need. I'm not trying to convince you to use Yesod for this, but I would be very appreciative to hear what Yesod's widgets are lacking to get the functionality you want.

...

Following my normal sets of advice, I would recommend that any component that you build that is not intrinsically tied to your framework be released as a separate package so we can all benefit and share resources where relevant. And if you have questions/requests on the WAI side, just let me know.

Sure.

One thing I am looking for is a convenient way to autogenerate JavaScript. I could use the "blaze" string builders (or builders for Text or whatever), but if there is some library for building JS in a safer way than just string concatenation. As with the HTML, I'm also going to be generating this JS extremely dynamically. I know about Julius, but what I (think) I really need is a sort of AST datatype for JavaScript with a toText/toBS printer.

Alan Zimmerman put together a Javascript minification package[1], and in the process created a full-blown Javascript lexer/parser/AST package[2]. That may be somewhere do start. Michael [1] http://hackage.haskell.org/package/hjsmin [2] http://hackage.haskell.org/package/language-javascript-0.4.2

On Jan 25, 2011, at 1:33 AM, Bardur Arantsson wrote:

I'm actually considering using WAI for two of my projects as well. I haven't had the time to actually try it, but I'm pretty sure I'll be doing it for at least one of these projects.

The second one *is* a framework (just started, hopefully I'll actually finish it) for Rich Internet Applications a la Wicket, Vaadin and Echo3. Essentially you (as the framework user) program your application in a "magical" monad which turns widgets, components and other logic into a combination of browser JavaScript/ Ajax calls and server-side logic for processing state updates. I think using a low-level API like WAI is also quite justified in this case simply to avoid huge dependencies. The only thing that I'm likely to have to "reimplement" is a little bit of routing (but that's a trivial amount) and some handing of cookies/sessions (though I may be able to avoid that in the short term). There will also be some autogenerated JavaScript and such, but that's such a niche thing that I wouldn't really expect much support from a general framework for such a thing anyway.

I think there are a lot of things you might want from a slightly higher level framework like happstack-server. Note that you can install happstack-server with out happstack-state or any of the templating libraries. You could then create your magical monad by adding some additional layers to the ServerPartT monad transformer. By using happstack-server you would get access to: 1. RESTful-style routing combinators like (dir, path, etc), or type- safe urls via web-routes 2. a cookie API 3. a rich API for serving static files (which handles things like 'if-modified-since' automatically) 4. a rich API for extracting values from cookies, query parameters, and the request body 5. support for file uploads 6. support for automatic compression of Response bodies And the default ServerPartT monad gives you easy support for accessing the Request, applying filters to the Response, aborting/escaping the current computation and returning a different Response, support for throw/catch, and more. I have done some work on trying to create a high-level haskell DSL which automatically generates client-side javascript+html, communication via JSON, etc. I personally found building on top of happstack to be useful. The dependency list for happstack-server is not all that different from wai+wai-extra+warp. So concerns of a 'huge dependency list' are perhaps overblown. The point being -- if happstack-server were built on top of WAI today, so that the choice was between wai or happstack+wai, I would still recommend that you consider happstack+wai as your foundation not just wai. :) - jeremy p.s. I am interested in possibly moving happstack to be built on top of WAI. I even did an experimental port once just to confirm that it would be feasible. The biggest hold up at the moment is probably the lack of timeout handling in warp. Though maybe that has been added now ?

On Jan 25, 2011, at 12:12 PM, Michael Snoyman wrote:

Yes, the timeout handling is in. I intend to move to your new timeout code when I have some free time to work on it. Of course, the whole point of having WAI/Warp would be that we could collaborate on these kinds of projects, so if you want to help out, it will definitely be appreciated.

Cool. I am all for that. I am certain that it will be a lot better use of my time to contribute to an existing low-level, enumerator based backend than to reinvent my own that is mostly the same and superficially different ;) That is the big feature for Happstack 8. (I am going to let warp and snap battle it out for a bit first while I work on MACID). - jeremy

On Jan 25, 2011, at 1:18 PM, Bardur Arantsson wrote:

On 2011-01-25 18:35, Jeremy Shaw wrote:

...

On Jan 25, 2011, at 1:33 AM, Bardur Arantsson wrote:

...

I think there are a lot of things you might want from a slightly higher level framework like happstack-server. Note that you can install happstack-server with out happstack-state or any of the templating libraries.

Possibly. However, I'm not really sure it's worth the extra dependency.

Btw, I actually tried to convert my Hums project to Happstack. There was some crucial problem which I was not able to overcome. I think it *might* have had something to do with not wanting to use Sendfile (the old thread/FD leak problem) and not finding another way to do the same thing without using Lazy I/O (which I *definitely* don't want for transferring huge files to a crappy HTTP client). I'm not entirely sure that was what the problem was, but if it sounds plausible, then there might be a feature request lurking there :).

SendFile should not leak file handles. I have run millions of sendfile requests through happstack (in darcs) with no signs of FD/thread leakage. There were fd leaks problems with the lazy IO file serving code. That is one of the reasons we switched to sendfile :p - jeremy

Ah yes, that issue. I remember it well. Unfortunately, I was never able to recreate the issue, so it is difficult to create an optimal low-level solution. However, in the latest Happstack, we will kill the thread if there is no progress made for 30 seconds. So, I do not believe it will leak threads for an arbitrary amount of time anymore. - jeremy On Tue, Jan 25, 2011 at 3:22 PM, Bardur Arantsson wrote:

On 2011-01-25 21:53, Jeremy Shaw wrote:

...

On Jan 25, 2011, at 1:18 PM, Bardur Arantsson wrote:

...

On 2011-01-25 18:35, Jeremy Shaw wrote:

[--snip--]

...

SendFile should not leak file handles. I have run millions of sendfile requests through happstack (in darcs) with no signs of FD/thread leakage.

You're using a well-behaved TCP/HTTP client. The PS/3 isn't that well-behaved -- it cuts connections IMMEDIATELY (no TCP shutdown or anything of that nature) once it's decided that it doesn't need the connection any more. Under those circumstances I believe that the sendfile library still leaks threads for an essentially arbitrary amount of time.

If you recall, there was a huge thread on -cafe on this back in Feb. of 2010. This was 100% reproducible with a PS3 client, and I haven't seen any evidence that the thread leak has been addressed.

Anyway, didn't want to hijack this thread. I'll go back to lurking now :).

Cheers,

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

Hi, I tried switch Hoogle to Wai and Warp today. The results were good, and once a few little things are cleared up I'm fairly certain that the the next Hoogle release will be based on them. As I wrote the code, I made some notes: 1) Why not add statusOK as an alias for status200? As someone who doesn't spend all day dealing with HTTP response codes, statusOK is far better documenting - similarly for all the other status values. I added statusOK as a local definition in Hoogle. 2) Changing from HTTP to Wai, it was a shame that I had to define all the header keys myself - could they be included - it would have better performance, and give static guarantees. I added locally: hdrContentType = mkCIByteString "Content-Type" hdrCacheControl = mkCIByteString "Cache-Control" 3) The documentation for ResponseFile doesn't say what happens if the file is not found. I am sure it must throw an exception, but for a very brief second I wondered if it might change my status code to 404 on my behalf. Could the documentation be clarified. 4) I defined a Wai handler that threw an error, and Warp did nothing. Should it? Or should I just be careful and include my own exception handling? I'm not really sure what the right thing to do is - silently eating the error isn't great, but bring down my server would be even less appreciated. 5) Hoogle is currently written as lazy IO + String. Converting this format to a response was easy with liftIO, LBS.pack and responseLBS. Converting back was rather tricky. I eventually came up with: responseFlatten :: Response -> IO (Status, ResponseHeaders, LBS.ByteString) responseFlatten r = responseEnumerator r $ \status headers -> let f res E.EOF = return (status, headers, Blaze.toLazyByteString res) f res (E.Chunks xs) = E.continue (f $ mconcat $ res:xs) in E.continue (f mempty) It's not too long, but it has a relatively high level of complexity. Is there something simpler I could have done? 6) I want responseFlatten for two purposes, the first of which is to output a CGI representation of the Response, which is easy with the above function. However, outputting a response in CGI format is trivial, and is probably the easiest way to debug/view the contents of a Response. Would it be worth adding showResponse :: Response -> IO LBS.ByteString to the Wai library, which produced the result representing how it would be sent in CGI format, and was also useful for debugging? 7) The second reason for using responseFlatten is that I want to write a function Response -> IO Response which modifies a response to rewrite "href='file://" to "href='/file/". I currently do this with responseFlatten, converting to a String, and then stepping through it Char by Char with isPrefixOf. Given the performance focus of this email list, I can imagine several of you are now crying. What would have been the Iterator/Builder approved way to do it? 8) The last point is the only point that is a blocker for Hoogle, and regards the Warp server, Windows and ghci. In ghci on Windows, running the network function accept doesn't terminate if you hit Ctrl+C. If you run Warp in the main thread ghci will never respond to you again. If you run Warp on a second thread then hitting Ctrl+C twice returns to the prompt, but the thread serving the web page is still running (and still serving web pages), and if you run the server again the old server continues to serve the pages instead. I develop Hoogle on Windows through ghci in server mode, so without a workaround it's fairly fatal. When writing Hoogle, I found that even though accept doesn't terminate on exceptions, if you do sClose on the socket it raises an exception in accept and does terminate. I experimented in a local copy of Warp and found the following worked very well: import Control.Concurrent import Control.Monad run :: Port -> Application -> IO () run port app = withSocketsDo $ do var <- newMVar Nothing let clean = modifyMVar_ var $ \s -> maybe (return ()) sClose s >> return Nothing forkIO $ bracket (listenOn $ PortNumber $ fromIntegral port) (const clean) (\s -> do modifyMVar_ var (\_ -> return $ Just s); serveConnections port app s) forever (threadDelay maxBound) `finally` clean If you hit Ctrl+C in ghci on Windows you get the message "Network.Socket.accept: failed (No error)" - but it terminates perfectly. That error could even be dropped silently if you reach the `finally` clause. Can this code be included in Warp? Thanks, Neil