ANN: HTTPbis / HTTP-4000.x package available

Hi, I guess it's time to publish more widely the availability of a modernization of the venerable and trusted HTTP package, which I've been working on off&on for a while. Bunch of changes, but the headline new feature of this new version is the parameterization of the representation of payloads in both HTTP requests and responses. Two new representations are supported, strict and lazy ByteStrings. Some people have reported quietly pleasing speedups as a result of this change. (If they want to report numbers, please do..) Another change/fix in this release is the _alleged_ fix to the long-standing bug in the use of absolute URIs vs absolute paths in requests (for non-proxied and proxied use.) Give it a go.. Notice that the HTTP-4000.x version will require you to make some modifications to your existing HTTP-using code -- I've tried to keep the API backwards compatible minimal despite the change in functionality and underlying types. If you do not want to deal with this right away, please introduce a <4000 dependency on the HTTP package in your .cabal files. I've also taken on the maintainership of the package, with the highly esteemed Bjorn Bingert no longer having the usual abundance of cycles to look after it (hope I'm not misrepresenting facts here, Bjorn!) However, I've yet to gain access to www.haskell.org and update http://www.haskell.org/http, so for now you may pick up a new version the lib via http://hackage.haskell.org/cgi-bin/hackage-scripts/package/HTTP GIT repo for HTTP-4000 / HTTPbis is here git://code.galois.com/HTTPbis.git enjoy --sigbjorn

On Thu, 15 Jan 2009, Sigbjorn Finne wrote:
I guess it's time to publish more widely the availability of a modernization of the venerable and trusted HTTP package, which I've been working on off&on for a while.
I was always afraid that a fork may happen during I work on HTTP in order to get it more lazy. That's why I started discussion on web-devel mailing list, but got only limited response. I also notified Bjorn, that my changes need still a little time and that I'm worried about darcs conflicts. Seems that we now run into a perfect conflict. I don't like to throw away the work that I have done the last weeks. My version is at http://code.haskell.org/~thielema/http/

lemming:
On Thu, 15 Jan 2009, Sigbjorn Finne wrote:
I guess it's time to publish more widely the availability of a modernization of the venerable and trusted HTTP package, which I've been working on off&on for a while.
I was always afraid that a fork may happen during I work on HTTP in order to get it more lazy. That's why I started discussion on web-devel mailing list, but got only limited response. I also notified Bjorn, that my changes need still a little time and that I'm worried about darcs conflicts. Seems that we now run into a perfect conflict. I don't like to throw away the work that I have done the last weeks.
My version is at http://code.haskell.org/~thielema/http/
There's absolutely no need to throw away work. Hackage is a broad church, release it as http-lazy, at the very least. -- Don

Yes, please do. At some point soon, I will do and release a feature
and performance benchmark on HTTP-3xxx, -4xxx, -lazy (if you release
it), and cURL, that way people can use what's best for their
application.
On Fri, Jan 16, 2009 at 12:29 PM, Don Stewart
lemming:
On Thu, 15 Jan 2009, Sigbjorn Finne wrote:
I guess it's time to publish more widely the availability of a modernization of the venerable and trusted HTTP package, which I've been working on off&on for a while.
I was always afraid that a fork may happen during I work on HTTP in order to get it more lazy. That's why I started discussion on web-devel mailing list, but got only limited response. I also notified Bjorn, that my changes need still a little time and that I'm worried about darcs conflicts. Seems that we now run into a perfect conflict. I don't like to throw away the work that I have done the last weeks.
My version is at http://code.haskell.org/~thielema/http/
There's absolutely no need to throw away work. Hackage is a broad church, release it as http-lazy, at the very least.
-- Don _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Don Stewart schrieb:
lemming:
On Thu, 15 Jan 2009, Sigbjorn Finne wrote:
I guess it's time to publish more widely the availability of a modernization of the venerable and trusted HTTP package, which I've been working on off&on for a while. I was always afraid that a fork may happen during I work on HTTP in order to get it more lazy. That's why I started discussion on web-devel mailing list, but got only limited response. I also notified Bjorn, that my changes need still a little time and that I'm worried about darcs conflicts. Seems that we now run into a perfect conflict. I don't like to throw away the work that I have done the last weeks.
My version is at http://code.haskell.org/~thielema/http/
There's absolutely no need to throw away work. Hackage is a broad church, release it as http-lazy, at the very least.
Somehow yes. As I said, it still needs cleanup and I wonder whether I invest that time in my version or whether I should merge my changes immediately into HTTP-4000. As it now stands, I can provide laziness and HTTP-4000 provides ByteStrings and I'm sure people wants both of them.

Just as a reported speedup, downloading a 5MB file from my own local
machine (via http) went from 1.05 secs to 0.053 secs. Yes, it's
really an order of magnitude better. Performance now is on par or
slightly better than cURL (however to get more protocols than HTTP,
you'll still need the ubiquitous cURL library)
On Fri, Jan 16, 2009 at 1:36 AM, Sigbjorn Finne
Hi,
I guess it's time to publish more widely the availability of a modernization of the venerable and trusted HTTP package, which I've been working on off&on for a while.
Bunch of changes, but the headline new feature of this new version is the parameterization of the representation of payloads in both HTTP requests and responses. Two new representations are supported, strict and lazy ByteStrings. Some people have reported quietly pleasing speedups as a result of this change. (If they want to report numbers, please do..)
Another change/fix in this release is the _alleged_ fix to the long-standing bug in the use of absolute URIs vs absolute paths in requests (for non-proxied and proxied use.) Give it a go..
Notice that the HTTP-4000.x version will require you to make some modifications to your existing HTTP-using code -- I've tried to keep the API backwards compatible minimal despite the change in functionality and underlying types. If you do not want to deal with this right away, please introduce a <4000 dependency on the HTTP package in your .cabal files.
I've also taken on the maintainership of the package, with the highly esteemed Bjorn Bingert no longer having the usual abundance of cycles to look after it (hope I'm not misrepresenting facts here, Bjorn!) However, I've yet to gain access to www.haskell.org and update http://www.haskell.org/http, so for now you may pick up a new version the lib via
http://hackage.haskell.org/cgi-bin/hackage-scripts/package/HTTP
GIT repo for HTTP-4000 / HTTPbis is here
git://code.galois.com/HTTPbis.git
enjoy --sigbjorn
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Thanks Jeff, regarding having to use both HTTP and cURL -- or perhaps only the latter for code simplicitly -- that will probably remain the case for quite a while still. To help with that situation, I put together an over-arching 'webclient' library that abstracts over the transport layers (HTTP, curl, curl-shell) giving you an API that's consistent across backends. I could release that at some point if there's sufficient interest.. It also adds WebDAV support. Re: curl - as an author of the 'curl' package, I'm also keen on finding ways of making that better -- both in terms of using the underlying lib functionality and API on the Haskell side. Suggestions/contribs most welcome. Ditto for HTTP too, of course :) --sigbjorn On 1/16/2009 05:47, Jeff Heard wrote:
Just as a reported speedup, downloading a 5MB file from my own local machine (via http) went from 1.05 secs to 0.053 secs. Yes, it's really an order of magnitude better. Performance now is on par or slightly better than cURL (however to get more protocols than HTTP, you'll still need the ubiquitous cURL library)
On Fri, Jan 16, 2009 at 1:36 AM, Sigbjorn Finne
wrote: Hi,
I guess it's time to publish more widely the availability of a modernization of the venerable and trusted HTTP package, which I've been working on off&on for a while.
Bunch of changes, but the headline new feature of this new version is the parameterization of the representation of payloads in both HTTP requests and responses. Two new representations are supported, strict and lazy ByteStrings. Some people have reported quietly pleasing speedups as a result of this change. (If they want to report numbers, please do..)
Another change/fix in this release is the _alleged_ fix to the long-standing bug in the use of absolute URIs vs absolute paths in requests (for non-proxied and proxied use.) Give it a go..
Notice that the HTTP-4000.x version will require you to make some modifications to your existing HTTP-using code -- I've tried to keep the API backwards compatible minimal despite the change in functionality and underlying types. If you do not want to deal with this right away, please introduce a <4000 dependency on the HTTP package in your .cabal files.
I've also taken on the maintainership of the package, with the highly esteemed Bjorn Bingert no longer having the usual abundance of cycles to look after it (hope I'm not misrepresenting facts here, Bjorn!) However, I've yet to gain access to www.haskell.org and update http://www.haskell.org/http, so for now you may pick up a new version the lib via
http://hackage.haskell.org/cgi-bin/hackage-scripts/package/HTTP
GIT repo for HTTP-4000 / HTTPbis is here
git://code.galois.com/HTTPbis.git
enjoy --sigbjorn
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Hello Sigbjorn, Friday, January 16, 2009, 5:42:06 PM, you wrote: first question: are these packages (http, curl, curl-shell, webclient) windows-compatible? second - that is advantages of using http (or webclient) over curl? sorry if questions are stupid - i'm pretty ignorant here :)
Thanks Jeff,
regarding having to use both HTTP and cURL -- or perhaps only the latter for code simplicitly -- that will probably remain the case for quite a while still. To help with that situation, I put together an over-arching 'webclient' library that abstracts over the transport layers (HTTP, curl, curl-shell) giving you an API that's consistent across backends. I could release that at some point if there's sufficient interest.. It also adds WebDAV support.
Re: curl - as an author of the 'curl' package, I'm also keen on finding ways of making that better -- both in terms of using the underlying lib functionality and API on the Haskell side. Suggestions/contribs most welcome.
Ditto for HTTP too, of course :)
--sigbjorn
On 1/16/2009 05:47, Jeff Heard wrote:
Just as a reported speedup, downloading a 5MB file from my own local machine (via http) went from 1.05 secs to 0.053 secs. Yes, it's really an order of magnitude better. Performance now is on par or slightly better than cURL (however to get more protocols than HTTP, you'll still need the ubiquitous cURL library)
On Fri, Jan 16, 2009 at 1:36 AM, Sigbjorn Finne
wrote: Hi,
I guess it's time to publish more widely the availability of a modernization of the venerable and trusted HTTP package, which I've been working on off&on for a while.
Bunch of changes, but the headline new feature of this new version is the parameterization of the representation of payloads in both HTTP requests and responses. Two new representations are supported, strict and lazy ByteStrings. Some people have reported quietly pleasing speedups as a result of this change. (If they want to report numbers, please do..)
Another change/fix in this release is the _alleged_ fix to the long-standing bug in the use of absolute URIs vs absolute paths in requests (for non-proxied and proxied use.) Give it a go..
Notice that the HTTP-4000.x version will require you to make some modifications to your existing HTTP-using code -- I've tried to keep the API backwards compatible minimal despite the change in functionality and underlying types. If you do not want to deal with this right away, please introduce a <4000 dependency on the HTTP package in your .cabal files.
I've also taken on the maintainership of the package, with the highly esteemed Bjorn Bingert no longer having the usual abundance of cycles to look after it (hope I'm not misrepresenting facts here, Bjorn!) However, I've yet to gain access to www.haskell.org and update http://www.haskell.org/http, so for now you may pick up a new version the lib via
http://hackage.haskell.org/cgi-bin/hackage-scripts/package/HTTP
GIT repo for HTTP-4000 / HTTPbis is here
git://code.galois.com/HTTPbis.git
enjoy --sigbjorn
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
-- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

On Fri, 2009-01-16 at 17:57 +0300, Bulat Ziganshin wrote:
Hello Sigbjorn,
Friday, January 16, 2009, 5:42:06 PM, you wrote:
first question: are these packages (http, curl, curl-shell, webclient) windows-compatible? second - that is advantages of using http (or webclient) over curl?
I don't know enough about the others to do a proper comparison, but one data point is that for cabal-install we picked the HTTP package because it *does* work on Windows and does not need any C libraries, making it easy to deploy. [Packages that require external C libraries are possible on Windows of course but are a pain. For cabal-install we also use the zlib package which does need the zlib C lib. Fortunately the zlib C lib is sufficiently simple that it can be built by Cabal and thus it is possible to have it bundled with the zlib Haskell package.] I've not looked at the API of the other packages but the HTTP package provides the Browser module which makes making sequences of requests fairly easy. It handles proxies, authentication, redirection etc. With the latest release it should also be rather quicker than it used to be, and it fixes problems we used to have with some proxies. Speaking of proxies, I've been looking into how to find the right proxy to use on Windows systems. Turns out that to do it properly you need a JavaScript interpreter! Yes, really. Apparently we have Netscape to thank for that decision. The easiest way appears to be to link to the MS WinHTTP lib which provides a couple functions for doing this: http://hackage.haskell.org/trac/hackage/ticket/460 Duncan

On Fri, 16 Jan 2009, Sigbjorn Finne wrote:
gain access to www.haskell.org and update http://www.haskell.org/http, so for now you may pick up a new version the lib via
http://hackage.haskell.org/cgi-bin/hackage-scripts/package/HTTP
GIT repo for HTTP-4000 / HTTPbis is here
git://code.galois.com/HTTPbis.git
Can you please add a motd to the old darcs repository, which tells the downloader that this repository has moved to a git repository?

On Thu, 2009-01-15 at 22:36 -0800, Sigbjorn Finne wrote:
Another change/fix in this release is the _alleged_ fix to the long-standing bug in the use of absolute URIs vs absolute paths in requests (for non-proxied and proxied use.) Give it a go..
cabal-install with HTTP-3001.x: $ cabal update -v3 Downloading package list from server 'http://hackage.haskell.org/packages/archive' Sending: GET http://hackage.haskell.org/packages/archive/00-index.tar.gz HTTP/1.1 User-Agent: cabal-install/0.6.1 Creating new connection to hackage.haskell.org cabal-install with HTTP-4000.0.2: $ cabal update -v3 Downloading package list from server 'http://hackage.haskell.org/packages/archive' Sending: GET /packages/archive/00-index.tar.gz HTTP/1.1 Connection: close User-Agent: cabal-install/0.6.1 Host: hackage.haskell.org Creating new connection to hackage.haskell.org Hooray! Thanks Sigbjorn! That should fix the bug with cabal-install and transparent proxies. It should also make downloads a tad quicker. Thanks also for changing the names back to ordinary Request/Response. I note we're now using "Connection: close". Is there a way in the Browser monad to do pipelined requests? It would be useful in cabal-install when users want to install a package with several dependencies so we can make all the requests in a single connection (if the server supports it etc). I realise it's not trivial since the Browser monad would have to hide establishing new connections and re-sending requests if the server closes the connection etc. Duncan

On Thu, 15 Jan 2009, Sigbjorn Finne wrote:
I guess it's time to publish more widely the availability of a modernization of the venerable and trusted HTTP package, which I've been working on off&on for a while.
Bunch of changes, but the headline new feature of this new version is the parameterization of the representation of payloads in both HTTP requests and responses. Two new representations are supported, strict and lazy ByteStrings.
After looking into your code it seems that an easy merge will not be possible. I'm not quite ready with Lazy I/O, but close to it. I introduced a new monad which allows to get rid of explicit usage of the IO monad. I also rewrote Network.HTTP in order to handle asynchronous exceptions in a clean way, thus making Network.HTTP really lazy. See also the thread: http://www.haskell.org/pipermail/web-devel/2008/000048.html I'm really frustrated now, since I invested much time into improvement of the HTTP package. I did a lot of communication in order to prevent me from doing duplicate work, which already happened in other Haskell projects, and now an official successor appears from nowhere which can almost not be merged with my effort.

Sounds very good to me. However I would like to as one question
regarding the HTTP lib. On hackage I read: "HTTP: A library for
client-side HTTP". Maybe you or someone on this list can tell me what
the restrictions of the HTTP library are that restrict it to client
side. What would be required to enable it for server-side use as well?
Is it planed to complete HTTP this way?
Many thanks,
Levi
On Fri, Jan 16, 2009 at 7:36 AM, Sigbjorn Finne
Hi,
I guess it's time to publish more widely the availability of a modernization of the venerable and trusted HTTP package, which I've been working on off&on for a while.
Bunch of changes, but the headline new feature of this new version is the parameterization of the representation of payloads in both HTTP requests and responses. Two new representations are supported, strict and lazy ByteStrings. Some people have reported quietly pleasing speedups as a result of this change. (If they want to report numbers, please do..)
Another change/fix in this release is the _alleged_ fix to the long-standing bug in the use of absolute URIs vs absolute paths in requests (for non-proxied and proxied use.) Give it a go..
Notice that the HTTP-4000.x version will require you to make some modifications to your existing HTTP-using code -- I've tried to keep the API backwards compatible minimal despite the change in functionality and underlying types. If you do not want to deal with this right away, please introduce a <4000 dependency on the HTTP package in your .cabal files.
I've also taken on the maintainership of the package, with the highly esteemed Bjorn Bingert no longer having the usual abundance of cycles to look after it (hope I'm not misrepresenting facts here, Bjorn!) However, I've yet to gain access to www.haskell.org and update http://www.haskell.org/http, so for now you may pick up a new version the lib via
http://hackage.haskell.org/cgi-bin/hackage-scripts/package/HTTP
GIT repo for HTTP-4000 / HTTPbis is here
git://code.galois.com/HTTPbis.git
enjoy --sigbjorn
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Hi Levi, I'm guessing that you are reading something different into that than what's intended - it's "client-side" in the sense that it can only issue web requests and handle their responses. i.e., it doesn't handle incoming HTTP requests and issue suitable responses. Web server implementation is an interesting problem in its own right, and many packages/frameworks do an admirable job of that already, so no plans (by me) to tackle that via the HTTP package. But, utilizing the HTTP package as part of any web app you expose on a web server is very much on and not out of bounds. Go for it! :-) Does that answer your Q? (my apologies if I'm stating the obvious above.) cheers --sigbjorn Levi Greenspan wrote:
Sounds very good to me. However I would like to as one question regarding the HTTP lib. On hackage I read: "HTTP: A library for client-side HTTP". Maybe you or someone on this list can tell me what the restrictions of the HTTP library are that restrict it to client side. What would be required to enable it for server-side use as well? Is it planed to complete HTTP this way?
Many thanks, Levi

Sigbjorn Finne schrieb:
Hi Levi,
I'm guessing that you are reading something different into that than what's intended - it's "client-side" in the sense that it can only issue web requests and handle their responses. i.e., it doesn't handle incoming HTTP requests and issue suitable responses. Web server implementation is an interesting problem in its own right, and many packages/frameworks do an admirable job of that already, so no plans (by me) to tackle that via the HTTP package.
There's however still no framework which supports both HTTP client and server functions using the same Request and Response data type, right? I don't know whether I am the only one who needs this (e.g. for the Real Monad Transformer). E.g. a proxy would need this, too.

There's however still no framework which supports both HTTP client and server functions using the same Request and Response data type, right? I don't know whether I am the only one who needs this (e.g. for the Real Monad Transformer). E.g. a proxy would need this, too.
I've wanted this for a while now. "Me Too." Tim Newsham http://www.thenewsh.com/~newsham/

On 1/17/2009 08:21, Tim Newsham wrote:
There's however still no framework which supports both HTTP client and server functions using the same Request and Response data type, right? I don't know whether I am the only one who needs this (e.g. for the Real Monad Transformer). E.g. a proxy would need this, too.
I've wanted this for a while now. "Me Too."
Tim Newsham http://www.thenewsh.com/~newsham/ There's the basic receiveHTTP and respondHTTP that
Network.HTTP.Stream Network.HTTP.HandleStream exports. But that is probably not enough for your uses, I suspect. To ensure that this feature request isn't dropped or forgotten about, please add a ticket for it via the (new-homed) HTTP web pages -- http://projects.haskell.org/http/ Or, better still :-), contribute the code...to HTTP or some existing web server framework. thanks --sigbjorn

On Fri, Jan 16, 2009 at 8:11 PM, Sigbjorn Finne
I'm guessing that you are reading something different into that than what's intended - it's "client-side" in the sense that it can only issue web requests and handle their responses. i.e., it doesn't handle incoming HTTP requests and issue suitable responses. Web server implementation is an interesting problem in its own right, and many packages/frameworks do an admirable job of that already, so no plans (by me) to tackle that via the HTTP package.
But, utilizing the HTTP package as part of any web app you expose on a web server is very much on and not out of bounds. Go for it! :-)
Does that answer your Q? (my apologies if I'm stating the obvious above.)
Thanks for your reply Sigbjorn. Correct me if I am wrong, but it seems to me that by using sockets for incoming connections and receiveHTTP :: Stream s => s -> IO (Result Request_String) and respondHTTP :: Stream s => s -> Response_String -> IO () from Network.HTTP.Stream that I could easily handle incoming requests and send responses back. OK, I need to build up the response myself, but that would be no real limitation for me. Thanks again, Levi

Hi Sigbjorn, On Thu, Jan 15, 2009 at 10:36:35PM -0800, Sigbjorn Finne wrote:
I've yet to gain access to www.haskell.org and update http://www.haskell.org/http,
Perhaps this would be a good point to move the website to the community server? Thanks Ian
participants (10)
-
Bulat Ziganshin
-
Don Stewart
-
Duncan Coutts
-
Henning Thielemann
-
Henning Thielemann
-
Ian Lynagh
-
Jeff Heard
-
Levi Greenspan
-
Sigbjorn Finne
-
Tim Newsham