On Tue, 2 Dec 2008, Paul Brown wrote:
I'm still a little lost as to why this is necessary and/or desirable. Some (if not all) of your use cases would be achievable via proper use of HTTP protocol features (100 Continue, Conditional GET, etc.), and it is a question of whether those are well supported by the current implementation.
I do not know, how '100 Continue' would help, since this is for conditionally sending the body of the HTTP request, not conditionally receiving the body of the HTTP response, right? However, you are right, in principle it would help me to use HTTP features, like the Accept header. Using it, I could tell, that I only want certain types of data to be send. However, e.g. the original HWS ignores the Accept header and when I checked other web servers, they do so, as well. So it still seems to be the best, to get the header via a GET request and cancel the transfer, once I have seen enough of the header.
For the laziness in handling response bodies, I think you'd actually want to expose an iteration semantic implemented on top of HTTP chunked encoding and ranged requests, and that would live at a higher level in the stack than the TCP layer.
Yes, I also want lazy download of the body. Certainly an iteratee implementation could be cleaner, but probably considerably different from the current implementation. I'm afraid 'Range' is supported as bad as 'Accept' - and what about dynamically generated (maybe even infinite) content? So I think I should not bother the web server with the way I want to download its content, but just mimic a stupid browser. My current state can be found here: http://code.haskell.org/~thielema/http/ I do not want to fork and I hope the patches are accepted somewhen and nobody adds conflicting patches in the meantime, since this would mean I have to re-record all my changes in order to avoid the dreaded exponential time darcs conflict resolution. However, currently there are only changes behind the scenes and no new visible features. I'm now using monad transformers from my explicit-exception package, which simplified a lot of the manual exception handling before. I removed some 'reverse's, which are a laziness killer, but I'm still not at the end. Chunked transfer is particularly difficult with respect to laziness, although I do not know why. I have added a Monad which further abstracts the Stream concept. It allows us to do HTTP communication in absence of IO, which is perfect for unit testing and allows lazy processing without unsafeInterleaveIO - you only need the unsafeInterleaveIO hidden in hGetContents.