Neil Mitchell wrote:
I will be releasing this function as part of a library shortly
Alistair Bayley wrote:
no! The code was merely meant to illustrate how a really basic HTTP GET might work. It certainly doesn't deal with a lot of the additional cases, like redirects and resource moves, and non-standards-compliant HTTP servers... there are a large number of webserver implementations which do not respect the HTTP standards (1.0 or 1.1), and HTTP clients (like web browsers) have to go to some lengths in order to get sensible responses out of most of them... if your needs really are very simple, then fine. But be aware that "doing the right thing" with real-world HTTP responses can be a can-o'-worms.
Let's not complicate things too much at the HTTP level. Low-level HTTP is a simple protocol, not hard to implement. You send a request with headers and data, and get a like response. Possibly reuse the connection. That's it. HTTP is useful for many things besides just loading web pages from large public servers. We need a simple, easy to use module that just implements HTTP. I think we have that, or we are close. Loading URLs on the web is an entirely different matter. There is a whole layer of logic that is needed to deal with the mess out there. It builds not just on HTTP, but on various other standard and non-standard protocols. URL loading is a hard problem, but usable solutions are well-known and available. I would suggest that we not re-invent the wheel here. If we want a pure Haskell solution - and that would be nice - we should start with an existing code base that is widely used, stable, and not too messy. Then re-write it in Haskell. Otherwise, just keep spawning wget or cUrl, or use MissingPy. But please don't confuse concerns by mixing URL-loading logic into the HTTP library. They made that mistake in Perl in the early days of the web, before it was clear what was about to happen. There is no reason for us to repeat the mistake. Thanks, Yitz