At 00:12 09/06/04 +0200, Bjorn Bringert wrote:
Graham Klyne wrote:
I'm trying to add an HTTP entity retrieval capability to HaXml using simpleHTTP as the basis of a new function, readHTTP [1], that works very similarly to prelude.readFile (except that its argument is a Network.URI value). Function simpleHTTP still leaves a fair amount of result analysis to be done by the calling program. I'm thinking that it might be convenient to provide a simple function, say: hasResponseData :: Response -> Bool that can be used to drive a simple binary decision along the lines of: return $ if hasResponseData response then (Right $ rspBody response) else (Left $ show response)
Unfortunately you cannot decide from the Response structure alone whether the response has no body or just a zero-length body. RFC 2616 [1] says:
"For response messages, whether or not a message-body is included with a message is dependent on both the request method and the response status code (section 6.1.1). All responses to the HEAD request method MUST NOT include a message-body, even though the presence of entity- header fields might lead one to believe they do. All 1xx (informational), 204 (no content), and 304 (not modified) responses MUST NOT include a message-body. All other responses do include a message-body, although it MAY be of zero length."
However, if we accept that hasResponseData is not accurate for HEAD requests, it should be doable.
Ah, of course. I was glossing the non-GET case, trying to make it easy to to the "readFile" equivalent.
Do you think this would be a reasonable addition to the HTTP module, to make it very easy for a program to issue a simple HTTP GET to retrieve a resource?
That sounds very reasonable.
Another thought I wanted to raise with you concerns the URI authority parser that is currently part of the HTTP module. My revised version of URI already does most of what this micro-parser does (apart from not separating the username and password in userinfo). When the revised URI specification (successor to RFC2396) looks stable, I'm planning to update the Network.URI module in the hierarchical libraries. It occurs to me that the added functionality here could mean that module HTTP might be simplified.
I agree that the HTTP module should use Network.URI to do URI parsing. As soon as the new URI module is in the hierarchical libraries, the HTTP module should start using it. Hmm, actually we may have to wait until GHC and Hugs come with the new URI module. This seems to be general problem with having things in the standard libraries; you are tied to the release cycles of the Haskell implementations.
Yes... although I have permission and access, I'm still wary about updating the library CVS because I don't want to create any problems for existing code. (Though, of course, CVS exists to help in these situations, n'est pas?) E.g. I'm very aware that all my testing is being done on Windows, and mostly using Hugs, which leaves me slightly out-of-step with most library devlopers. (I think that's not a bad thing, but it makes me a little more wary.) I'm sure there's a good solution for this. Meanwhile, copies of my working code are at my web site: http://www.ninebynine.org/Software/HaskellUtils/ in particular: http://www.ninebynine.org/Software/HaskellUtils/Network/ http://www.ninebynine.org/Software/HaskellUtils/HaXml-1.12/
[1] here's a simple readHTTP function I've cooked up... does it look workable?:
This looks like a fine addition to the HTTP module, with some minor tweaks mentioned below.
Ah, I didn't consider including *that* in HTTP.
readHTTP :: URI -> IO String readHTTP uri = withSocketsDo $ do { res <- simpleHTTP (defaultGETRequest uri) ; case res of Left err -> return ("\nError! failed to read "++show uri++": "++show err++"\n") Right rsp -> return $ if hasResponseData rsp then rspBody rsp else show rsp }
I think it might be cleaner if the Left case and the case where there is no response data called fail instead of returning a string. The user could then catch those errors.
I agree that the string return is not a good general solution. (In much of my work on the XML parser, I've been trying to avoid fail, because that can't be caught outside the IO monad. But of course, this case is already in the IO monad, so that's not a problem.)
Concerning the use of withSocketsDo, the original homepage [2] of the HTTP module says:
"It is quite safe to call withSocketsDo multiple times, but that technique has earnt a place on the Winsock Programmers FAQ Lame List [3], since winsock initialisation has performance overhead."
So ideally, the programmer using readHTTP should call withSocketsDo at the top level of the program. But that requires extra work to use the library. I'm not sure what is the right thing to do here. Any thoughts?
This is stretching my Haskell experience, but is it possible to use memoization to achieve just one call? e.g. socketsInitialized :: Bool socketsInitialized = unsafePerformIO ( withSocketsDo ( return True ) ) Then: readHTTP uri = seq socketsInitialized $ do ... or readHTTP uri = if socketsInitialized then ( do ... ) else fail I don't particularly like the use of unsafePerformIO here; is it possible to do it more cleanly? OTOH, it seems that the whole point here is to be impure with respect to 'withSocketsDo', so maybe the "unsafe" option is the only one?
hasResponseData :: Response -> Bool hasResponseData rsp = case rspCode rsp of (2,_,_) -> True otherwise -> False
Just needs some simple modifications as per the RFC 2616 quote above.
OK, it would be good to get that cleaned up.
defaultGETRequest uri = Request { rqURI=uri , rqBody="" , rqHeaders=[ Header HdrContentLength "0" , Header HdrUserAgent "haskell-haxml/0.1" ] , rqMethod=GET }
If I understand RFC 2616 correctly, there may only be a Content-length header if the request contains a body, which as far as I understand (though I can't find that in the RFC) GET requests can't.
Maybe... I just copied that from the Browser module. (No original thought here!) ... Anyway, here's a revised version that I think incorporates your suggestions (I've finessed the HEAD issue somewhat by changing the name and specification of the response analysis function): [[ -- Memoization of withSocketsDo to prevent multiple calls. -- (cf. http://tangentsoft.net/wskfaq/articles/lame-list.html) socketsInitialized :: Bool socketsInitialized = unsafePerformIO ( withSocketsDo $ return True ) readHTTP :: URI -> IO String readHTTP uri = seq socketsInitialized $ do { let req = defaultGETRequest uri ; res <- simpleHTTP req ; case res of Left err -> fail ("Failed to read "++show uri++": "++show err) Right rsp -> return $ if requestCompletedOK req rsp then rspBody rsp else fail (show rsp) } -- Simplified response handling: -- determine if the expected final result from the given request -- is available from the given response value. (e.g. in the case of -- a GET operation, this will be a representation of the resource.) requestCompletedOK :: Request -> Response -> Bool requestCompletedOK req rsp = case (rqMethod req,rspCode rsp) of (_, (2,0,0)) -> True -- OK (_, (2,0,1)) -> True -- Created (_, (2,0,2)) -> False -- Accepted (operation deferred) (_, (2,0,3)) -> True -- Non-authoritative info (GET, (2,0,4)) -> False -- No-content (_, (2,0,4)) -> True -- No-content (GET, (2,0,5)) -> False -- Reset content (can't happen?) (_, (2,0,5)) -> True -- Reset content (_, (2,0,6)) -> True -- Partial content (GET with range) (GET, (3,0,4)) -> True -- Conditional GET, not modified otherwise -> False defaultGETRequest uri = Request { rqURI=uri , rqBody="" , rqHeaders=[ Header HdrContentLength "0" , Header HdrUserAgent "haskell-haxml/0.1" ] , rqMethod=GET } ]] (This code works in an HaXml test case, but is not exhaustively tested.) #g --
[1] http://www.ietf.org/rfc/rfc2616.txt RFC 2616
[2] http://homepages.paradise.net.nz/warrickg/haskell/http/ The HTTP and Browser Modules
[3] http://tangentsoft.net/wskfaq/articles/lame-list.html Winsock Programmer's FAQ, Articles: The Lame List
------------ Graham Klyne For email: http://www.ninebynine.org/#Contact