http-enumerator: redirects, streaming and keep-alive

Hi all, There are a number of new feature requests for http-enumerator, and I wanted to discuss some possible API changes with everyone: * Allow keep-alive requests, which will reuse the same connection. * Allow client code to determine whether it accepts a server's SSL certificate. * Allow the request body to be an Enumerator so we can send large request bodies without using lazy IO. This is already implemented via streamingHttp, but it would make more sense to modify the Request datatype to replace L.ByteString with (Int, Enumerator ByteString IO ()). * Fix redirect behavior. It seems that the right thing to do is change a POST into a GET and remove the request body for a 303, and resend the same request for all other 3xx codes. However, when using an Enumerator for the request body, it is not guaranteed that we can resend an Enumerator. For addressing the last two, my plan is to: * Change the Request datatype as mentioned, but provide a convenience method for converting a lazy ByteString into a (Int, Enumerator ByteString IO ()). * Implement redirecting as I have described, and explain in the documentation that automatic redirecting can only work with Enumerators that can be run multiple times. AFAIK, this works with most standard Enumerators, such as enumFile, and with the helper function I will provide to convert lazy ByteStrings to Enumerators. * Remove streamingHttp, as it will no longer be necessary. As far as keep-alive goes, I still need to do a bit more research, but my basic idea (with credit to Bryan O'Sullivan): * http (and family) will all take an extra argument, Maybe Manager. * Manager will be an abstract type that will keep an MVar (Map (Host, Port, IsSecure) Socket). * If http is provided with a Manager, then it uses the Socket available in the Manager. If none is available, it creates a new Socket and places it in the Manager. * If http is *not* provided with a Manager, then it creates a new socket and closes it before returning. * There will be a newManager :: IO Manager, and a closeManager :: Manager -> IO (), which closes all Sockets in the Manager and empties out the inner Map. Some open questions are: * Who is responsible to recover if a server is no longer responding on a Socket from the Manager: http or the client code? I would assume http. * I haven't fully thought through how this will work with secure connections: most likely in addition to storing the Socket in the Manager I will need to store some certificate data. And speaking of certificates, the main concern I have here is data types: since http-enumerator can use either tls or OpenSSL, there isn't a single certificate datatype I can use. If someone wants to help out and write some code to convert an OpenSSL X509 datatype into a Certificate from the certificate package (which tls uses), I will be very grateful. Assuming we had such a unified datatype, I would recommend changing Request's secure record to be Certificate -> IO Bool, where returning True means the certificate is accepted and False means it is rejected. To get the current functionality of trusting any certificate, we would use (const $ return True). Any thoughts? Michael

On Wed, Feb 2, 2011 at 11:57 AM, Michael Snoyman
As far as keep-alive goes, I still need to do a bit more research, but my basic idea (with credit to Bryan O'Sullivan):
* http (and family) will all take an extra argument, Maybe Manager. * Manager will be an abstract type that will keep an MVar (Map (Host, Port, IsSecure) Socket). * If http is provided with a Manager, then it uses the Socket available in the Manager. If none is available, it creates a new Socket and places it in the Manager. * If http is *not* provided with a Manager, then it creates a new socket and closes it before returning. * There will be a newManager :: IO Manager, and a closeManager :: Manager -> IO (), which closes all Sockets in the Manager and empties out the inner Map.
How about concurrent use of Manager? Should we do A) do m <- newManager forM xs $ forkIO $ doSomething m B) forM xs $ forkIO $ do m <- newManager doSomething m While B) should work with any sane Manager implementation, it is not optimal. If all your connections are to the same host, than both approaches are the same. But if access hosts O and P, for example, than it is possible that Manager m1 has an open connection to O, but you try connect to O using another Manager m2. That means that ideally we should support approach A) as well. However, to support A a simple Map inside an MVar isn't sufficient. Cheers! =) -- Felipe.

On Wed, Feb 2, 2011 at 10:15 PM, Felipe Almeida Lessa
On Wed, Feb 2, 2011 at 11:57 AM, Michael Snoyman
wrote: As far as keep-alive goes, I still need to do a bit more research, but my basic idea (with credit to Bryan O'Sullivan):
* http (and family) will all take an extra argument, Maybe Manager. * Manager will be an abstract type that will keep an MVar (Map (Host, Port, IsSecure) Socket). * If http is provided with a Manager, then it uses the Socket available in the Manager. If none is available, it creates a new Socket and places it in the Manager. * If http is *not* provided with a Manager, then it creates a new socket and closes it before returning. * There will be a newManager :: IO Manager, and a closeManager :: Manager -> IO (), which closes all Sockets in the Manager and empties out the inner Map.
How about concurrent use of Manager? Should we do
A) do m <- newManager forM xs $ forkIO $ doSomething m
B) forM xs $ forkIO $ do m <- newManager doSomething m
While B) should work with any sane Manager implementation, it is not optimal. If all your connections are to the same host, than both approaches are the same. But if access hosts O and P, for example, than it is possible that Manager m1 has an open connection to O, but you try connect to O using another Manager m2. That means that ideally we should support approach A) as well. However, to support A a simple Map inside an MVar isn't sufficient.
Good point: it should be a MVar (Map HostInfo (MVar Socket)) I think* Thanks, Michael * It's late here, just make sure I'm not saying something stupid ;).
participants (2)
-
Felipe Almeida Lessa
-
Michael Snoyman