Re: [Haskell-cafe] [web-devel] http-enumerator: redirects, streaming and keep-alive

On Wed, Feb 2, 2011 at 11:06 PM, Michael Snoyman wrote:

On Thu, Feb 3, 2011 at 12:00 AM, Antoine Latter wrote:

...

On Wed, Feb 2, 2011 at 3:01 PM, Felipe Almeida Lessa wrote:

...

On Wed, Feb 2, 2011 at 6:30 PM, Antoine Latter wrote:

...

Or you could remove the socket from the map while it's in use.

And what about connection limits?  We shouldn't create a thousand connections to the same host =).

Not a bad idea, but that may be creeping outside the scope of the bug as reported.

If you're writing a web-scraper/spider, it is true you might need some sort of higher-level manager to handle concurrent access. I'm not sure what that would look like, though

...

-- Felipe.

I think Felipe's point is that using the approach I outlined, we will never spawn more than one connection to a single host. By simply taking the socket out of the map, there's no limit to the number of sockets that will be created.

Ah, then you would block one thread on another if the host info is in the map? I get it. Strictly one socket per host might be a tricky invariant to maintain - you'd have to insert an empty socket MVar in the map on lookup failure, and the shuffling of the nested MVars seems tricky off the top of my head. But even without all of that I don't think you'd be flooding the host any more than without any keepalive support, which was what I was getting at. This is just speculation on my part - I don't have a stake in this, it just sounded like an interesting problem. I'd hate to be steering this into a design black hole. Antoine

Michael