HTTP package: connection closing bug?
Hi, I'm using HTTP-3001.0.4 with GHC 6.8.3 under Mac OS X 10.5 and Debian Lenny. If I open and close many connections, I eventually get the error. Anyone else seen this before? *** Exception: socket: resource exhausted (Too many open files) Apparently, the socket is not closing. in TCP.hs:168, the call to shutdown is raising a "socket already closed" exception. So, the program never reaches the sClose on line 171. A simple fix is: wanderlust:Network arjun$ diff TCP.hs.original TCP.hs 172c172 < } ---
} `Exception.catch` (\_ -> sClose sk)
The code that demonstrates this problem on Mac OS X is:
module Main where
import Network.HTTP import Network.URI (parseURI) import Data.Maybe (fromJust) import Control.Monad
googleM n = do s <- simpleHTTP $ Request (fromJust $ parseURI "http://www.google.com ") GET [] "" when (n `mod` 100 == 0) $ putStrLn (show n) return ()
main = mapM_ googleM [1..5000]
On Debian Lenny, the code works fine with google.com. However, when I try to connect to my local CouchDB server, it throws an exception after roughly 1000 connections. Thanks. Arjun
On Aug 21, 2008, at 3:23 PM, Arjun Guha wrote:
Hi,
I'm using HTTP-3001.0.4 with GHC 6.8.3 under Mac OS X 10.5 and Debian Lenny. If I open and close many connections, I eventually get the error. Anyone else seen this before?
I've never looked into it deeply, but I just wanted to chime in that I've seen this, too. Aaron
On Thu, Aug 21, 2008 at 11:44 PM, Aaron Tomb
I've never looked into it deeply, but I just wanted to chime in that I've seen this, too.
I may be confusing it with something else, but I seem to recall something very similar from a while back, in a blog post somewhere? I can't remember the details, but someone was writing a server application of some kind and came across a socket leaking error that was a one-line fix line this. The description and the fix just seem so familiar. :-\ Sorry, this is the most incoherent and useless post! Maybe someone else has been paying more attention and can remember who it was, and where. Cheers, D -- Dougal Stanton dougal@dougalstanton.net // http://www.dougalstanton.net
On Thu, Aug 21, 2008 at 11:56 PM, Dougal Stanton
Aha, I knew I wasn't dreaming!
http://mult.ifario.us/p/a-short-adventure-with-simplehttp
Paul Brown posted this discussion back in February. It looks like the same thing. Has there been an update of HTTP since then?
Nope, it hasn't been updated. I'm just running a locally patched version. This is the same issue, thanks! Arjun
On Fri, 22 Aug 2008 11:12:08 Arjun Guha wrote:
Aha, I knew I wasn't dreaming!
http://mult.ifario.us/p/a-short-adventure-with-simplehttp
Paul Brown posted this discussion back in February. It looks like the same thing. Has there been an update of HTTP since then?
Nope, it hasn't been updated. I'm just running a locally patched version. This is the same issue, thanks!
Beware of the browse function in Browser, there is a connection leak in there as well, at least there was back in March or April. Not sure if it has been fixed in the official repos. I seem to recall the sockets getting put into some half-closed state, presumably when GHC GC'd them, eventually exhausting limits. That may be particular to running on Linux, I guess. Dan
On Thu, 2008-08-21 at 19:12 -0400, Arjun Guha wrote:
Aha, I knew I wasn't dreaming!
http://mult.ifario.us/p/a-short-adventure-with-simplehttp
Paul Brown posted this discussion back in February. It looks like the same thing. Has there been an update of HTTP since then?
Nope, it hasn't been updated. I'm just running a locally patched version. This is the same issue, thanks!
If someone could figure out the right fix and send it to the maintainer that'd be great. We do need a decent working pure-haskell http package. For one thing cabal-install uses the HTTP package and it has to work for downloading dozens or hundreds of packages in a single session. As an aside, cabal-install could use the Network.Browser module better, by using a single call to browse for all downloads rather than one per download. As I understand it, that'd allow it to re-use a single connection. Duncan
On Fri, 22 Aug 2008 14:15:48 Duncan Coutts wrote:
On Thu, 2008-08-21 at 19:12 -0400, Arjun Guha wrote:
Aha, I knew I wasn't dreaming!
http://mult.ifario.us/p/a-short-adventure-with-simplehttp
Paul Brown posted this discussion back in February. It looks like the same thing. Has there been an update of HTTP since then?
Nope, it hasn't been updated. I'm just running a locally patched version. This is the same issue, thanks!
Looks like there is a patch committed to http's darcs repo, 3rd May, from prb to address this problem.
If someone could figure out the right fix and send it to the maintainer that'd be great. We do need a decent working pure-haskell http package.
Agreed. I think http needs serious work before it would fit the bill though.
For one thing cabal-install uses the HTTP package and it has to work for downloading dozens or hundreds of packages in a single session.
As an aside, cabal-install could use the Network.Browser module better, by using a single call to browse for all downloads rather than one per download. As I understand it, that'd allow it to re-use a single connection.
It's quite likely you are leaking a connection on every call to browse. I recall someone asking about how to configure hackage's Apache to handle more than 1000 open connections... maybe cabal-install is the reason for it needing so many concurrent connections. I've sent a patch for Network.Browser.browse to Bjorn. Dan
participants (5)
-
Aaron Tomb -
Arjun Guha -
Daniel McAllansmith -
Dougal Stanton -
Duncan Coutts