Tomasz Zielonka wrote:

AFAIK, Handles have finalisers which close them, but I don't know if GHC triggers garbage collection when file descriptors run out. If not, you will have problems if you manage to run out of fds between GCs.

Yes, in GHC and Hugs handles are flushed and closed automatically during GC if they are garbage and not already closed. But no GC is triggered when one runs out of native file descriptors. I'm not exactly sure about NHC, but I guess it's the same there.

How about using bracket to introduce explicit close on end of scope?

I would recommend this, too, e.g. via something like: withFile :: FilePath -> IOMode -> (Handle -> IO a) -> IO a withFile fileName mode = bracket (openFile fileName mode) hClose This ensures that the handle is closed immediately even in the presence of exceptions. Cheers, S.

On Sun, Oct 24, 2004 at 02:16:50PM +0200, Peter Simons wrote:

Tomasz Zielonka writes:

...

AFAIK, Handles have finalisers which close them, but I don't know if GHC triggers garbage collection when file descriptors run out. If not, you will have problems if you manage to run out of fds between GCs.

Thank you for answering.

Now there is only one problem: Assuming I could _not_ use 'bracket', 'withFile', 'finally' or any of the other usual scope-guarding techniques, what would I do? (The handle has to be passed up to the outside of the scope in which it was opened.) Refactoring comes to the mind... ;)

If I stored the handle in an 'MVar' and attached an MVar-finalizer to that, would that work better? Would the MVar's finalizer be run any sooner than the one attached to the handle anyway? Both are just finalizers, so it won't make any difference AFAICS.

Or can I explicitly trigger garbage collection somehow? Say, in case I receive an exception telling me that file descriptors are running out? Which exception would I even get in this case?

Peter

You could try using something like this instead of the normal openFile: (or make it even less portable by importing GHC.IOBase and only retrying on ResourceExhausted) module Main where import IO import System.Mem myOpenFile path mode = catch (openFile path mode) $ \_ -> do putStrLn "==> Open failed. Retrying <==" performGC openFile path mode open = openFile "/tmp/foo" ReadMode myOpen = myOpenFile "/tmp/foo" ReadMode Example: -- Let's eat all available filedescriptors. *Main> sequence (repeat open) Loading package haskell98 ... linking ... done. *** Exception: /tmp/foo: openFile: resource exhausted (Too many open files) -- They are indeed all gone. *Main> h <- open *** Exception: /tmp/foo: openFile: resource exhausted (Too many open files) -- What about performing garbage collection? *Main> h <- myOpen ==> Open failed. Retrying <== *Main> h {handle: /tmp/foo} *Main> Groeten, Remi -- Nobody can be exactly like me. Even I have trouble doing it.

Peter Simons writes:

I wish that were possible! I use bracket-style resource allocation wherever I can, but in this case the Handle is the _result_ of a rather complex computation, so it has to leave the scope in which is was opened. A finalizer is all I can use.

It does not necessarily make bracket unavailable. Just put it somewhere outside, when it's known when the file will need to be closed. The whole action which opens the file inside it should be the "acquire" part of the bracket. Asynchronous exceptions will be blocked during executing it, so it should not be too long unless it unblocks them for longer parts. Of course there are also some cases when the file handle is deeply buried in some data structures, and we don't know when they will be freed. -- __("< Marcin Kowalczyk \__/ qrczak@knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/

The _result_ of a rather complex computation, so it has to

return a function that returns the handle. for example: handleOpener = complex function bracket handleOpener close (\h -> do Keean Peter Simons wrote:

Remi Turk writes:

...

...

Assuming I could _not_ use 'bracket', 'withFile', 'finally' or any of the other usual scope-guarding techniques [...]

...

Refactoring comes to the mind... ;)

I wish that were possible! I use bracket-style resource allocation wherever I can, but in this case the Handle is the _result_ of a rather complex computation, so it has to leave the scope in which is was opened. A finalizer is all I can use.

...

performGC

That is the function I was looking for.

Well, it's actually not at all the function I was looking for, but given the circumstances, it seems to be the only solution. Alternatives are highly sought after, though!

Peter

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

You return the handle in the state, are you saying the complex funtion does some IO on the handle first? I don't see why this excludes doing it the other way round, passing the work function to your complex function as an argument. Keean Peter Simons wrote:

Keean Schupke writes:

...

...

The _result_ of a rather complex computation

...

return a function that returns the handle.

The idea is good. :-) You'll find my other posting explains that a bit more: My main driver doesn't know about the handle; that's just one more entry in the state of the connection handler. So the connection handler cannot return a complex function, because _it_ is the one who wants to use the handle, not the function that called it. What I return is a state which contains the handle, not the handle per se.

Peter

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

"Conal Elliott" writes:

I'm puzzled why explicit bracketing is seen as an acceptable solution. It seems to me that bracketing has the same drawbacks as explicit memory management, namely that it sometimes retains the resource (e.g., memory or file descriptor) longer than necessary (resource leak) and sometimes not long enough (potentially disastrous programmer error).

If used correctly, it will retain it about the right amount of time, while with garbage collection it's *impossible* to ensure that it will not be retained too long. Most GC schemes don't guarantee promptness of collection. It doesn't matter for pure memory, because GC will be performed as soon as enough memory has been allocated, but it matters for other resources, except for programs which don't open many files at all. -- __("< Marcin Kowalczyk \__/ qrczak@knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/

File descriptors aren't simply a "resource" in the sense that memory is. Closing a descriptor may have significance beyond the process which closes it. If it refers to the write end of a pipe or socket, closing it may cause the reader to receive EOF; if it refers to a file, any locks will be released; and so on.

Yes, I think this is the crux of the matter: closing a file-descriptor is not just a question of releasing resources, but it has actual side-effects visible by the outside world (e.g. closing a pipe, or actual write-back if the file is on NFS, ...) so it is important for programs to close FDs explicitly when it needs to happen, rather than leave it to the GC to do it whenever it feels like (which can be much much later if you get unlucky with your generations). Stefan

Conal Elliott wrote:

I'm puzzled why explicit bracketing is seen as an acceptable solution. It seems to me that bracketing has the same drawbacks as explicit memory management, namely that it sometimes retains the resource (e.g., memory or file descriptor) longer than necessary (resource leak) and sometimes not long enough (potentially disastrous programmer error). Whether the resource is system RAM, file descriptors, video memory, fonts, brushes, bitmaps, graphics contexts, 3D polygon meshes, or whatever, I'd like GC to track the resource use and free unused resources correctly and efficiently.

The lifetime problems are worse without explicit bracketing IMHO: One has to wait for the next GC to free them (resource leaks) and even without bracketing there's nothing to stop you from using a Handle after closing. Furthermore, the main difference between simple RAM and more "external things" like Handles is that the effect of keeping them occupied is observable from outside our happy Haskell world, e.g. a server socket might stay open, taking away a port number, an external device with exclusive access is unusable until the next GC, my valuable texture memory on the graphics card contains unused textures until the GC runs, etc. etc. Finalizers are not a solution for these problems, a fact that e.g. a lot of novice Java programmers have to learn when they do their first "real world" programs... IMHO it would be best to use explicit bracketing where possible, and hope for the RTS/GC to try its best when one runs out of a given resource. Admittedly the current Haskell implementations could be improved a little bit in the last respect. Cheers, S.

I'm puzzled why explicit bracketing is seen as an acceptable solution. It seems to me that bracketing has the same drawbacks as explicit memory management, namely that it sometimes retains the resource (e.g., memory or file descriptor) longer than necessary (resource leak) and sometimes not long enough (potentially disastrous programmer error). Whether

Thanks, Remi (and other responders). I did indeed mean that GC of non-RAM resources would be triggered by _those_ resources getting low rather than RAM. That is, there would be many per-type collectors, each with its own trigger. And maybe really only a small number different collection algorithms, but clearly separated spaces and triggers. This idea sounds fairly simple to me, so maybe not "only theoretical". More comments? - Conal -----Original Message----- From: Remi Turk [mailto:rturk@science.uva.nl] Sent: Sunday, October 24, 2004 1:08 PM To: Conal Elliott Cc: haskell-cafe@haskell.org Subject: Re: [Haskell-cafe] Are handles garbage-collected? On Sun, Oct 24, 2004 at 12:19:59PM -0700, Conal Elliott wrote: the

resource is system RAM, file descriptors, video memory, fonts, brushes, bitmaps, graphics contexts, 3D polygon meshes, or whatever, I'd like GC to track the resource use and free unused resources correctly and efficiently.

Cheers,

- Conal

IMO, it does indeed have those same drawbacks. (Although the traditional "explicit memory management model" is alloc/free, which is much worse than bracket/withFile) However, Garbage Collection is usually based only on memory. Using GC for file-handle-closing therefore means that one will close garbage file-handles when memory is getting low, instead of when file-handles are almost used up... The theoretical solution (and probably _only_ theoretical) is implementing a lot of garbage collectors: one for memory, one for open files, one for sockets, one for 3D polygon meshes etc etc... Greetings, Remi -- Nobody can be exactly like me. Even I have trouble doing it.

On Mon, Oct 25, 2004 at 08:46:41AM +0200, Ketil Malde wrote:

Remi Turk writes:

...

IMO, [bracket] does indeed have those same drawbacks. (Although the traditional "explicit memory management model" is alloc/free, which is much worse than bracket/withFile)

Isn't bracket more like stack allocated memory? And most problems with explicit memory management related to heap (as you indicate)?

I think you're right. (WRT usage (only): AFAICS even Foreign.Marshal.Alloc.alloca doesn't actually use the stack.)

...

The theoretical solution (and probably _only_ theoretical) is implementing a lot of garbage collectors: one for memory, one for open files, one for sockets, one for 3D polygon meshes etc etc...

...or have a number of available file handles that is limited by memory? :-)

Would be nice if OSes actually worked that way. Then again, I don't think _everything_ can be made dependent only on available memory. Groetjes, Remi -- Nobody can be exactly like me. Even I have trouble doing it.