Conduit missing closing file handles
I have a pretty simple program for converting ZIP to TAR archives using the 'zip' and 'tar-conduit' packages, both using Conduit for streaming data. In principle the program works, but if the ZIP archive contains a non-trivial number of files, the program breaks because of exceeded limit of open file handles: $ zip2tar /usr/share/libreoffice/share/config/images_colibre.zip >/tmp/images_colibre.tar zip2tar: Uncaught exception ghc-internal:GHC.Internal.IO.Exception.IOException: /usr/share/libreoffice/share/config/images_colibre.zip: openFile: resource exhausted (Too many open files) While handling /usr/share/libreoffice/share/config/images_colibre.zip: openFile: resource exhausted (Too many open files) HasCallStack backtrace: throwIO, called at ./Control/Monad/Trans/Resource.hs:195:13 in resourcet-1.3.0-8b355f9483371034fb75ff40adb2e07157502721c941fbf8a6f607df6dd81546:Control.Monad.Trans.Resource The main function is essentially this one: [1] convertZipToTar :: (Zip.EntrySelector -> Zip.EntryDescription -> CTar.FileInfo) -> Sort -> FilePath -> IO () convertZipToTar fileInfoFromZipEntry_ sortMode zipPath = Zip.withArchive zipPath $ do entries <- Map.toList <$> Zip.getEntries files <- for entries $ \(selector, descr) -> flip fmap (Zip.getEntrySource selector) $ \src -> src .| (C.yield (Left $ fileInfoFromZipEntry_ selector descr) >> C.mapC Right) C.liftIO $ runConduitRes $ sequence_ files .| void CTar.tar .| C.stdoutC 'Zip.getEntrySource selector' generates a conduit for reading a file from the ZIP archive. It opens the archive anew for every accessed file, but it calls Conduit.Binary.sourceIOHandle, which promises to close a file promptly after reading. [2] It seems that files are not immediately closed after reading, but I cannot spot the cause. I suspected that "sequence_ files" might concatenate the Conduits while accumulating open file handles, but I think it should not and I did not find a matching alternative to sequence_. [1] https://hackage-content.haskell.org/package/zip2tar-0.0/src/src/Main.hs [2] https://hackage-content.haskell.org/package/zip-2.2.1/docs/src/Codec.Archive...
On Thu, Apr 30, 2026 at 12:07:08PM +0200, Henning Thielemann wrote:
'Zip.getEntrySource selector' generates a conduit for reading a file from the ZIP archive. It opens the archive anew for every accessed file, but it calls Conduit.Binary.sourceIOHandle, which promises to close a file promptly after reading. [2]
You have to be a bit careful about what "immediately after reading" means. Actually, I think it means "immediately after the file handle indicates it has reached end of file". `getEntrySource`[1] calls `sourceEntry`[2] which runs source .| CB.isolate (fromIntegral edCompressedSize) .| decompress Now, source is source = CB.sourceIOHandle $ do ... But `sourceIOHandle` will only close its handle after it determines that it has reached EOF. If you look at the argument to `sourceIOHandle` in [2] you'll see that it just seeks to the start position of the target file in the archive. It's CB.isolate which is used to limit reading to the extent of the target file only. It brings down the pipeline once it is done but because EOF is never reached in the handle (i.e. the final bytes of the entire .zip file are never read) the cleanup action doesn't trigger at this point. The cleanup action will nonetheless trigger at the end of the runConduitRes block, but that's too late. You've already opened too many files. I have a simpler reproducer for this below. Maybe this is salvagable by putting a ResourceT block within the definition of `sourceEntry`, maybe not. I don't have the finer details of ResourceT in working memory. However, there are numerous problems with Conduit and Pipes bracketing that make it very difficult to use them safely. My effect system Bluefin solves this problem, as I elaborated in my article "Bluefin streams finalize promptly". https://h2.jaguarpaw.co.uk/posts/bluefin-streams-finalize-promptly/ However, there is sadly not a large ecosystem around Bluefin, and no Zip library. I'm happy to help you implement what you need if you're interested! Tom [1] https://hackage-content.haskell.org/package/zip-2.2.1/docs/src/Codec.Archive... [2] https://hackage-content.haskell.org/package/zip-2.2.1/docs/src/Codec.Archive... {- cabal: build-depends: conduit, base -} {-# LANGUAGE GHC2021 #-} import Conduit (liftIO, mapM_C, runConduitRes, (.|)) import Control.Monad (forever, when) import Data.Conduit.Combinators (sourceIOHandle) import Data.Conduit.List qualified as CB import Data.IORef (modifyIORef', newIORef, readIORef, writeIORef) import System.IO (IOMode (ReadMode), openFile) -- cabal-script-test27.hs: /dev/zero: openFile: resource exhausted (Too -- many open files) main :: IO () main = do ref <- newIORef @Int 0 runConduitRes $ forever $ do sourceIOHandle (openFile "/dev/zero" ReadMode) .| CB.isolate 1 .| mapM_C ( \_ -> liftIO $ do modifyIORef' ref (+ 1) n <- readIORef ref when (n `mod` 20 == 0) $ do Prelude.print n )
On Thu, 30 Apr 2026, Tom Ellis wrote:
You have to be a bit careful about what "immediately after reading" means. Actually, I think it means "immediately after the file handle indicates it has reached end of file".
Interesting. This would explain my observations. But if the file handle is not closed immediately, this Conduit is not much better than lazy readFile, is it?
I have a simpler reproducer for this below.
Yes, this is a nice minimal demonstration of the problem.
However, there is sadly not a large ecosystem around Bluefin, and no Zip library.
My program would also require a Tar library.
On Fri, May 01, 2026 at 11:26:34AM +0200, Henning Thielemann wrote:
On Thu, 30 Apr 2026, Tom Ellis wrote:
You have to be a bit careful about what "immediately after reading" means. Actually, I think it means "immediately after the file handle indicates it has reached end of file".
Interesting. This would explain my observations. But if the file handle is not closed immediately, this Conduit is not much better than lazy readFile, is it?
In this case `conduit` is no better than lazy readFile. Unfortunately I think the approaches to streaming effects in `conduit`, `pipes` and `streaming` which build effects out of data (which I call the "synthetic" approach to effects) is doomed with regard to resource safety. In fact I'm astonished they persisted so long. Bluefin been a direct wrapper of `IO` (what I call an "analytic" approach to effects) has no such problems, however. (I don't know about `streamly`. I haven't looked at it and I understand it takes a somewhat different approach.)
However, there is sadly not a large ecosystem around Bluefin, and no Zip library.
My program would also require a Tar library.
Can you please try this Bluefin version of convertZipToTar and confirm whether it does what you want without leaking handles? https://github.com/tomjaguarpaw/ad/blob/master/BluefinZipTar/BluefinZipTar.h... If so I can tidy up the library and publish it as something minimally usable. Tom
On Fri, 1 May 2026, Tom Ellis wrote:
On Fri, May 01, 2026 at 11:26:34AM +0200, Henning Thielemann wrote:
My program would also require a Tar library.
Can you please try this Bluefin version of convertZipToTar and confirm whether it does what you want without leaking handles?
https://github.com/tomjaguarpaw/ad/blob/master/BluefinZipTar/BluefinZipTar.h...
Ok great, I tried and now I get an error from tar-conduit: zip2tar-bluefin: Uncaught exception tar-conduit-0.4.1-3860201d51f91b125377913a0184ee22827a51e5a78eec1b1230907a77ca3cf7:Data.Conduit.Tar.Types.TarCreateException: TarCreationError "<tarPayload>: Too much payload (32752) for file with size (412): avmedia/res/av02048.png" HasCallStack backtrace: throwIO, called at ./Control/Concurrent/Async/Internal.hs:667:24 in async-2.2.6-d55fde70b8a48643acd98dd03fa271c8842e964017add43e3365c491d02ab733:Control.Concurrent.Async.Internal However, this may still be a mistake in the Zip part, reading too much data from a compressed file.
On Fri, May 01, 2026 at 04:53:33PM +0200, Henning Thielemann wrote:
On Fri, 1 May 2026, Tom Ellis wrote:
On Fri, May 01, 2026 at 11:26:34AM +0200, Henning Thielemann wrote:
My program would also require a Tar library.
Can you please try this Bluefin version of convertZipToTar and confirm whether it does what you want without leaking handles?
https://github.com/tomjaguarpaw/ad/blob/master/BluefinZipTar/BluefinZipTar.h...
Ok great, I tried and now I get an error from tar-conduit:
zip2tar-bluefin: Uncaught exception tar-conduit-0.4.1-3860201d51f91b125377913a0184ee22827a51e5a78eec1b1230907a77ca3cf7:Data.Conduit.Tar.Types.TarCreateException:
TarCreationError "<tarPayload>: Too much payload (32752) for file with size (412): avmedia/res/av02048.png"
HasCallStack backtrace: throwIO, called at ./Control/Concurrent/Async/Internal.hs:667:24 in async-2.2.6-d55fde70b8a48643acd98dd03fa271c8842e964017add43e3365c491d02ab733:Control.Concurrent.Async.Internal
Thanks for trying it. Do you have some code you can share that shows how to plug it all together into an executable? In particular, I'm not sure what I should use for the argument of type: Zip.EntrySelector -> Zip.EntryDescription -> CTar.FileInfo Tom
On Fri, 1 May 2026, Tom Ellis wrote:
On Fri, May 01, 2026 at 04:53:33PM +0200, Henning Thielemann wrote:
Ok great, I tried and now I get an error from tar-conduit:
zip2tar-bluefin: Uncaught exception tar-conduit-0.4.1-3860201d51f91b125377913a0184ee22827a51e5a78eec1b1230907a77ca3cf7:Data.Conduit.Tar.Types.TarCreateException:
TarCreationError "<tarPayload>: Too much payload (32752) for file with size (412): avmedia/res/av02048.png"
HasCallStack backtrace: throwIO, called at ./Control/Concurrent/Async/Internal.hs:667:24 in async-2.2.6-d55fde70b8a48643acd98dd03fa271c8842e964017add43e3365c491d02ab733:Control.Concurrent.Async.Internal
Thanks for trying it. Do you have some code you can share that shows how to plug it all together into an executable? In particular, I'm not sure what I should use for the argument of type:
Zip.EntrySelector -> Zip.EntryDescription -> CTar.FileInfo
This is fileInfoFromZipEntry [3] after applying it to the options userName userId groupName groupId attrSrc. For simplicity you can call it with default values: fileInfoFromZipEntry mempty 0 mempty 0 AttributeWindows [3] https://hackage-content.haskell.org/package/zip2tar-0.0/src/src/Main.hs
On Fri, May 01, 2026 at 05:38:00PM +0200, Henning Thielemann wrote:
On Fri, 1 May 2026, Tom Ellis wrote:
On Fri, May 01, 2026 at 04:53:33PM +0200, Henning Thielemann wrote:
Ok great, I tried and now I get an error from tar-conduit:
TarCreationError "<tarPayload>: Too much payload (32752) for file with size (412): avmedia/res/av02048.png"
Thanks for trying it. Do you have some code you can share that shows how to plug it all together into an executable?
This is fileInfoFromZipEntry [3] after applying it to the options userName userId groupName groupId attrSrc.
Thanks, I fixed the bug. You can get the fixed version from: https://github.com/tomjaguarpaw/ad/blob/353fd275123a1ca50c7981349ec25c3997d4... Please let me know if it works. (I included a main for testing purposes, but you still just need convertZipToTar and its dependencies.) Tom PS The bug was that I was using Data.Conduit.List.isolate: https://hackage-content.haskell.org/package/conduit-1.3.6.1/docs/Data-Condui... but I should have used Data.Conduit.Binary.isolate: https://hackage-content.haskell.org/package/conduit-extra-1.3.8/docs/Data-Co... Fixed in this commit: https://github.com/tomjaguarpaw/ad/commit/15f179e6827da5b58123f1dbb0f6542a6f...
On Fri, 1 May 2026, Tom Ellis wrote:
Thanks, I fixed the bug. You can get the fixed version from:
https://github.com/tomjaguarpaw/ad/blob/353fd275123a1ca50c7981349ec25c3997d4...
Please let me know if it works. (I included a main for testing purposes, but you still just need convertZipToTar and its dependencies.)
It works indeed, now! That is, you are in a way able to convert Conduits to your Bluefin transformers?
The bug was that I was using Data.Conduit.List.isolate:
https://hackage-content.haskell.org/package/conduit-1.3.6.1/docs/Data-Condui...
but I should have used Data.Conduit.Binary.isolate:
https://hackage-content.haskell.org/package/conduit-extra-1.3.8/docs/Data-Co...
I see, that's easy to mix up.
On Fri, May 01, 2026 at 07:42:00PM +0200, Henning Thielemann wrote:
On Fri, 1 May 2026, Tom Ellis wrote:
Thanks, I fixed the bug. You can get the fixed version from:
https://github.com/tomjaguarpaw/ad/blob/353fd275123a1ca50c7981349ec25c3997d4...
Please let me know if it works. (I included a main for testing purposes, but you still just need convertZipToTar and its dependencies.)
It works indeed, now!
Great. If you think this kind of thing is useful to you I can try and make it suitable for publishing in a library.
That is, you are in a way able to convert Conduits to your Bluefin transformers?
Yes, it's fairly easy to run a Conduit in a form compatible with Bluefin, like fromConduit does here: https://github.com/tomjaguarpaw/ad/blob/a6a681587114af3c2210cee4b31d6f505a65... But getEntrySource was written with the wrong primitives, blocking prompt finalization, so I had to rewrite that. Tom
On Fri, 1 May 2026, Tom Ellis wrote:
On Fri, May 01, 2026 at 07:42:00PM +0200, Henning Thielemann wrote:
On Fri, 1 May 2026, Tom Ellis wrote:
Thanks, I fixed the bug. You can get the fixed version from:
https://github.com/tomjaguarpaw/ad/blob/353fd275123a1ca50c7981349ec25c3997d4...
Please let me know if it works. (I included a main for testing purposes, but you still just need convertZipToTar and its dependencies.)
It works indeed, now!
Great. If you think this kind of thing is useful to you I can try and make it suitable for publishing in a library.
I'd say with the current excess of open file handles, zip2tar is pretty useless.
That is, you are in a way able to convert Conduits to your Bluefin transformers?
Yes, it's fairly easy to run a Conduit in a form compatible with Bluefin, like fromConduit does here:
https://github.com/tomjaguarpaw/ad/blob/a6a681587114af3c2210cee4b31d6f505a65...
But getEntrySource was written with the wrong primitives, blocking prompt finalization, so I had to rewrite that.
Instead of writing and maintaining two new libraries for ZIP and TAR handling, would it work to submit improved getEntrySource based on Conduit to the 'zip' package project and use Bluefin only in zip2tar? Or do I still need Bluefin at all, once getEntrySource is updated?
On Fri, May 01, 2026 at 09:56:29PM +0200, Henning Thielemann wrote:
On Fri, 1 May 2026, Tom Ellis wrote:
But getEntrySource was written with the wrong primitives, blocking prompt finalization, so I had to rewrite that.
Instead of writing and maintaining two new libraries for ZIP and TAR handling, would it work to submit improved getEntrySource based on Conduit to the 'zip' package project and use Bluefin only in zip2tar? Or do I still need Bluefin at all, once getEntrySource is updated?
I'll summarize what I know: * getEntrySource[1] is buggy because its return Conduit doesn't release the file handle it uses internally when consumption is finished. * The bug can be traced back to `sourceEntry`[2] * and then to `sourceIOHandle`[3] in combination with `isolate`[4]. The bug manifests in the Handle created by `sourceIOHandle` not being released when the result of `isolate` has been fully consumed. That's because `isolate` causes the Conduit returned from `sourceIOHandle` not to be fully consumed. This was demonstrated in my original post to this thread. * The definition of `sourceIOHandle` is sourceIOHandle alloc = bracketP alloc IO.hClose sourceHandle so it looks like it is upholding its responsibility to manage resources. `isolate` is about as innocuous as can be. `sourceEntry` uses `sourceIOHandle ... .| isolate ...` so it is also innocuous, modulo the correctness of its components. * Therefore I hypothesise that this is an unsolvable design flaw of Conduit. So my guess is you can't achieve what you want with conduit. Instead you could try to reuse some conduit pieces in the context of a safe setup, such as Bluefin. A simpler challenge for conduit is whether there can be a conduit that yields the first n lines of each file it is given as input. I've asked that on Discourse. Let's see if it can be done. https://discourse.haskell.org/t/conduit-to-yield-first-lines-from-file-witho... Tom [1] https://hackage-content.haskell.org/package/zip-2.2.1/docs/src/Codec.Archive... [2] https://hackage-content.haskell.org/package/zip-2.2.1/docs/src/Codec.Archive... [3] https://hackage-content.haskell.org/package/conduit-1.3.6.1/docs/src/Data.Co... [4] https://hackage.haskell.org/package/conduit-extra-1.3.6/docs/src/Data.Condui... [5] https://hackage-content.haskell.org/package/conduit-1.3.6.1/docs/Data-Condui...
On Sat, May 02, 2026 at 12:02:34PM +0100, Tom Ellis wrote:
So my guess is you can't achieve what you want with conduit. Instead you could try to reuse some conduit pieces in the context of a safe setup, such as Bluefin.
A simpler challenge for conduit is whether there can be a conduit that yields the first n lines of each file it is given as input. I've asked that on Discourse. Let's see if it can be done.
It looks like what you want can be done with conduit, at the cost of using a non-compositional (CPS) style: https://github.com/snoyberg/conduit/issues/532#issuecomment-4412941508 This would probably be sufficient for your needs in this case. Tom
On Sat, 9 May 2026, Tom Ellis wrote:
It looks like what you want can be done with conduit, at the cost of using a non-compositional (CPS) style:
https://github.com/snoyberg/conduit/issues/532#issuecomment-4412941508
This would probably be sufficient for your needs in this case.
Great! Of the four variants, I think I should not use "broken" and "sjshuck". Which one of "snoyberg" and "bracket" shall I try?
On Wed, May 13, 2026 at 10:25:47PM +0200, Henning Thielemann wrote:
On Sat, 9 May 2026, Tom Ellis wrote:
It looks like what you want can be done with conduit, at the cost of using a non-compositional (CPS) style:
https://github.com/snoyberg/conduit/issues/532#issuecomment-4412941508
This would probably be sufficient for your needs in this case.
Of the four variants, I think I should not use "broken" and "sjshuck". Which one of "snoyberg" and "bracket" shall I try?
You should use "bracket".
On Wed, 13 May 2026, Tom Ellis wrote:
On Wed, May 13, 2026 at 10:25:47PM +0200, Henning Thielemann wrote:
On Sat, 9 May 2026, Tom Ellis wrote:
It looks like what you want can be done with conduit, at the cost of using a non-compositional (CPS) style:
https://github.com/snoyberg/conduit/issues/532#issuecomment-4412941508
This would probably be sufficient for your needs in this case.
Of the four variants, I think I should not use "broken" and "sjshuck". Which one of "snoyberg" and "bracket" shall I try?
You should use "bracket".
Excellent, with the help of your code snippet I could prepare a patch for the "zip" package: https://github.com/mrkkrp/zip/pull/143 Thanks a lot!
participants (3)
-
Dan Dart -
Henning Thielemann -
Tom Ellis