I'm working with data spanning a lot of separate files and am having difficulty getting ghc to close the files. I've reduced my code to the following bit of code which should show my problem:
go filelist = do return (mapM go' filelist) where go' f = do h <- openFile f ReadMode text <- hGetContents h let c = head $! text return (c `seq` (hClose h `seq` c))
basically, you give "go" a list of files and it'll give you the first character from each of these. if i run this with 30-40 files, it's okay. but when i run it with a list of 400 files, i get: *** Exception: resource exhausted Action: openFile Reason: process file table full File: /nfs/isd/marcu/RST-CORPUS-LDC/RSTtrees-WSJ-main-1.0/DATA/wsj_1355.out.dis which doesn't make sense to me, since it should be (as far as i can tell) closing each of the handles (the yucky "c `seq` (hClose h `seq` c)" is my way of making sure that it reads the first character before it closes the file and then makes sure to close the file before returning the character -- if there is a better way, please let me know). if anyone can tell me what my problem is, that would be great! also, along the same lines, are there any good papers on using haskell with large amounts of data? i frequently have problems like this and if there were a unified theory behind dealing with them, it would be great. any pointers would be helpful. thanks! -- Hal Daume III "Computer science is no more about computers | hdaume@isi.edu than astronomy is about telescopes." -Dijkstra | www.isi.edu/~hdaume
I'm sorry, I had a small error in my code. It should have been:
go filelist = mapM go' filelist -- no "do return" where go' f = do h <- openFile f ReadMode text <- hGetContents h let c = head $! text return (c `seq` (hClose h `seq` c))
- Hal Hal Daume wrote:
I'm working with data spanning a lot of separate files and am having difficulty getting ghc to close the files. I've reduced my code to the following bit of code which should show my problem:
go filelist = do return (mapM go' filelist) where go' f = do h <- openFile f ReadMode text <- hGetContents h let c = head $! text return (c `seq` (hClose h `seq` c))
basically, you give "go" a list of files and it'll give you the first character from each of these.
if i run this with 30-40 files, it's okay. but when i run it with a list of 400 files, i get:
*** Exception: resource exhausted Action: openFile Reason: process file table full File: /nfs/isd/marcu/RST-CORPUS-LDC/RSTtrees-WSJ-main-1.0/DATA/wsj_1355.out.dis
which doesn't make sense to me, since it should be (as far as i can tell) closing each of the handles (the yucky "c `seq` (hClose h `seq` c)" is my way of making sure that it reads the first character before it closes the file and then makes sure to close the file before returning the character -- if there is a better way, please let me know).
if anyone can tell me what my problem is, that would be great!
also, along the same lines, are there any good papers on using haskell with large amounts of data? i frequently have problems like this and if there were a unified theory behind dealing with them, it would be great. any pointers would be helpful.
thanks!
-- Hal Daume III
"Computer science is no more about computers | hdaume@isi.edu than astronomy is about telescopes." -Dijkstra | www.isi.edu/~hdaume
_______________________________________________ Haskell mailing list Haskell@haskell.org http://www.haskell.org/mailman/listinfo/haskell
-- Hal Daume III "Computer science is no more about computers | hdaume@isi.edu than astronomy is about telescopes." -Dijkstra | www.isi.edu/~hdaume
Hal Daume writes: | I'm sorry, I had a small error in my code. It should have been: | | >go filelist = mapM go' filelist -- no "do return" | > where go' f = do h <- openFile f ReadMode | > text <- hGetContents h | > let c = head $! text | > return (c `seq` (hClose h `seq` c)) Hi. I don't have a handy installation of ghc to check this on, but here's a speculative attempt. go' f = do h <- openFile f ReadMode text <- hGetContents h case text of (c:_) -> do hClose h return c This way, the pattern match on `text' happens before the innermost `do', which in turn closes the handle before letting the rest of the computation continue. You may be able to get a similar effect with one more strictness operator ... return $! (c `seq` hClose h `seq` c) but that's not quite as readable. BTW I'm not sure that the strictness operator in `head $! text' makes any difference. It ensures that when the expression's evaluated, `text' is reduced to WHNF before `head' is applied to it. But then the very next thing which happens is a pattern match (head (x:_) = x) which would force that reduction anyway. Hope that (isn't mistake-ridden and) helps. - Tom
Tom Pledger wrote:
[...] go' f = do h <- openFile f ReadMode text <- hGetContents h case text of (c:_) -> do hClose h return c
This way, the pattern match on `text' happens before the innermost `do', which in turn closes the handle before letting the rest of the computation continue.
I think the above code will lead to problems if file input is done when c is evaluated and not earlier. Pattern-matching against (c:_) will not evaluate c. So the hClose will AFAICS be executed before c is evaluated. If reading is caused by an evaluation of c taking place later the file will be already closed.
[...]
Hal Daume wrote:
[...] return (c `seq` (hClose h `seq` c))
I don't think that this causes the hClose h to be executed because evaluating an IO expression doesn't mean to execute it. Similarily an return (hClose h) wouldn't close any file AFAICS.
[...]
Wolfgang
participants (3)
-
Hal Daume -
Tom Pledger -
Wolfgang Jeltsch