There seems to be a difficult-to-justify interaction between lazy evaluation and monadic I/O: [[ -- file: SpikeIOMonadCloseHandle.hs -- Does hClose force completion of lazy I/O? import IO showFile fnam = do { fh <- openFile fnam ReadMode ; fc <- hGetContents fh ; hClose fh ; putStr fc } test = showFile "SpikeIOMonadCloseHandle.hs" ]] If I load this into Hugs and run it, the output is a single blank line. If I reverse the order of hClose and putStr, the source code is displayed. I think I can understand why this is happening, but it seems to me that there's a violation of referential transparency here: I can't see any reasonable justification for the value of 'fc' to vary depending on whether it's actually used before or after some other I/O operation. I suppose I was expecting the call of hClose to force complete evaluation of any value that depends on the state prior to hClose. I've no idea if there's a reasonable way to implement that. My concern is that this weakens the claim for monads that they provide a seamless integration between pure functional and stateful code; cf.: [[ We believe that, on the contrary, there are very significant differences between writing programs in C and writing in Haskell with monadic state transformers and IO: [...] - Usually, most of the program is neither stateful nor directly concerned with IO. The monadic approach allows the graceful coexistence of a small amount of imperative code and the large purely functional part of the program [...] - The usual coroutining behaviour of lazy evaluation, in which the consumer of a data structure coroutines with its producer, extends to stateful computation as well. As Hughes argues (Hughes 1989), the ability to separate what is computed from how much of it is computed is a powerful aid to writing modular programs ]] -- http://research.microsoft.com/Users/simonpj/Papers/state-lasc.ps.gz #g ------------------- Graham Klyne <GK@NineByNine.org> PGP: 0FAA 69FF C083 000B A2E9 A131 01B9 1C7A DBCA CB5E
Yes. This is because hGetContents (and hence readFile, etc.) use lazy IO. Just as in this case you might want hClose to force the file to be read, in a case like:
do h <- openFile "really_large_file" ReadMode c <- hGetContents h >>= return . head hClose h return c
you probably don't want the close to read the whole file. I'd argue that that problem is not with hClose, but with hGetContents. Really, a strict version should be used in most situations. Something like:
hGetContentsStrict h = do b <- hIsEOF h if b then return [] else do c <- hGetChar h r <- hGetContentsStrict h return (c:r)
of course, you could be smarter with buffering, etc. Another way would be to do something using seq/deepSeq. - Hal -- Hal Daume III | hdaume@isi.edu "Arrest this man, he talks in maths." | www.isi.edu/~hdaume On Tue, 20 May 2003, Graham Klyne wrote:
There seems to be a difficult-to-justify interaction between lazy evaluation and monadic I/O:
[[ -- file: SpikeIOMonadCloseHandle.hs -- Does hClose force completion of lazy I/O?
import IO
showFile fnam = do { fh <- openFile fnam ReadMode ; fc <- hGetContents fh ; hClose fh ; putStr fc }
test = showFile "SpikeIOMonadCloseHandle.hs" ]]
If I load this into Hugs and run it, the output is a single blank line.
If I reverse the order of hClose and putStr, the source code is displayed.
I think I can understand why this is happening, but it seems to me that there's a violation of referential transparency here: I can't see any reasonable justification for the value of 'fc' to vary depending on whether it's actually used before or after some other I/O operation.
I suppose I was expecting the call of hClose to force complete evaluation of any value that depends on the state prior to hClose. I've no idea if there's a reasonable way to implement that.
My concern is that this weakens the claim for monads that they provide a seamless integration between pure functional and stateful code; cf.: [[ We believe that, on the contrary, there are very significant differences between writing programs in C and writing in Haskell with monadic state transformers and IO: [...] - Usually, most of the program is neither stateful nor directly concerned with IO. The monadic approach allows the graceful coexistence of a small amount of imperative code and the large purely functional part of the program [...] - The usual coroutining behaviour of lazy evaluation, in which the consumer of a data structure coroutines with its producer, extends to stateful computation as well. As Hughes argues (Hughes 1989), the ability to separate what is computed from how much of it is computed is a powerful aid to writing modular programs ]] -- http://research.microsoft.com/Users/simonpj/Papers/state-lasc.ps.gz
#g
------------------- Graham Klyne <GK@NineByNine.org> PGP: 0FAA 69FF C083 000B A2E9 A131 01B9 1C7A DBCA CB5E
_______________________________________________ Haskell mailing list Haskell@haskell.org http://www.haskell.org/mailman/listinfo/haskell
Hal, I agree with the surface points you make. It's easy enough to fix the problem once you realize it's there. (In my own "real" program, I moved the hClose, which meant I had to pass the handle out of the function which opened it.) The underlying thrust of my post was that I thought pure functional languages, like Haskell, were supposed to help one avoid such traps by ensuring the messy dependencies on ordering didn't arise in the first place. As things stand, I don't think I could begin articulate a reliable set of rules for avoiding such problems, short of something like "use only strict functions in monad-chains". I'm hoping the Haskell community has some experience with this kind of issue to offer some more helpful advice, or even tools to detect unsafe combinations. Maybe a discussion of safe programming patterns would be a useful interim step? (e.g. Ketil Z. Malde's suggestion of renaming the function to hUnsafeGetContents maybe a small step in the right direction?) ... Thinking some more... I'm reminded of some discussions I had a few years ago about the timing of calls to Java finalizers, and problems this could cause for network I/O programs because using finalizers to close network sockets would lead to unexpected resource problems. The only reliable solution was to always close the sockets explicitly when done. With Java, coming from C/C++, it was possible to get into a mindset that automatic memory management also meant automatic management of all resources, including all those that weren't directly visible to the programmer. Maybe there's a similar trap for the unware in Haskell? Anyway, my thoughts are leading me to the idea that the problem is a disconnect (lack of formal connection or interlock) between the actions of opening a file, reading its contents and closing it. For example, one could imagine a structure: hSafeGetContents :: Handle -> (String -> a) -> a hSafeGetContents handle function = function $ hUnsafeGetContents handle Now the result string can be as lazy as you like, but I think one can guarantee that the handle won't be closed until the function has used as much of the content as it may need. #g -- At 13:27 20/05/03 -0700, Hal Daume III wrote:
Yes. This is because hGetContents (and hence readFile, etc.) use lazy IO. Just as in this case you might want hClose to force the file to be read, in a case like:
do h <- openFile "really_large_file" ReadMode c <- hGetContents h >>= return . head hClose h return c
you probably don't want the close to read the whole file. I'd argue that that problem is not with hClose, but with hGetContents. Really, a strict version should be used in most situations. Something like:
hGetContentsStrict h = do b <- hIsEOF h if b then return [] else do c <- hGetChar h r <- hGetContentsStrict h return (c:r)
of course, you could be smarter with buffering, etc. Another way would be to do something using seq/deepSeq.
- Hal
-- Hal Daume III | hdaume@isi.edu "Arrest this man, he talks in maths." | www.isi.edu/~hdaume
On Tue, 20 May 2003, Graham Klyne wrote:
There seems to be a difficult-to-justify interaction between lazy evaluation and monadic I/O:
[[ -- file: SpikeIOMonadCloseHandle.hs -- Does hClose force completion of lazy I/O?
import IO
showFile fnam = do { fh <- openFile fnam ReadMode ; fc <- hGetContents fh ; hClose fh ; putStr fc }
test = showFile "SpikeIOMonadCloseHandle.hs" ]]
If I load this into Hugs and run it, the output is a single blank line.
If I reverse the order of hClose and putStr, the source code is displayed.
I think I can understand why this is happening, but it seems to me that there's a violation of referential transparency here: I can't see any reasonable justification for the value of 'fc' to vary depending on whether it's actually used before or after some other I/O operation.
I suppose I was expecting the call of hClose to force complete evaluation of any value that depends on the state prior to hClose. I've no idea if there's a reasonable way to implement that.
My concern is that this weakens the claim for monads that they provide a seamless integration between pure functional and stateful code; cf.: [[ We believe that, on the contrary, there are very significant differences between writing programs in C and writing in Haskell with monadic state transformers and IO: [...] - Usually, most of the program is neither stateful nor directly concerned with IO. The monadic approach allows the graceful coexistence of a small amount of imperative code and the large purely functional part of the program [...] - The usual coroutining behaviour of lazy evaluation, in which the consumer of a data structure coroutines with its producer, extends to stateful computation as well. As Hughes argues (Hughes 1989), the ability to separate what is computed from how much of it is computed is a powerful aid to writing modular programs ]] -- http://research.microsoft.com/Users/simonpj/Papers/state-lasc.ps.gz
#g
------------------- Graham Klyne <GK@NineByNine.org> PGP: 0FAA 69FF C083 000B A2E9 A131 01B9 1C7A DBCA CB5E
_______________________________________________ Haskell mailing list Haskell@haskell.org http://www.haskell.org/mailman/listinfo/haskell
------------------- Graham Klyne <GK@NineByNine.org> PGP: 0FAA 69FF C083 000B A2E9 A131 01B9 1C7A DBCA CB5E
Hi Graham,
strict functions in monad-chains". I'm hoping the Haskell community has some experience with this kind of issue to offer some more helpful advice, or even tools to detect unsafe combinations. Maybe a discussion of safe programming patterns would be a useful interim step?
I don't really know what sort of advice is really helpful, but I can share a few observations: When choosing between openFile/?/hClose and readFile, use the openFile/?/hClose combination only if you expect to open a lot of files. Rationale: readFile supposedly always closes the handle when you're done but sometimes really just puts it in a semi-closed state. This means that if you're reading a lot of files, you're going to run out of handles. If you need to use openFile/?/hClose because of file handle issues, read the file (or what parts of it you need) strictly. I actually have a function in my general library: readFileCloseBy :: DeepSeq a => FilePath -> (String -> a) -> IO a which opens the file, parses it using the supplied function, deepSeqs it to make it strict and then closes the handle. By supplying id as the function, you get a version of readFile which is strict and always closes the Handle.
Thinking some more... I'm reminded of some discussions I had a few years ago about the timing of calls to Java finalizers, and problems this could cause for network I/O programs because using finalizers to close network sockets would lead to unexpected resource problems. The only reliable
This sounds very similar to the semi-closed handle issue in readFile.
Anyway, my thoughts are leading me to the idea that the problem is a disconnect (lack of formal connection or interlock) between the actions of opening a file, reading its contents and closing it. For example, one could imagine a structure:
hSafeGetContents :: Handle -> (String -> a) -> a hSafeGetContents handle function = function $ hUnsafeGetContents handle
Now the result string can be as lazy as you like, but I think one can guarantee that the handle won't be closed until the function has used as much of the content as it may need.
Alas, this is not true :). Let function=id and you'll see the problem. You need to put a seq or a deepSeq in there somewhere, otherwise just applying the function won't cause any of the file to be read. - Hal
At 13:27 20/05/03 -0700, Hal Daume III wrote:
Yes. This is because hGetContents (and hence readFile, etc.) use lazy IO. Just as in this case you might want hClose to force the file to be read, in a case like:
do h <- openFile "really_large_file" ReadMode c <- hGetContents h >>= return . head hClose h return c
you probably don't want the close to read the whole file. I'd argue that that problem is not with hClose, but with hGetContents. Really, a strict version should be used in most situations. Something like:
hGetContentsStrict h = do b <- hIsEOF h if b then return [] else do c <- hGetChar h r <- hGetContentsStrict h return (c:r)
of course, you could be smarter with buffering, etc. Another way would be to do something using seq/deepSeq.
- Hal
-- Hal Daume III | hdaume@isi.edu "Arrest this man, he talks in maths." | www.isi.edu/~hdaume
On Tue, 20 May 2003, Graham Klyne wrote:
There seems to be a difficult-to-justify interaction between lazy evaluation and monadic I/O:
[[ -- file: SpikeIOMonadCloseHandle.hs -- Does hClose force completion of lazy I/O?
import IO
showFile fnam = do { fh <- openFile fnam ReadMode ; fc <- hGetContents fh ; hClose fh ; putStr fc }
test = showFile "SpikeIOMonadCloseHandle.hs" ]]
If I load this into Hugs and run it, the output is a single blank line.
If I reverse the order of hClose and putStr, the source code is displayed.
I think I can understand why this is happening, but it seems to me that there's a violation of referential transparency here: I can't see any reasonable justification for the value of 'fc' to vary depending on whether it's actually used before or after some other I/O operation.
I suppose I was expecting the call of hClose to force complete evaluation of any value that depends on the state prior to hClose. I've no idea if there's a reasonable way to implement that.
My concern is that this weakens the claim for monads that they provide a seamless integration between pure functional and stateful code; cf.: [[ We believe that, on the contrary, there are very significant differences between writing programs in C and writing in Haskell with monadic state transformers and IO: [...] - Usually, most of the program is neither stateful nor directly concerned with IO. The monadic approach allows the graceful coexistence of a small amount of imperative code and the large purely functional part of the program [...] - The usual coroutining behaviour of lazy evaluation, in which the consumer of a data structure coroutines with its producer, extends to stateful computation as well. As Hughes argues (Hughes 1989), the ability to separate what is computed from how much of it is computed is a powerful aid to writing modular programs ]] -- http://research.microsoft.com/Users/simonpj/Papers/state-lasc.ps.gz
#g
------------------- Graham Klyne <GK@NineByNine.org> PGP: 0FAA 69FF C083 000B A2E9 A131 01B9 1C7A DBCA CB5E
_______________________________________________ Haskell mailing list Haskell@haskell.org http://www.haskell.org/mailman/listinfo/haskell
------------------- Graham Klyne <GK@NineByNine.org> PGP: 0FAA 69FF C083 000B A2E9 A131 01B9 1C7A DBCA CB5E
Hal, Thanks for your comments. (I had overlooked "readFile", which certainly looks safer. I'm not yet fully used to the distribution of functions between standard libraries and the Prelude. As for my 'hSafeGetContents', I realized exactly your point about using 'id' too late, after I'd sent my email. :-( ) The broader debate I was trying to evoke was: how, as a programmer using a combination of monads and non-strict functions, can I be confident that there aren't hidden interactions waiting to bite? Using the hOpen ... hClose vs readFile case as an example, is the difficulty here to do with the interleaving of multiple statements that modify some state with non-strict functions that return results based on some particular instance of that state? Is this really a fundamental mismatch? I read recently that other functional languages (ML?) with strict evaluation semantics can be used to mimic lazy evaluation, but saw no clue how that would work. I'm wondering if any insight might be gained by looking at how such mechanisms might interact with functional state-transformer mechanisms? #g -- [I'm aware this discussion thread is starting to run on a bit... is it on-topic for this list?] ------------------- Graham Klyne <GK@NineByNine.org> PGP: 0FAA 69FF C083 000B A2E9 A131 01B9 1C7A DBCA CB5E
On Thursday 22 May 2003 10:22 am, Graham Klyne wrote:
I'm not yet fully used to the distribution of functions between standard libraries and the Prelude.
The policy for exporting a function from the Prelude is, roughly speaking, Is this function useful in small, simple examples? The idea being that teachers, students and people writing quick scripts don't want to get bogged down in the module system on day 1. Unfortunately, the definition of 'small' and 'simple' is somewhat subjective and we also carry some historical baggage (i.e., we don't want to remove an export if it still exists even if it is not very useful) so it's a bit hard to see the pattern of where things come from. If you're writing a non-trivial program, you might be best importing things directly from the standard libraries even if it is exported by the Prelude. Alternatively, you might consider importing things from the hierarchical libraries. The advantage of the hierarchical libraries is that their interfaces are not frozen by the Haskell-98 definition but that is, of course, also their chief disadvantage. -- Alastair Reid
Thanks for your explanations. At 12:10 22/05/03 +0100, Alastair Reid wrote:
If you're writing a non-trivial program, you might be best importing things directly from the standard libraries even if it is exported by the Prelude.
Yes, I do that. My problem was that I was looking at the IO library documentation [1] and overlooked the prelude functions. I think it's just a learning-curve problem. [1] http://www.haskell.org/onlinereport/io.html
Alternatively, you might consider importing things from the hierarchical libraries. The advantage of the hierarchical libraries is that their interfaces are not frozen by the Haskell-98 definition but that is, of course, also their chief disadvantage.
I've come to see the Haskell 98 definition as a useful baseline rather than an endpoint. Since I started using the state transformer monad I've migrated my code to use the hierarchical libraries (under Hugs, so far), which was a mostly painless process. By way of feedback, a couple of oddities I noticed were: (a) I couldn't pick up other libraries from the hierarchical tree (Parsec, HUnit), and still have to name their directories explicitly in the Hugs registry entry. (b) I had to explicitly import a couple of Data.Char values that were previously picked up automatically (isSpace somes to mind). #g ------------------- Graham Klyne <GK@NineByNine.org> PGP: 0FAA 69FF C083 000B A2E9 A131 01B9 1C7A DBCA CB5E
On Thursday, 2003-05-22, 13:27, CEST Graham Klyne wrote:
[...]
and still have to name their directories explicitly in the Hugs registry entry.
Could it be that you can avoid this by using the +N command line option?
(b) I had to explicitly import a couple of Data.Char values that were previously picked up automatically (isSpace somes to mind).
This may have something to do with the fact that isSpace and others are included in the Hugs prelude although they are not part of the standard prelude.
[...]
Graham Klyne
Wolfgang Jeltsch
At 16:27 23/05/03 +0200, Wolfgang Jeltsch wrote:
On Thursday, 2003-05-22, 13:27, CEST Graham Klyne wrote:
[...]
and still have to name their directories explicitly in the Hugs registry entry.
Could it be that you can avoid this by using the +N command line option?
The problem was slightly more subtle: the library files were found OK, but error messages were generated (I forget the exact message) which I've previously seen when the module name does not match the filename.
(b) I had to explicitly import a couple of Data.Char values that were previously picked up automatically (isSpace somes to mind).
This may have something to do with the fact that isSpace and others are included in the Hugs prelude although they are not part of the standard prelude.
That would seem consistent with what I observed. #g ------------------- Graham Klyne <GK@NineByNine.org> PGP: 0FAA 69FF C083 000B A2E9 A131 01B9 1C7A DBCA CB5E
Graham Klyne wrote:
Thanks for your comments. (I had overlooked "readFile", which certainly looks safer.
It isn't: readFile :: FilePath -> IO String readFile name = openFile name ReadMode >>= hGetContents
The broader debate I was trying to evoke was: how, as a programmer using a combination of monads and non-strict functions, can I be confident that there aren't hidden interactions waiting to bite?
The problem isn't monads, or lazy functions, or even the interactions between them. The problem is specifically lazy I/O, which is basically a fudge which usually works in simple cases. The "normal" (i.e. strict) I/O functions are perfectly safe.
Using the hOpen ... hClose vs readFile case as an example, is the difficulty here to do with the interleaving of multiple statements that modify some state with non-strict functions that return results based on some particular instance of that state? Is this really a fundamental mismatch?
The problems arise from the use of unsafeInterleaveIO (with which hGetContents is implemented), which essentially allows I/O operations to "escape" from the IO monad. The net result is that the actual I/O operations (for which ordering matters) occur as a side-effect of evaluating pure expressions. The simple solution is not to use unsafeInterleaveIO, either directly or via hGetContents, getContents, readFile or interact. -- Glynn Clements <glynn.clements@virgin.net>
Glynn Clements <glynn.clements@virgin.net> writes:
Graham Klyne wrote:
Thanks for your comments. (I had overlooked "readFile", which certainly looks safer.)
It isn't:
readFile :: FilePath -> IO String readFile name = openFile name ReadMode >>= hGetContents
Is too! IMHO, the difference is that the program doesn't get the handle to mess with, so there's no chance of doing something unexpected (like closing it before all required input is read). You still risk somebody messing with the file behind your back, but I don't see how strictness changes that. -kzm -- If I haven't seen further, it is by standing in the footprints of giants
Ketil Z. Malde wrote:
Thanks for your comments. (I had overlooked "readFile", which certainly looks safer.)
It isn't:
readFile :: FilePath -> IO String readFile name = openFile name ReadMode >>= hGetContents
Is too!
IMHO, the difference is that the program doesn't get the handle to mess with, so there's no chance of doing something unexpected (like closing it before all required input is read).
OK; it does solve the "premature close" issue, which is what started this thread. It also ensures that the stream isn't a socket, which eliminates another class of problems. I suppose that there could theoretically be issues if readFile was used on a device or FIFO (named pipe). If you know that you are accessing such, you probably know about any related issues, but what if the filename is taken from the command line, and the user specifies a device or FIFO?
You still risk somebody messing with the file behind your back, but I don't see how strictness changes that.
Strictness won't solve the problem where a completely unrelated process modifies the file; you would need to use locking for that (although locking would itself require the use of strict I/O). However, with lazy I/O, there could be issues if the modification is triggered in some way by the reading process. -- Glynn Clements <glynn.clements@virgin.net>
On 22-May-2003 Glynn Clements wrote:
The problem isn't monads, or lazy functions, or even the interactions between them. The problem is specifically lazy I/O, which is basically a fudge which usually works in simple cases.
The "normal" (i.e. strict) I/O functions are perfectly safe.
Since Haskell is lazy by default, I consider lazy file reading to be the more natural choice, and also a safe one on operating systems like unix who lock a copy of the file at the time it's opened, and given that such a copy almost always does not come at such a high cost. Unfortunately, the number of file hand les is sometimes limited so that one is sometimes forced to close lazily read files explizitly which is rather tedious... Elke Kasimir. -- Elke Kasimir EsPresto AG ----------------------------------------------------------------- - Software Development- Breite Str. 30-31 Tel/Fax: +49-30-90 226-750/-760 10178 Berlin/Germany
On Fri, 23 May 2003 14:03:47 +0200 (CEST) Elke Kasimir <elke@espresto.com> wrote:
On 22-May-2003 Glynn Clements wrote:
The problem isn't monads, or lazy functions, or even the interactions between them. The problem is specifically lazy I/O, which is basically a fudge which usually works in simple cases.
The "normal" (i.e. strict) I/O functions are perfectly safe.
Since Haskell is lazy by default, I consider lazy file reading to be the more natural choice, and also a safe one on operating systems like unix who lock a copy of the file at the time it's opened, and given that such a copy almost always does not come at such a high cost.
This is -a- problem with lazy IO but not -the- problem. This problem applies (or not) to strict IO, albeit at least with strict IO you'll definitely have what you have. The problem is that-within the language- you can apply strict IO functions (in this case, hClose) to a file that is being read lazily. This is also the reason why Ketil Z. Malde said that readFile -is- safe, as without access to the handle you are limited to what you can do to a file.
Unfortunately, the number of file handles is sometimes limited so that one is sometimes forced to close lazily read files explizitly which is rather tedious...
Combining explicit file closing to a lazily read file is dangerous, that is the point of this whole thread. Either use strict IO throughout or solve this problem with strictness annotations (deepSeq/rnf may help here). (There is one example where I solved this problem with a single well-placed $!) If you do explicitly close a lazily read file then you better mean what you are saying.
Elke Kasimir wrote:
The problem isn't monads, or lazy functions, or even the interactions between them. The problem is specifically lazy I/O, which is basically a fudge which usually works in simple cases.
The "normal" (i.e. strict) I/O functions are perfectly safe.
Since Haskell is lazy by default, I consider lazy file reading to be the more natural choice, and also a safe one on operating systems like unix who lock a copy of the file at the time it's opened,
I'm not quite sure what you're saying here, but I think that you're wrong. It appears that you are confusing Unix (which doesn't lock files automatically) with Windows (which automatically locks a file unless the program indicates that it allows concurrent access). However, that isn't particularly relevant to the issue of lazy vs strict I/O. Even if you use strict I/O, that doesn't eliminate the problems associated with reading from a file which is being modified concurrently. To handle that, you would need to use explicit locking operations; even then, unless you use mandatory locking (which not all Unices support), you are relying upon other programs honouring the locks. -- Glynn Clements <glynn.clements@virgin.net>
On 23-May-2003 Glynn Clements wrote:
Elke Kasimir wrote:
Since Haskell is lazy by default, I consider lazy file reading to be the more natural choice, and also a safe one on operating systems like unix who lock a copy of the file at the time it's opened,
I'm not quite sure what you're saying here, but I think that you're wrong. It appears that you are confusing Unix (which doesn't lock files automatically) with Windows (which automatically locks a file unless the program indicates that it allows concurrent access).
You're right - My use of the term "locked" is inappropriate and obviously confusing. What I was trying to indicates is that if some file "Foo" is opened using >> readFile "Foo" << the effect is that you get an object that can't be changed from the outside, so that it is not so important to ensure that the read operation on that file is finished at some specific point in time, and even may be never known to have been finished for sure. If it was possible to open arbitrary many files that way - and opening "the same file" arbitrary many times for reading, "not so important" could be changed into "not important" in the previous statement. Best, Elke. -- Elke Kasimir EsPresto AG ----------------------------------------------------------------- - Software Development- Breite Str. 30-31 Tel/Fax: +49-30-90 226-750/-760 10178 Berlin/Germany
Elke Kasimir wrote:
Since Haskell is lazy by default, I consider lazy file reading to be the more natural choice, and also a safe one on operating systems like unix who lock a copy of the file at the time it's opened,
I'm not quite sure what you're saying here, but I think that you're wrong. It appears that you are confusing Unix (which doesn't lock files automatically) with Windows (which automatically locks a file unless the program indicates that it allows concurrent access).
You're right - My use of the term "locked" is inappropriate and obviously confusing. What I was trying to indicates is that if some file "Foo" is opened using >> readFile "Foo" << the effect is that you get an object that can't be changed from the outside,
While readFile doesn't "leak" a handle to the rest of the program, nothing prevents the program from doing: openFile "Foo" WriteMode and nothing prevents other processes from writing to that file. In either of those situations, the exact data which readFile returns depends upon when readFile actually performs the reads.
so that it is not so important to ensure that the read operation on that file is finished at some specific point in time, and even may be never known to have been finished for sure. If it was possible to open arbitrary many files that way - and opening "the same file" arbitrary many times for reading, "not so important" could be changed into "not important" in the previous statement.
Exhausting file handles isn't the only problem which occurs if the consumer doesn't close the file. E.g.: createTheFile "Foo" string <- readFile "Foo" consumeString string removeFile "Foo" IIRC, Windows refuses to delete open files, so removeFile will fail if the file is still open (i.e. if consumeString doesn't read its entire input). -- Glynn Clements <glynn.clements@virgin.net>
Glynn Clements (Thu, May 22, 2003 at 08:11:02PM +0100):
Graham Klyne wrote:
Thanks for your comments. (I had overlooked "readFile", which certainly looks safer.
It isn't:
readFile :: FilePath -> IO String readFile name = openFile name ReadMode >>= hGetContents
Why does not hGetContents set a flag associated to the handle. The flag is turned off if no lazy io thru the handle can happen. hClose checks the flag and eventually passes the close to the OS if no flag is set anymore. Sincerly, -- Stefan Karrmann I like work; it fascinates me; I can sit and look at it for hours.
Graham Klyne wrote:
I agree with the surface points you make. It's easy enough to fix the problem once you realize it's there. (In my own "real" program, I moved the hClose, which meant I had to pass the handle out of the function which opened it.)
The underlying thrust of my post was that I thought pure functional languages, like Haskell, were supposed to help one avoid such traps by ensuring the messy dependencies on ordering didn't arise in the first place.
Unfortunately, unless you also have a pure functional operating system, the ordering of I/O operations matters ;) As things stand, I don't think I could begin articulate a reliable
set of rules for avoiding such problems, short of something like "use only strict functions in monad-chains". I'm hoping the Haskell community has some experience with this kind of issue to offer some more helpful advice, or even tools to detect unsafe combinations. Maybe a discussion of safe programming patterns would be a useful interim step?
(e.g. Ketil Z. Malde's suggestion of renaming the function to hUnsafeGetContents maybe a small step in the right direction?)
Well, it is implemented using unsafeInterleaveIO; I'm not sure why lazy I/O generally is considered unsafe but the specific cases of readFile, hGetContents etc aren't.
Thinking some more... I'm reminded of some discussions I had a few years ago about the timing of calls to Java finalizers, and problems this could cause for network I/O programs because using finalizers to close network sockets would lead to unexpected resource problems. The only reliable solution was to always close the sockets explicitly when done. With Java, coming from C/C++, it was possible to get into a mindset that automatic memory management also meant automatic management of all resources, including all those that weren't directly visible to the programmer. Maybe there's a similar trap for the unware in Haskell?
More generally, the concept of "visible" semantics depends upon your (highly subjective) definition of "visible". I'm reminded of a recent BugTraq post regarding wiping sensitive information from memory; the code was basically: char password[...]; read_password(password); do_something(password); memset(password, 0, sizeof(password)); return; The compiler inlined the memset(), then noted that the contents of the password array weren't used after the overwrite, so it optimised the overwrite away. -- Glynn Clements <glynn.clements@virgin.net>
At 19:00 21/05/03 +0100, Glynn Clements wrote:
More generally, the concept of "visible" semantics depends upon your (highly subjective) definition of "visible". I'm reminded of a recent BugTraq post regarding wiping sensitive information from memory; the code was basically:
char password[...]; read_password(password); do_something(password); memset(password, 0, sizeof(password)); return;
The compiler inlined the memset(), then noted that the contents of the password array weren't used after the overwrite, so it optimised the overwrite away.
Security is always a tough case, since it has to be effective against activities that are outside the rules by which the legitimate players are operating. (e.g. demonstrations of weaknesses in smart cards by subjecting them to physical environments in which they're never intended to function normally.) So my definition of "visible", here, is restricted to what is visible through the language (as in my original example). Your other point:
Unfortunately, unless you also have a pure functional operating system, the ordering of I/O operations matters ;)
is well made, though I think there would be no problem in my case but for non-strict evaluation of the I/O result. But, looking beyond I/O, I can't help wondering if these considerations extend to other situations in which one might use monads. e.g. under what circumstances are monads safe to use in a multiprocessor environment? #g ------------------- Graham Klyne <GK@NineByNine.org> PGP: 0FAA 69FF C083 000B A2E9 A131 01B9 1C7A DBCA CB5E
Graham Klyne <gk@ninebynine.org> writes:
showFile fnam = do { fh <- openFile fnam ReadMode ; fc <- hGetContents fh ; hClose fh ; putStr fc }
If I load this into Hugs and run it, the output is a single blank line. If I reverse the order of hClose and putStr, the source code is displayed.
I think I can understand why this is happening, but it seems to me that there's a violation of referential transparency here: I can't see any reasonable justification for the value of 'fc' to vary depending on whether it's actually used before or after some other I/O operation.
hClose is a strict operation, hGetContents is a lazy one. You want hClose to be strict, since the next action may be writing to the same file; having an unevaluated hGetContents around to be evaluated after other operations may not give you the result you expect. For a simple example like the above, use readFile, which reads lazily and closes the file when it's done. For complex examples, use handles with strict IO operations (but be prepared for high memory cost for those lists of chars) Perhaps it should be hUnsafeGetContents :-) ? -kzm -- If I haven't seen further, it is by standing in the footprints of giants
participants (10)
-
Alastair Reid -
Derek Elkins -
Elke Kasimir -
Glynn Clements -
Graham Klyne -
Graham Klyne
-
Hal Daume III -
ketil＠ii.uib.no -
Stefan Karrmann -
Wolfgang Jeltsch