
The function below always returns "", rather than the file's contents. _Real World Haskell_ touches on how laziness can make this problimatic, without really giving a solution, other than throwing in a putStr to force evaluation, which can't be done here. How can I make hGetContents strict, to ensure the file contents are really read before it gets closed? readFile' file = do f <- openFile file ReadMode -- locking will go here s <- hGetContents f hClose f return s Also, I noticed that opening a file and locking it involves a very verbose seeming dance. (It's 2 lines of code in most other languages.) Does this indicate that few people bother with file locking in Haskell and so it still has these rough edges, or that there's a better way to do it that I have not found yet? openLocked file = do handle <- openFile file ReadMode lockfd <- handleToFd handle -- closes handle waitToSetLock lockfd (ReadLock, AbsoluteSeek, 0, 0) handle' <- fdToHandle lockfd return handle' -- see shy jo

On 10/9/10 5:49 PM, Joey Hess wrote:
The function below always returns "", rather than the file's contents. _Real World Haskell_ touches on how laziness can make this problimatic, without really giving a solution, other than throwing in a putStr to force evaluation, which can't be done here. How can I make hGetContents strict, to ensure the file contents are really read before it gets closed?
readFile' file = do f<- openFile file ReadMode -- locking will go here s<- hGetContents f hClose f return s
Haskell won't actually read the file unless you need the contents of the file. In order to ensure that the file contents are read before it gets close, you need to actually perform an operation that will force evaluation of the file. Haskell will read the file then to perform whatever operation you specify. for example sendFile file = do f<- openFile file ReadMode s <- hGetContents f sendToNetwork s --made up function hClose f In your example, you're getting "" because the handle is closed, so Haskell doesn't have anything to read. You say you can't use putStr in this scenario, but if you're reading the file you must be doing something with the contents. Wait to close the handle until after that operation takes place.
Also, I noticed that opening a file and locking it involves a very verbose seeming dance. (It's 2 lines of code in most other languages.) Does this indicate that few people bother with file locking in Haskell and so it still has these rough edges, or that there's a better way to do it that I have not found yet?
openLocked file = do handle<- openFile file ReadMode lockfd<- handleToFd handle -- closes handle waitToSetLock lockfd (ReadLock, AbsoluteSeek, 0, 0) handle'<- fdToHandle lockfd return handle'
You should go to haskell.org/hoogle, and search openFile, or check out the Haskell98 report, but I'm under the impression that ghc locks the file for you when you open the file.: From haskell98 report: 21.2.3 File locking Implementations should enforce as far as possible, at least locally to the Haskell process, multiple-reader single-writer locking on files. That is, /there may either be many handles on the same file which manage input, or just one handle on the file which manages output/. If any open or semi-closed handle is managing a file for output, no new handle can be allocated for that file. If any open or semi-closed handle is managing a file for input, new handles can only be allocated if they do not manage output. Whether two files are the same is implementation-dependent, but they should normally be the same if they have the same absolute path name and neither has been renamed, for example. 21.3.1 Opening Files ... /Error reporting/: the openFile computation may fail with isAlreadyInUseError if the file is already open and cannot be reopened; isDoesNotExistError if the file does not exist; orisPermissionError if the user does not have permission to open the file Hope this helps, Jimmy Wylie

On 10/09/10 18:49, Joey Hess wrote:
The function below always returns "", rather than the file's contents. _Real World Haskell_ touches on how laziness can make this problimatic, without really giving a solution, other than throwing in a putStr to force evaluation, which can't be done here. How can I make hGetContents strict, to ensure the file contents are really read before it gets closed?
There are various hacks to do it (including ones with no external side-effects that just use "case" / "seq" / etc. quite thoroughly)... But realistically you should use a library that doesn't add laziness, if laziness is not what you want. For example, "strict" on Hackage http://hackage.haskell.org/package/strict http://hackage.haskell.org/packages/archive/strict/0.3.2/doc/html/System-IO-... -Isaac

Joey Hess
The function below always returns "", rather than the file's contents. _Real World Haskell_ touches on how laziness can make this problimatic, without really giving a solution, other than throwing in a putStr to force evaluation, which can't be done here. How can I make hGetContents strict, to ensure the file contents are really read before it gets closed?
In general you don't want to read a file as a String non-lazily. That uses a lot of memory, because String is just a type alias for [Char], which means a linked list of Char values. To read an entire file eagerly the proper way is in most cases using Data.ByteString or Data.ByteString.Char8, which have a strict interface. A ByteString from those modules is always either non-evaluated or completely evaluated, and it is a real byte array instead of a linked list. Both modules feature hGetContents: import qualified Data.ByteString.Char8 as B readFileLocked :: FilePath -> IO B.ByteString readFileLocked fn = withFile fn ReadMode $ \h -> do lockFile h -- for a suitable function lockFile B.hGetContents h It is, BTW, always preferable to use withFile over openFile, if you can. This makes your code cleaner and also exception-safe. Greets, Ertugrul -- nightmare = unsafePerformIO (getWrongWife >>= sex) http://ertes.de/

Ertugrul Soeylemez wrote:
readFileLocked :: FilePath -> IO B.ByteString readFileLocked fn = withFile fn ReadMode $ \h -> do lockFile h -- for a suitable function lockFile B.hGetContents h
It is, BTW, always preferable to use withFile over openFile, if you can. This makes your code cleaner and also exception-safe.
Unless there is a better locking primative than waitToSetLock available, I don't know how to build your lockFile function. It seems that it would have a side effect of closing the handle. It could return a new handle like this, but then withFile's automatic close of the file would be defeated. lockFile h = do lockfd <- handleToFd h -- closes h waitToSetLock lockfd (lockType mode, AbsoluteSeek, 0, 0) newh <- fdToHandle lockfd return newh Here's what I'm using instead. withFileLocked file mode action = do -- TODO: find a way to use bracket here handle <- openFile file mode lockfd <- handleToFd handle -- closes handle waitToSetLock lockfd (lockType mode, AbsoluteSeek, 0, 0) handle' <- fdToHandle lockfd ret <- action handle' hClose handle' return ret where lockType ReadMode = ReadLock lockType _ = WriteLock BTW, thanks for the hint that ByteString has a strict getContents! I was prototyping my code with String and thought I'd worry about optimisation later, but that is a good reason to use ByteString up front. Jimmy Wylie wrote:
Implementations should enforce as far as possible, at least locally to the Haskell process, multiple-reader single-writer locking on files.
According to strace, this does not involve any system-level locking with flock/fcntl/lockf. It is done internally to the ghc process. -- see shy jo

I don't know how to build your lockFile function. It seems that it would have a side effect of closing the handle. It could return a new handle like this, but then withFile's automatic close of the file would be defeated.
lockFile h = do lockfd<- handleToFd h -- closes h waitToSetLock lockfd (lockType mode, AbsoluteSeek, 0, 0) newh<- fdToHandle lockfd return newh
Here's what I'm using instead.
withFileLocked file mode action = do -- TODO: find a way to use bracket here handle<- openFile file mode lockfd<- handleToFd handle -- closes handle waitToSetLock lockfd (lockType mode, AbsoluteSeek, 0, 0) handle'<- fdToHandle lockfd ret<- action handle' hClose handle' return ret where lockType ReadMode = ReadLock lockType _ = WriteLock
I was looking here: http://www.haskell.org/ghc/docs/6.12.2/html/libraries/unix-2.4.0.1/System-Po... Instead of creating the handle, then converting to an fd, only to return a new handle, why don't you start with the file descriptor and convert at the end of the process. I don't have time for a full piece of code, but maybe something like this: lockFile file = do fd <- openFd ReadOnly Nothing defaultFileFlags waitToSetLock fd (lockType mode, AbsoluteSeek, 0, 0) handle <- fdToHandle fd return handle You could also use the same sort of code in your withFileLocked function.
According to strace, this does not involve any system-level locking with flock/fcntl/lockf. It is done internally to the ghc process.
Thanks for testing that out. I appreciate the information. Ciao, Jimmy

Joey Hess
Ertugrul Soeylemez wrote:
readFileLocked :: FilePath -> IO B.ByteString readFileLocked fn = withFile fn ReadMode $ \h -> do lockFile h -- for a suitable function lockFile B.hGetContents h
It is, BTW, always preferable to use withFile over openFile, if you can. This makes your code cleaner and also exception-safe.
Unless there is a better locking primative than waitToSetLock available, I don't know how to build your lockFile function. It seems that it would have a side effect of closing the handle. It could return a new handle like this, but then withFile's automatic close of the file would be defeated.
I wasn't addressing the locking issue, but rather the laziness issue. The lockFile function was just a placeholder. Greets, Ertugrul -- nightmare = unsafePerformIO (getWrongWife >>= sex) http://ertes.de/

It is, BTW, always preferable to use withFile over openFile, if you can. This makes your code cleaner and also exception-safe.
Hi Ertugrul, I don't think I quite understand. How is withFile exception-safe? Under the covers it's using openFile. I was under the impression withFile was just a nice way to remove boilerplate file operation code. Here's what I found on hoogle: "withFile http://haskell.org/hoogle/?hoogle=withFile name mode act opens a file using openFile http://haskell.org/hoogle/?hoogle=openFile and passes the resulting handle to the computation act. The handle will be closed on exit from withFile http://haskell.org/hoogle/?hoogle=withFile, whether by normal termination or by raising an exception." Thanks, Jimmy

Jimmy Wylie
It is, BTW, always preferable to use withFile over openFile, if you can. This makes your code cleaner and also exception-safe.
I don't think I quite understand. How is withFile exception-safe? Under the covers it's using openFile. I was under the impression withFile was just a nice way to remove boilerplate file operation code. Here's what I found on hoogle: "withFile http://haskell.org/hoogle/?hoogle=withFile name mode act opens a file using openFile http://haskell.org/hoogle/?hoogle=openFile and passes the resulting handle to the computation act. The handle will be closed on exit from withFile http://haskell.org/hoogle/?hoogle=withFile, whether by normal termination or by raising an exception."
The last statement is the point: "[...] whether by normal termination or by raising an exception." If you openFile and hClose manually, then you need to take care of exceptions yourself. You need to make sure that hClose is called in all cases. For example, what if reading the file throws an exception, which is not catched? hClose may be skipped that way. withFile ensures that the handle is always closed, even if an exception escapes your function. Greets, Ertugrul -- nightmare = unsafePerformIO (getWrongWife >>= sex) http://ertes.de/

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/11/10 01:14 , Jimmy Wylie wrote:
I don't think I quite understand. How is withFile exception-safe? Under the covers it's using openFile. I was under the impression withFile was just
Under the covers it's also using Control.Exception.bracket, which you should probably look at. - -- brandon s. allbery [linux,solaris,freebsd,perl] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAky0RykACgkQIn7hlCsL25UEDwCgrx0fhk1b/MGjWdwixp9vIZW9 zyYAoJyV5p0GG4Ye/wXhvW0UjYZyMuNW =1Ces -----END PGP SIGNATURE-----
participants (5)
-
Brandon S Allbery KF8NH
-
Ertugrul Soeylemez
-
Isaac Dupree
-
Jimmy Wylie
-
Joey Hess