On Tuesday 29 January 2008 14:44:42 Mads Lindstrøm wrote:

If you replace the `putStrLn (show res)` with this:

  mapM_ (\x -> putStr (show x) >> hFlush stdout) res

it works.

Hm, unfortunately not for me (Linux, GHC 6.8.2) ...

I _think_ the problem is that `putStrLn  (show res)` will wait until it has read all of res. But as the client do not know when the server is finished sending data, the client will wait forever.

But if this is the cause, it should happen on the server as well, because the call to hGetContents on the server side will also continue to wait for data. I also suspected some laziness issues here (like both sides waiting for each other to start evaluating), but lazy network IO doesn't make that much sense, does it? If I tell the program to send something, it should send it _now_. Any further hints are still appreciated :) Thanks, Timo

On Tuesday 29 January 2008 17:12:19 you wrote:

There was a similar bug in lazy bytestring's hGetContents a while back which involve it waiting for a whole chunk and not returning short reads, but from watching the strace of this code, GHC is reading byte-by-byte (which is actually pretty dumb, but functions).

I have to apologize, I probably got something wrong when trying the solution from Mads. I did it again now and now it seems to work (apart from the client still waiting for more data instead of exiting, but thats probably due to the way the data is read). I actually want to transmit ByteStrings in exactly this way (one request to the server, the server does some processing and sends a response back), but couldn't get it to work and therefore tried with ordinary strings. Now I can move forward to ByteStrings. Thanks! Timo

On Wednesday 30 January 2008 13:03:27 you wrote:

Just don't use hGetContents in any serious code, or any program longer than 4 lines.

What else do you suggest? I just want to read something out of the socket without knowing it's length beforehand (my example here used ordinary Strings, but actually I want to do it with ByteStrings). Thanks, Timo

Timo B. Hübel wrote:

On Wednesday 30 January 2008 13:03:27 you wrote:

...

Just don't use hGetContents in any serious code, or any program longer than 4 lines.

What else do you suggest? I just want to read something out of the socket without knowing it's length beforehand (my example here used ordinary Strings, but actually I want to do it with ByteStrings).

How much shall you read? Will you wait if not that much data is available? This is a question all network protocols have to answer! There are two traditional solutions: Implement a line based protocol. Read one line at a time. In that case hGetLine is your friend. (Actually any delimeter, but it's traditionally lines) Implement a known-chunk-size protocol, either fixed to a constant N, or transmit a length word as the first word. In that case, the extraordinarly ugly and clumsy hGetBuf is your friend, but you might wrap it into something more comfortable. The third, but more sophisticated answer is to use non-blocking IO, read 'only what is available', decide if it's enough to process, if not store it in some local buffer until next time. This is much more work and easy to implement bugs in, but you need it for true streaming protocols. In that case hGetBufNonBlocking is your friend. The vast majority of internet protocols are line based, at some level, and so use solution 1. In cases 2 and 3 it happens that ByteString offers a cleaner API than System.IO, even if you didn't really want to use ByteString, since it provides hGet and hGetNonBlocking, no messing around with Ptrs. I strongly suspect for your example you want solution 1 and hGetLine, though. (Which works just as well with or without ByteString) Jules PS "whatever you do, just don't use hGetContents" , print this out onto a T-shirt transfer and apply it to the front of your monitor.

Timo B. Hübel wrote:

On Wednesday 30 January 2008 13:32:42 you wrote:

...

Timo B. Hübel wrote:

...

On Wednesday 30 January 2008 13:03:27 you wrote:

...

Just don't use hGetContents in any serious code, or any program longer than 4 lines. What else do you suggest? I just want to read something out of the socket without knowing it's length beforehand (my example here used ordinary Strings, but actually I want to do it with ByteStrings). [...]

I strongly suspect for your example you want solution 1 and hGetLine, though. (Which works just as well with or without ByteString)

Okay, but then I have to make sure that my strings won't contain any newline characters, right? If this is the case, another question raises up: I am using Data.Binary to do the serialization of my data structures to ByteString, so does anybody know if this makes guarantees about newline characters in the resulting ByteString? Otherwise I would go for the "transmit the length of what to expect"-solution.

Fortunately there is an easy way to hide newlines. Use "show" That will wrap newlines as \n, as well as coping with other odd characters like NULL which might upset a C library (if you're talking to C at any point). Then you use "read" on the far end. Otherwise, you make your protocol more sophisticated in some way, like "messages is ended by a line which only contains '.'", which is the SMTP and, AFAICR, NNTP solution to this particular sub-problem. Of course show/read *is* one way of making your protocol more sophisticated. It just happens to be a really easy hack, for haskell users :) Jules

Timo B. Hübel wrote:

On Wednesday 30 January 2008 13:51:58 you wrote:

...

...

Okay, but then I have to make sure that my strings won't contain any newline characters, right? If this is the case, another question raises up: I am using Data.Binary to do the serialization of my data structures to ByteString, so does anybody know if this makes guarantees about newline characters in the resulting ByteString? Otherwise I would go for the "transmit the length of what to expect"-solution. Fortunately there is an easy way to hide newlines.

Use "show"

That will wrap newlines as \n, as well as coping with other odd characters like NULL which might upset a C library (if you're talking to C at any point).

Then you use "read" on the far end.

This sounds good, but don't I throw away all (possible) performance gains of transmitting ByteStrings directly when using show/read to convert them to ordinary strings and back?

Probably not all of them, but some of them, definitely. If you want to transmit an arbitrary bytestring then I'm pretty sure that transmitting a length word first is the way to go. An arbitrary bytestring can have any value in it, so there are no values left to act as delimiters :) You'd have to have some kind of escaping mechanism, like show, which is expensive. Jules

On Jan 30, 2008 4:32 AM, Jules Bean wrote:

The third, but more sophisticated answer is to use non-blocking IO, read 'only what is available', decide if it's enough to process, if not store it in some local buffer until next time. This is much more work and easy to implement bugs in, but you need it for true streaming protocols. In that case hGetBufNonBlocking is your friend.

If using bytestrings from the network, the network-bytestring package is what you need. If you're parsing small lumps from the network, Data.Binary's failures will probably make life harder than it should be for you since they can only be caught in IO. See http://www.haskell.org/haskellwiki/DealingWithBinaryData about a strict Get which might work for you. Also, if you want the above approach (read a bit, see if it's enough), see IncrementalGet in the binary-strict package which is a Get with a continuation monad that stops when it runs out of bytes and returns a continuation that you can give more data to in the future. AGL -- Adam Langley agl@imperialviolet.org http://www.imperialviolet.org 650-283-9641

On Jan 30, 2008 12:04 PM, Bryan O'Sullivan wrote:

Adam Langley wrote: I'd have expected it to look more like this:

data Result a = Failed String | Finished B.ByteString a | Partial (B.ByteString -> Result a)

(The change here is from a list to a singleton.) I don't think I care much for the extra boxing and reversing this involves.

Well, since you're probably the /only/ user you can pretty much say how it works ;) The original interface was designed so that you can yield a list of results as you parse. I guess that, since you get the remainder anyway, you can chain these together if you like anyway. So, if I don't hear otherwise soon, I'll probably push a new version of binary-strict later on today with the interface above. AGL -- Adam Langley agl@imperialviolet.org http://www.imperialviolet.org 650-283-9641

Adam Langley wrote:

On Jan 30, 2008 1:07 PM, Adam Langley wrote:

...

So, if I don't hear otherwise soon, I'll probably push a new version of binary-strict later on today with the interface above.

It's in the darcs now, http://darcs.imperialviolet.org/binary-strict

Thanks!

From: haskell-cafe-bounces@haskell.org [mailto:haskell-cafe-bounces@haskell.org] On Behalf Of Peter Verswyvelen

As a newbie I made a nice little program that called readFile and writeFile on the same filename, but of course the file handle of the readFile was not closed yet => access denied. A nice case of getting bitten by my imperative background.

So I guess hGet/hGetNonBlocking/ByteString is also the correct way to solve this?

More than one person has posted previously about the flaws and traps of lazy IO. A common position seems to be "don't do lazy IO". http://article.gmane.org/gmane.comp.lang.haskell.cafe/20106/ Peter Simons has this library: http://cryp.to/blockio/ BTW, where's the tutorial that Peter wrote? http://article.gmane.org/gmane.comp.lang.haskell.cafe/4011/ And there are other IO libraries out there. Bulat has done a lot of work on stream IO, I recall.

PS: I would love to see an immutable filesystem that does not allow writing to files, it only creates new files and garbage collects files that have no incoming reference anymore... Just like a garbage collected heap, and a bit like an OLAP databases (as far as I remember my DB theory...) Besides the performance bottleneck, does something like that exists?

This might interest you: http://okmij.org/ftp/Computation/Continuations.html#zipper-fs Alistair ***************************************************************** Confidentiality Note: The information contained in this message, and any attachments, may contain confidential and/or privileged material. It is intended solely for the person(s) or entity to which it is addressed. Any review, retransmission, dissemination, or taking of any action in reliance upon this information by persons or entities other than the intended recipient(s) is prohibited. If you received this in error, please contact the sender and delete the material from any computer. *****************************************************************

On 31 Jan 2008, at 1:23 AM, Reinier Lamers wrote:

Bayley, Alistair wrote:

...

More than one person has posted previously about the flaws and traps of lazy IO. A common position seems to be "don't do lazy IO".

Still, when I was browsing the Haskell' wiki a few days ago, I couldn't find any proposal to remove lazy I/O or move it into some special System.IO.Lazy (or System.IO.UnsafeEvilFunctionsThatSacrificeBabies) ...

Sacrificing babies is not a unique characteristic of lazy IO, of course; it's true of any file IO on a non-versioning file system. jcc

Peter Verswyvelen wrote:

Then I tried the "seq" hack to force the handle opened by readFile to be closed, but that did not seem to work either. For example, the following still gave access denied:

main = do cs <- readFile "L:/Foo.txt" writeFile "L:/Foo.txt" $ seq (length cs) cs

This is unfortunately a classic beginner's mistake. You got the seq wrong here, which is very common. If you think about the way Haskell evaluates your code, writeFile isn't going to need the data that it's writing until after it's opened the file. Thus the seq won't be reduced until writeFile needs to write the file. The file is still open behind the scenes when writeFile begins, since the contents of cs have not yet been demanded, so writeFile's attempt to open the file fails. You need to float the call to seq out so that it's evaluated before the call to writeFile: length cs `seq` writeFile cs Almost everyone makes this mistake early on. Quite often, it's *exactly* this mistake that is made, with just the sequence of transformations you described. There's nothing wrong with hGetContents or readFile. They just ought to appear on level two, after you've defeated the lazy evaluation boss at the end of level one.

Judah Jacobson wrote:

On Jan 30, 2008 8:31 AM, Bryan O'Sullivan wrote:

...

Peter Verswyvelen wrote:

...

Then I tried the "seq" hack to force the handle opened by readFile to be closed, but that did not seem to work either. For example, the following still gave access denied:

main = do cs <- readFile "L:/Foo.txt" writeFile "L:/Foo.txt" $ seq (length cs) cs This is unfortunately a classic beginner's mistake. You got the seq wrong here, which is very common. [...] You need to float the call to seq out so that it's evaluated before the call to writeFile:

length cs `seq` writeFile cs

Another way of doing things: I've recently become a fan of Control.Exception.evaluate:

main = do cs <- readFile "L:/Foo.txt" evalute (length cs) writeFile "L:/Foo.txt" cs

This might be easier for beginners to understand than messing around with seq's (as long as you're already in the IO monad).

And even better is main = do cs <- strictReadFile "L:/Foo.txt" writeFile "L:/Foo.txt" cs which can be rewritten as main = writeFile "L:/Foo.txt" =<< strictReadFile "L:/Foo.txt" if you like such things. The problem is that strictReadFile isn't in the standard lib. My opinion is that readFile should *be* strict, and the lazy version should be an option with caveats. In bos's notation, I'd say that readFile should be strict, and on level 1. It does what people expect. Sure it runs out of memory if the file is very big, but I don't find that unexpected. lazyReadFile can go on level 2 after the boss. Jules

Peter Verswyvelen wrote:

main = do cs <- getLine putStrLn ("Hello "++cs)

looks so much like

main = do cs <- readFile "foo" writeFile "foo" cs

but in the first one cs is strict, while the second it is lazy... But that's not obvious.

Now I'm confused (which happens quite a lot, I'm afraid!) Prelude> readFile undefined *** Exception: Prelude.undefined Prelude> readFile undefined >>= \cs -> putStrLn "Hello" *** Exception: Prelude.undefined It seems that readFile is strict in its argument. As for getLine, it has no argument to be strict in. What am I missing? Dan

PS: I would love to see an immutable filesystem that does not allow writing to files, it only creates new files and garbage collects files that have no incoming reference anymore... Just like a garbage collected heap, and a bit like an OLAP databases (as far as I remember my DB theory...) Besides the performance bottleneck, does something like that exists?

Plan 9's venti is somewhat similar to this. though it's really a storage backend that you implement a filesystem on top of, and the fs winds up having a write cache, which is mutable in practice. The interesting thing is that the block's location is the cryptographic hash of its contents, which leads to all sorts of neat properties (as well as requiring immutability).