Network client - reading and writing to a socket - Beginners

newer
force strict expression evaluation

Network client - reading and writing to a socket

Manfred Lotz

31 Jul 2011 31 Jul '11

3:39 p.m.

Hi there, I'm trying to write a network client which connects to an IMAP server thus reading from and writing to socket 143. I have an initial solution which works except that laziness bites me. Perhaps the whole solution using connectTo and handle is the wrong approach. Here is the code: module Main where import System.IO import qualified Data.ByteString.Char8 as B import Network server :: String server = "127.0.0.1" port :: PortID port = PortNumber 143 sendMsg :: Handle -> String -> IO () sendMsg h m = do putStrLn ("Sending command: " ++ m) B.hPutStrLn h (B.pack m) >> hFlush h recvMsg :: Handle -> IO B.ByteString recvMsg h = do c <- B.hGet h 1 c' <- B.hGetNonBlocking h 40000 return $ B.concat [c,c'] main :: IO () main = withSocketsDo $ do h <- connectTo server port hSetBuffering h LineBuffering imapDialog h imapDialog :: Handle -> IO () imapDialog h = do greet <- recvMsg h B.putStrLn greet sendMsg h "1 LOGIN manfred \"password\"" resp <- recvMsg h B.putStrLn resp sendMsg h "2 SELECT HAM-learn" resp1 <- recvMsg h B.putStrLn resp1 sendMsg h "3 FETCH 1:* (uid flags internaldate body[header.fields (Message-Id)])" resp2 <- recvMsg h B.putStrLn resp2 sendMsg h "4 FETCH 1 (rfc822)" resp3 <- recvMsg h B.putStrLn resp3 The problem is that the message itself is some 30K big and I only get some 16K of the message. How could I force to get the whole message? -- Manfred

Show replies by date

David Place

31 Jul 31 Jul

4:02 p.m.

New subject: Network client - reading and writing to a socket

On Jul 31, 2011, at 11:39 AM, Manfred Lotz wrote:

...

How could I force to get the whole message?

Did you try hFlush?

Manfred Lotz

4:17 p.m.

New subject: Network client - reading and writing to a socket

On Sun, 31 Jul 2011 12:02:16 -0400 David Place wrote:

...

On Jul 31, 2011, at 11:39 AM, Manfred Lotz wrote:

...
How could I force to get the whole message?

Did you try hFlush?

I tried it like this recvMsg :: Handle -> IO B.ByteString recvMsg h = do c <- B.hGet h 1 hFlush h c' <- B.hGetNonBlocking h 40000 return $ B.concat [c,c'] and it didn't help. Still an incomplete message. -- Manfred

David Place

4:20 p.m.

New subject: Network client - reading and writing to a socket

____________________ David Place Owner, Panpipes Ho! LLC http://panpipesho.com d@vidplace.com On Jul 31, 2011, at 12:17 PM, Manfred Lotz wrote:

...

c <- B.hGet h 1 hFlush h c' <- B.hGetNonBlocking h 40000

Try putting one here too. Can't hurt right?

...

return $ B.concat [c,c']

Manfred Lotz

4:25 p.m.

New subject: Network client - reading and writing to a socket

On Sun, 31 Jul 2011 12:20:17 -0400 David Place wrote:

...

____________________ David Place Owner, Panpipes Ho! LLC http://panpipesho.com d@vidplace.com

On Jul 31, 2011, at 12:17 PM, Manfred Lotz wrote:

...
c <- B.hGet h 1 hFlush h c' <- B.hGetNonBlocking h 40000

Try putting one here too. Can't hurt right?

...
return $ B.concat [c,c']

Now I have recvMsg h = do c <- B.hGet h 1 hFlush h c' <- B.hGetNonBlocking h 40000 hFlush h return $ B.concat [c,c'] Doesn't help either. -- Manfred

David Place

4:42 p.m.

New subject: Network client - reading and writing to a socket

Shouldn't be laziness that is biting you. You are using strict ByteStrings. I'm stumped! Did you try artificially inserting something like this to force c and c'? On Jul 31, 2011, at 12:25 PM, Manfred Lotz wrote:

...

recvMsg h = do c <- B.hGet h 1 hFlush h

print c

...

c' <- B.hGetNonBlocking h 40000

print c'

...

hFlush h return $ B.concat [c,c']

Manfred Lotz

5:20 p.m.

New subject: Network client - reading and writing to a socket

On Sun, 31 Jul 2011 12:42:45 -0400 David Place wrote:

...

Shouldn't be laziness that is biting you. You are using strict ByteStrings. I'm stumped! Did you try artificially inserting something like this to force c and c'?

On Jul 31, 2011, at 12:25 PM, Manfred Lotz wrote:

...
recvMsg h = do c <- B.hGet h 1 hFlush h

print c

...
c' <- B.hGetNonBlocking h 40000

print c'

...
hFlush h return $ B.concat [c,c']

Didn't help either. If I add another resp4 <- recvMsg h B.putStrLn resp4 at the end of imapDialog then I get the rest of the message. -- Manfred

David Place

5:46 p.m.

New subject: Network client - reading and writing to a socket

On Jul 31, 2011, at 1:20 PM, Manfred Lotz wrote:

...

Didn't help either.

If I add another

resp4 <- recvMsg h B.putStrLn resp4

at the end of imapDialog then I get the rest of the message.

Sorry, I'm just shooting in the dark here. Did you try this? hSetBuffering h NoBuffering

Manfred Lotz

5:57 p.m.

New subject: Network client - reading and writing to a socket

On Sun, 31 Jul 2011 13:46:56 -0400 David Place wrote:

...

On Jul 31, 2011, at 1:20 PM, Manfred Lotz wrote:

...
Didn't help either.

If I add another

resp4 <- recvMsg h B.putStrLn resp4

at the end of imapDialog then I get the rest of the message.

Sorry, I'm just shooting in the dark here. Did you try this?

No sorry. I'm happy you try to help me.

...

hSetBuffering h NoBuffering

Didn't make a difference. -- Manfred

Patrick LeBoutillier

7:14 p.m.

New subject: Network client - reading and writing to a socket

Manfred,

...

The problem is that the message itself is some 30K big and I only get some 16K of the message.

How could I force to get the whole message?

My guess is that you can't. This call: c' <- B.hGetNonBlocking h 40000 tries to read as much as it can (up to 40000 bytes) but it won't block to wait for data. Perhaps the rest of your message is in a different TCP packet or delayed or whatever, but I think you have to keep on reading (and maybe block) until you know you have read the entire message. The IMAP specs will tell you how to identify the "end of the message". BTW: This issue is not Haskell specific. If you implement the same code in C, Perl or Java you will have to deal with the same problem. When you read from a socket, there is no general way of knowing that the other side has sent everything. Patrick

...

-- Manfred

_______________________________________________ Beginners mailing list Beginners@haskell.org http://www.haskell.org/mailman/listinfo/beginners

-- ===================== Patrick LeBoutillier Rosemère, Québec, Canada

David McBride

10:21 p.m.

New subject: Network client - reading and writing to a socket

It seems you are expected to fetch until you have the entire thing. You can't just fetch 40000 and hope you get it all. If you do it blocking, it will be almost guaranteed to block unless there were 40000 to fetch. If you do it non blocking, you have to keep fetching until you are sure you have the entire message. If you are doing this as simply as possible, that means you have to fetch, and then repeatedly check the string until you are sure you have the whole thing. You'll do that by fetching once, checking the beginning of the string for the {342} in the following example from the rfc: C: a004 fetch 12 body[header] S: * 12 FETCH (BODY[HEADER] {342} S: Date: Wed, 17 Jul 1996 02:23:25 -0700 (PDT) S: From: Terry Gray S: Subject: IMAP4rev1 WG mtg summary and minutes S: To: imap@cac.washington.edu S: cc: minutes@CNRI.Reston.VA.US, John Klensin S: Message-Id: S: MIME-Version: 1.0 S: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII S: S: ) And then fetching until you have the next 342 characters. This is pretty tedious in any language, not just haskell, but that's how this protocol is. If you want to do this the haskell way, which I highly recommend, I would pull out the network-attoparsec library. You should be able to use parseWith and pass it a recv function that returns bytestrings. Write a simple parser which will look for "* <num> FETCH ([a-zA-Z]+\[[a-zA-Z]+\] {[0-9]+} and then use that last number to take (take is an attoparsec function) that last byte count. If you wire this up to your network socket, it will parse and slurp up the response and it will pull the exact number of bytes needed and your code will be very elegant. Attoparsec is a little hard to use at first, but it is well worth if if you have a mind to parse internet protocols. On Sun, Jul 31, 2011 at 3:14 PM, Patrick LeBoutillier wrote:

...

Manfred,

...
The problem is that the message itself is some 30K big and I only get some 16K of the message.

How could I force to get the whole message?

My guess is that you can't. This call:

c' <- B.hGetNonBlocking h 40000

tries to read as much as it can (up to 40000 bytes) but it won't block to wait for data. Perhaps the rest of your message is in a different TCP packet or delayed or whatever, but I think you have to keep on reading (and maybe block) until you know you have read the entire message. The IMAP specs will tell you how to identify the "end of the message".

BTW: This issue is not Haskell specific. If you implement the same code in C, Perl or Java you will have to deal with the same problem. When you read from a socket, there is no general way of knowing that the other side has sent everything.

Patrick

...
-- Manfred

_______________________________________________ Beginners mailing list Beginners@haskell.org http://www.haskell.org/mailman/listinfo/beginners

-- ===================== Patrick LeBoutillier Rosemère, Québec, Canada

_______________________________________________ Beginners mailing list Beginners@haskell.org http://www.haskell.org/mailman/listinfo/beginners

David McBride

10:22 p.m.

New subject: Network client - reading and writing to a socket

Sorry I said network-attoparsec, the attoparsec library is all you need. I'm not sure where I got network-attoparsec from. On Sun, Jul 31, 2011 at 6:21 PM, David McBride wrote:

...

It seems you are expected to fetch until you have the entire thing. You can't just fetch 40000 and hope you get it all. If you do it blocking, it will be almost guaranteed to block unless there were 40000 to fetch. If you do it non blocking, you have to keep fetching until you are sure you have the entire message.

If you are doing this as simply as possible, that means you have to fetch, and then repeatedly check the string until you are sure you have the whole thing. You'll do that by fetching once, checking the beginning of the string for the {342} in the following example from the rfc:

C: a004 fetch 12 body[header] S: * 12 FETCH (BODY[HEADER] {342} S: Date: Wed, 17 Jul 1996 02:23:25 -0700 (PDT) S: From: Terry Gray S: Subject: IMAP4rev1 WG mtg summary and minutes S: To: imap@cac.washington.edu S: cc: minutes@CNRI.Reston.VA.US, John Klensin S: Message-Id: S: MIME-Version: 1.0 S: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII S: S: )

And then fetching until you have the next 342 characters. This is pretty tedious in any language, not just haskell, but that's how this protocol is.

If you want to do this the haskell way, which I highly recommend, I would pull out the network-attoparsec library. You should be able to use parseWith and pass it a recv function that returns bytestrings. Write a simple parser which will look for "* <num> FETCH ([a-zA-Z]+\[[a-zA-Z]+\] {[0-9]+} and then use that last number to take (take is an attoparsec function) that last byte count. If you wire this up to your network socket, it will parse and slurp up the response and it will pull the exact number of bytes needed and your code will be very elegant. Attoparsec is a little hard to use at first, but it is well worth if if you have a mind to parse internet protocols.

On Sun, Jul 31, 2011 at 3:14 PM, Patrick LeBoutillier wrote:

...
Manfred,

...
The problem is that the message itself is some 30K big and I only get some 16K of the message.

How could I force to get the whole message?

My guess is that you can't. This call:

c' <- B.hGetNonBlocking h 40000

tries to read as much as it can (up to 40000 bytes) but it won't block to wait for data. Perhaps the rest of your message is in a different TCP packet or delayed or whatever, but I think you have to keep on reading (and maybe block) until you know you have read the entire message. The IMAP specs will tell you how to identify the "end of the message".

BTW: This issue is not Haskell specific. If you implement the same code in C, Perl or Java you will have to deal with the same problem. When you read from a socket, there is no general way of knowing that the other side has sent everything.

Patrick

...
-- Manfred

_______________________________________________ Beginners mailing list Beginners@haskell.org http://www.haskell.org/mailman/listinfo/beginners

-- ===================== Patrick LeBoutillier Rosemère, Québec, Canada

_______________________________________________ Beginners mailing list Beginners@haskell.org http://www.haskell.org/mailman/listinfo/beginners

Manfred Lotz

1 Aug 1 Aug

7:16 p.m.

New subject: Network client - reading and writing to a socket

On Sun, 31 Jul 2011 15:14:23 -0400 Patrick LeBoutillier wrote:

...

Manfred,

...
The problem is that the message itself is some 30K big and I only get some 16K of the message.

How could I force to get the whole message?

My guess is that you can't. This call:

c' <- B.hGetNonBlocking h 40000

tries to read as much as it can (up to 40000 bytes) but it won't block to wait for data. Perhaps the rest of your message is in a different TCP packet or delayed or whatever, but I think you have to keep on reading (and maybe block) until you know you have read the entire message. The IMAP specs will tell you how to identify the "end of the message".

BTW: This issue is not Haskell specific. If you implement the same code in C, Perl or Java you will have to deal with the same problem. When you read from a socket, there is no general way of knowing that the other side has sent everything.

Hmm. I'm not quite sure you are fully right. On the one hand I believe that this could be an issue which arises in python/perl etc. as well. On the other hand I believe it should be possible to receive from a socket what is available at a certain point of time. I found this link http://sequence.complete.org/node/257, and when I run the code I get the full message from the imap server even if the message is a couple of megabytes big. I have to figure out how to use the code for my need as I do not get the input from the keyboard. As a haskell beginner things like this are not always trivial. -- Thanks, Manfred

...

Patrick

...
-- Manfred

_______________________________________________ Beginners mailing list Beginners@haskell.org http://www.haskell.org/mailman/listinfo/beginners

-- Manfred

Chaddaï Fouché

2 Aug 2 Aug

12:59 a.m.

New subject: Network client - reading and writing to a socket

On Mon, Aug 1, 2011 at 9:16 PM, Manfred Lotz wrote:

...

On Sun, 31 Jul 2011 15:14:23 -0400 Patrick LeBoutillier wrote:

...
Manfred,

...
The problem is that the message itself is some 30K big and I only get some 16K of the message.

How could I force to get the whole message?

My guess is that you can't. This call:

c' <- B.hGetNonBlocking h 40000

tries to read as much as it can (up to 40000 bytes) but it won't block to wait for data. Perhaps the rest of your message is in a different TCP packet or delayed or whatever, but I think you have to keep on reading (and maybe block) until you know you have read the entire message. The IMAP specs will tell you how to identify the "end of the message".

BTW: This issue is not Haskell specific. If you implement the same code in C, Perl or Java you will have to deal with the same problem. When you read from a socket, there is no general way of knowing that the other side has sent everything.

Hmm. I'm not quite sure you are fully right. On the one hand I believe that this could be an issue which arises in python/perl etc. as well. On the other hand I believe it should be possible to receive from a socket what is available at a certain point of time.

Yes and that's exactly what you have done : your hGetLine blocks until there is something to read on the socket and then your hGetNonBlocking gets *everything* there is to read on this socket at this exact moment... Except as had been said by others that your message has been split into several packets and they're not all there when you hGet. What you want is not "what is available at a certain point of time", you want to read a whole message, except that this "message" notion is only in your head (and in a specific protocol, IMAP here) it has no direct relevance to how things happens on the network.

...

I found this link http://sequence.complete.org/node/257, and when I run the code I get the full message from the imap server even if the message is a couple of megabytes big.

Of course you get the whole message ! This code try to read (with blocking calls) forever what's on your socket, it reads it, send it on a TChan and then retry reading it, bit by bit it gets your whole message, of course it has no idea that it got your whole message and if nothing is done it will continue to wait on your socket for all eternity... The key point is this function :

...

listenLoop :: IO a -> TChan a -> IO () listenLoop act chan = sequence_ (repeat (act >>= atomically . writeTChan chan))

This does not stop short of an exception. Two of those loops are started each in their own thread (so that they don't block the rest of the program) to read stdin and a socket respectively.

...

I have to figure out how to use the code for my need as I do not get the input from the keyboard.

This code is probably not doing what you want, this is a toy example where most of the complexity comes from handling two source of input simultaneously and collating their answer on the same TChan. Its main problem is that it don't even try to read messages, it just read everything and doesn't know when it's finished. You need to listen to David's advice and read a bit on the protocol you're trying to handle and how it encode the notion of message, how it signals that the message has ended. -- Jedaï

Manfred Lotz

11:39 a.m.

New subject: Network client - reading and writing to a socket

On Tue, 2 Aug 2011 02:59:44 +0200 Chaddaï Fouché wrote:

...

On Mon, Aug 1, 2011 at 9:16 PM, Manfred Lotz wrote:

...
On Sun, 31 Jul 2011 15:14:23 -0400 Patrick LeBoutillier wrote:

...
Manfred,

...
The problem is that the message itself is some 30K big and I only get some 16K of the message.

How could I force to get the whole message?

My guess is that you can't. This call:

c' <- B.hGetNonBlocking h 40000

tries to read as much as it can (up to 40000 bytes) but it won't block to wait for data. Perhaps the rest of your message is in a different TCP packet or delayed or whatever, but I think you have to keep on reading (and maybe block) until you know you have read the entire message. The IMAP specs will tell you how to identify the "end of the message".

BTW: This issue is not Haskell specific. If you implement the same code in C, Perl or Java you will have to deal with the same problem. When you read from a socket, there is no general way of knowing that the other side has sent everything.

Hmm. I'm not quite sure you are fully right. On the one hand I believe that this could be an issue which arises in python/perl etc. as well. On the other hand I believe it should be possible to receive from a socket what is available at a certain point of time.

Yes and that's exactly what you have done : your hGetLine blocks until there is something to read on the socket and then your hGetNonBlocking gets *everything* there is to read on this socket at this exact moment... Except as had been said by others that your message has been split into several packets and they're not all there when you hGet.

What you want is not "what is available at a certain point of time", you want to read a whole message, except that this "message" notion is only in your head (and in a specific protocol, IMAP here) it has no direct relevance to how things happens on the network.

...
I found this link http://sequence.complete.org/node/257, and when I run the code I get the full message from the imap server even if the message is a couple of megabytes big.

Of course you get the whole message ! This code try to read (with blocking calls) forever what's on your socket, it reads it, send it on a TChan and then retry reading it, bit by bit it gets your whole message, of course it has no idea that it got your whole message and if nothing is done it will continue to wait on your socket for all eternity... The key point is this function :

...
listenLoop :: IO a -> TChan a -> IO () listenLoop act chan = sequence_ (repeat (act >>= atomically . writeTChan chan))

This does not stop short of an exception. Two of those loops are started each in their own thread (so that they don't block the rest of the program) to read stdin and a socket respectively.

...
I have to figure out how to use the code for my need as I do not get the input from the keyboard.

This code is probably not doing what you want, this is a toy example where most of the complexity comes from handling two source of input simultaneously and collating their answer on the same TChan. Its main

The code works pretty well, and nevertheless you are right. Because it is operating in its own thread it doesn't matter if it blocks. I guess I understand now. I have to know what to expect back from a certain command (so that I can adjust the receive accordingly) I did send to the imap server. Thanks to you and the others for explaining. -- Manfred

5239

Age (days ago)

5241

Last active (days ago)

List overview

Download

14 comments

5 participants

participants (5)

Chaddaï Fouché
David McBride
David Place
Manfred Lotz
Patrick LeBoutillier