On Sat, Jan 14, 2012 at 1:29 AM, Bardur Arantsson wrote:

So, the API becomes something like:

  runSocketServer :: ((Chan a, Chan b) -> IO ()) -> ... -> IO ()

where the first parameter contains the "client logic" and "A" is the type of the messages from the client and "B" is the type of the messages which are sent back to the client.

Thanks, that's a good idea. Even if I only plan to receive in one thread, placing the messages in a Chan or TChan helps separate my application thread from the complexities of connection management. Is there something on Hackage that will do this for me? Or will I need to roll my own? Namely, convert a network connection to a pair of channels, and close the connection automatically. Something like this: -- | Spawn two threads, one which populates the first channel with messages -- from the other host, and another which reads the second channel and sends -- its messages to the other host. -- -- Run the given computation, passing it these channels. When the computation -- completes (or throws an exception), sending and receiving will stop, and the -- connection will be closed. -- -- If either the receiving thread or sending thread encounter an exception, -- sending and receiving will stop, and an asynchronous exception will be -- thrown to your thread. channelize :: IO msg_in -- ^ Receive callback -> (msg_out -> IO () -- ^ Send callback -> IO () -- ^ Close callback -> (TChan msg_in -> TChan msg_out -> IO a) -- ^ Inner computation -> IO a

Disregard that last comment on `TChan`s; retry blocks. you learn a new thing every day [= Daniel On 14 January 2012 11:27, Daniel Waterworth wrote:

I've been trying to write networking code in Haskell too. I've also come to the conclusion that channels are the way to go. However, what's missing in the standard `Chan` type, which is essential for my use-case, is the ability to do the equivalent of the unix select call. My other slight qualm is that the type doesn't express the direction of data (though this is easy to add afterwards).

I know of the chp package, but in order to learn how it worked, I spent a day writing my own version. I've kept the API similar to that of the standard Chan's. If it would be useful to you as well, I'll happily open source it sooner rather than later,

Daniel

p.s I'd avoid the TChan for networking code as reading from a TChan is a busy operation. [1]

[1] http://hackage.haskell.org/packages/archive/stm/2.2.0.1/doc/html/src/Control...

On 14 January 2012 10:42, Joey Adams wrote:

...

On Sat, Jan 14, 2012 at 1:29 AM, Bardur Arantsson wrote:

...

So, the API becomes something like:

  runSocketServer :: ((Chan a, Chan b) -> IO ()) -> ... -> IO ()

where the first parameter contains the "client logic" and "A" is the type of the messages from the client and "B" is the type of the messages which are sent back to the client.

Thanks, that's a good idea.  Even if I only plan to receive in one thread, placing the messages in a Chan or TChan helps separate my application thread from the complexities of connection management.

Is there something on Hackage that will do this for me?  Or will I need to roll my own?  Namely, convert a network connection to a pair of channels, and close the connection automatically.  Something like this:

   -- | Spawn two threads, one which populates the first channel with messages    -- from the other host, and another which reads the second channel and sends    -- its messages to the other host.    --    -- Run the given computation, passing it these channels.  When the computation    -- completes (or throws an exception), sending and receiving will stop, and the    -- connection will be closed.    --    -- If either the receiving thread or sending thread encounter an exception,    -- sending and receiving will stop, and an asynchronous exception will be    -- thrown to your thread.    channelize :: IO msg_in             -- ^ Receive callback                   -> (msg_out -> IO ()     -- ^ Send callback                   -> IO ()                 -- ^ Close callback                   -> (TChan msg_in -> TChan msg_out -> IO a)                                            -- ^ Inner computation                   -> IO a

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Hi Peter, streamproc is a very interesting package, I'll surely use it somewhere in the future. However, I'm not convinced that this solves my immediate problem, but perhaps this is due to my inexperience with arrows. My problem is: I have a number of network connections and I have a system that does things. I want the network connections to interact with the system. I also want the system to be able to interact with the network connections by way of a pub/sub style message bus. The only way I can see stream processors working in this scenario is if all of the events of the system are handled in a single thread. The events are then pushed into the stream processor and actions are pulled out. This isn't acceptable because the amount of logic in the stream processor will be fairly small for my problem in comparison with the logic that is required to mux/demux events/actions onto sockets. It's also a problem that there's a single threaded bottleneck. Daniel On 14 January 2012 11:58, Peter Simons wrote:

Hi Daniel,

 > I've been trying to write networking code in Haskell too. I've also  > come to the conclusion that channels are the way to go.

isn't a tuple of input/output channels essentially the same as a stream processor arrow? I found the example discussed in the "arrow paper" [1] very enlightening in that regard. There also is a Haskell module that extends the SP type to support monadic IO at [2].

Take care, Peter

[1] http://www.ittc.ku.edu/Projects/SLDG/filing_cabinet/Hughes_Generalizing_Mona... [2] http://hackage.haskell.org/package/streamproc

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

On 01/14/2012 11:42 AM, Joey Adams wrote:

On Sat, Jan 14, 2012 at 1:29 AM, Bardur Arantsson wrote:

...

So, the API becomes something like:

runSocketServer :: ((Chan a, Chan b) -> IO ()) -> ... -> IO ()

where the first parameter contains the "client logic" and "A" is the type of the messages from the client and "B" is the type of the messages which are sent back to the client.

Thanks, that's a good idea. Even if I only plan to receive in one thread, placing the messages in a Chan or TChan helps separate my application thread from the complexities of connection management.

Unless TCP is an absolute requirement, something like 0MQ[1,2] may be worth investigating. It handles all the nasty details and you get a simple message-based interface with lots of nice things like pub-sub, request-reply, etc. etc. [1] http://hackage.haskell.org/package/zeromq-haskell-0.8.2 [2] http://www.zeromq.org/

I'd say use of asynchronous exceptions should be a last resort. Developers should be encouraged to explicitly model any event notification system they use. Regards, Dave On Tue, Jan 17, 2012 at 1:42 AM, Simon Marlow wrote:

This is an interesting problem, I think I might incorporate parts of it into the next revision of my Concurrent Haskell tutorial.

It sounds like you're getting overwhelmed by several different problems, and dealing with them separately would probably help. e.g. you want some infrastructure to run two threads and send an exception to one whenever the other one dies. You also want to be able to avoid a thread being interrupted while performing an operation that should be atomic, like sending a message - this is slightly tricky, because there's a tradeoff between keeping the thread responsive and not interrupting an operation. The biggest hammer is maskUninterruptible, which can be used if all else fails.

Whether Network.TLS supports simultaneous read and write I don't know, but you can examine the code or talk to the maintainer. If it doesn't, adding a layer of locking is straightforward, and doesn't increase overall complexity (it's localised).

Cheers, Simon

On 14/01/2012 05:24, Joey Adams wrote:

...

I'm not happy with asynchronous I/O in Haskell. It's hard to reason about, and doesn't compose well. At least in my code.

I'm currently trying to build a networking layer for my application using Network.TLS. Here is a rather minimalist API:

newtype Connection = Connection (TLSCtx Handle)

connectClient :: Handle -- ^ Connection handle, as returned by 'connectTo' -> X509 -- ^ TLS certificate (i.e. public key) -> IO Connection

connectServer :: Handle -- ^ Connection handle, as returned by 'accept' -> X509 -- ^ TLS certificate (i.e. public key) -> TLS.PrivateKey -- ^ TLS private key -> IO Connection

close :: Connection -> IO ()

sendMessage :: Connection -> Message -> IO ()

recvMessage :: Connection -> ByteString -> IO (Message, ByteString)

The module provides little more than connection initialization and message serialization. I don't try to use locks or STM to multiplex the connection or, in the case of recvMessage, hide connection state. I just be sure to only use sendMessage in one thread at a time, only use recvMessage in one thread at a time, and marshal the "extra bytes" parameter of recvMessage from call to call (with the help of StateT).

I wrote a simple "chat server" to test it. The client turned out okay:

main :: IO () main = do cert<- getCertificate handle<- connectTo "localhost" (PortNumber 1337) conn<- connectClient handle cert _<- forkIO $ forever $ do s<- getLine sendMessage conn $ TestMessage s forever $ flip runStateT B.empty $ do msg<- StateT $ recvMessage conn case msg of TestMessage s -> liftIO $ putStrLn s _ -> liftIO $ hPrintf stderr "Warning: unrecognized message from server: %s\n" (messageTypeName msg)

The only glaring problem is that, if the user presses Ctrl+D, the forked (sending) thread dies, but the main (receiving) thread lingers. I'd have to add exception handlers to ensure that when one thread dies, the other thread dies too.

However, the server is an abomination (see attachment).

Unfortunately, it's not as simple as "spawn one thread per client". We need at least two threads, one to listen for messages from the client, and another to send messages to the client. GHC won't let us simultaneously, in the same thread, wait for input from a connection and wait for an STM transaction to succeed.

Another source of complexity is: what if we throw an exception at a thread while it is in the middle of sending a packet? Then we can't shut down the connection properly (i.e. Network.TLS.bye), because the receiver might think the close_notify packet is part of the interrupted packet.

Having a thread for each client is good, as it:

* Lets us think about each client separately. No need to turn our code inside out or write one big loop that juggles all the clients.

* Isolates exceptions. If sendMessage or recvMessage throws an exception, it doesn't bring the whole server down.

On the other hand, having multiple threads interact with a single client is hard to think about:

* We have to synchronize the threads (e.g. when one dies, kill the other one)

* Multiple places where an exception can arise

* Can multiple threads interact with the connection handle simultaneously?

So why don't I make my connection API handle some of this? Well, I tried. There are so many ways to do it, and I couldn't find a way that simplified usage much. The approach used by Handle and by Network.TLS is to use MVars and IORefs to ensure that, if two threads access the same connection, the connection doesn't become totally corrupt. If I do the same, then I'll have *three* layers of locking under the hood.

Worse, the locking done by Handle and Network.TLS doesn't guarantee much. I don't know if it's safe to have one thread sending and another thread receiving. Especially in the case of Network.TLS, where 'recvData' automatically handshakes in some cases, which sends packets. Since I don't know how much thread safety to expect, I can't write networking code and know for sure that it is safe.

I'm certainly not protected from interleaved data if multiple threads send on the same handle. For example:

import Control.Concurrent import System.IO

main :: IO () main = do hSetBuffering stdout NoBuffering _<- forkIO $ putStrLn "One sentence." putStrLn "Another sentence."

produces:

AnOonteh esre nsteenntceen.c e.

That is, I can't rely on putStrLn being "atomic". To produce intelligible output (without changing the buffering mode), I have to "lock" the output each time I write something. putStrLn doesn't do it for me.

=== Summary ===

In Haskell, sound logic and a great type system lead to elegant, composable code in a variety of domains, such as:

* Expression evaluation * Parsing * Concurrent programming (thanks to STM)

Asynchronous I/O is tricky. However, Haskell currently does little to alleviate the complexity (at least for me).

How can we structure network protocol APIs so that they stack well (e.g. only lock once, rather than locking each layer's connection state)? How can we deal with I/O errors without having to think about them at every turn?

For now, how can I structure my application's communication API so it's less messy to use?

Thanks, - Joey

______________________________**_________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/**mailman/listinfo/haskell-cafehttp://www.haskell.org/mailman/listinfo/haskell-cafe

______________________________**_________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/**mailman/listinfo/haskell-cafehttp://www.haskell.org/mailman/listinfo/haskell-cafe