ANN: conduit-network-stream, A base layer for network protocols with Conduits
Hello everyone, I just uploaded my "conduit-network-stream" package to hackage. It's a base layer for network protocols based on the "Conduit" package by Michael Snoyman which makes it possible to send multiple messages over a continuous network connection in a convenient way. I wrote more about it at: https://github.com/mcmaniac/conduit-network-stream/blob/master/README.md It is available on hackage and github: http://hackage.haskell.org/package/conduit-network-stream https://github.com/mcmaniac/conduit-network-stream Until the documentation on hackage is generated, I also host the haddock documentation at: http://hs.nils.cc/conduit-network-stream/index.html For any questions/problems/requests, please ask! - Nils
Hello, Nils. Can you describe if there any difference with latest conduit API (yield, await) that can be used to write functions in a very similar style, but without using exeternal packages:
runTCPServer settings app = appSource ap $$ go =$= CL.mapM_ encode =$ appSink app where go = forever $ do bp <- decode <$> await {- decode is inlined here assuming it can return different types -} {- logic here-} yield $ result mapM (yeild) [result1,result2...]
more over you don't need to write appSource $$ ... appSink on the top. Here is an example of a client that authorizes and then reads commands from TBMChan and requests server using json format.
runTCPClient settings ad go where go = do source $$ await is <- authorize when is loop' authorize = do yield u >> yield (S8.pack "\n") $$ sink -- send authorization token mx <- source $$ sinkParser json -- get responce ... loop' = do minfo <- atomically $ readTBMChan ch case minfo of Nothing -> return () Just (message, respBox) -> do yield message $$ C.concatMap (SL.toChunks . encode) =$ sink resp <- source $$ sinkParser json atomically $ putTMVar respBox resp loop' source = ppSource ad sink = appSink ad
On 24 February 2013 22:23, Nils
Hello everyone,
I just uploaded my "conduit-network-stream" package to hackage. It's a base layer for network protocols based on the "Conduit" package by Michael Snoyman which makes it possible to send multiple messages over a continuous network connection in a convenient way.
I wrote more about it at:
https://github.com/mcmaniac/**conduit-network-stream/blob/** master/README.mdhttps://github.com/mcmaniac/conduit-network-stream/blob/master/README.md
It is available on hackage and github:
http://hackage.haskell.org/**package/conduit-network-streamhttp://hackage.haskell.org/package/conduit-network-stream https://github.com/mcmaniac/**conduit-network-streamhttps://github.com/mcmaniac/conduit-network-stream
Until the documentation on hackage is generated, I also host the haddock documentation at:
http://hs.nils.cc/conduit-**network-stream/index.htmlhttp://hs.nils.cc/conduit-network-stream/index.html
For any questions/problems/requests, please ask!
- Nils
______________________________**_________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/**mailman/listinfo/haskell-cafehttp://www.haskell.org/mailman/listinfo/haskell-cafe
-- Alexander
Hi Alexander, Am 25.02.2013 06:52, schrieb Alexander V Vershilov:
Can you describe if there any difference with latest conduit API (yield, await) that can be used to write functions in a very similar style, but without using exeternal packages:
I have indeed written this library for the 0.5.6 API, so things might have changed a bit in the 1.0 API, but I'd be surprised if the fundamental flaws of this approach would have been fixed.
runTCPServer settings app = appSource ap $$ go =$= CL.mapM_ encode =$ appSink app where go = forever $ do bp <- decode <$> await
`await` works on a conduit of strict `ByteString` chunks. The size/length of each `ByteString` is dependent on your network connection, so if you have a fast internet connection and your bytestrings are sufficiently separated because you're waiting for a response between each message your program might actually work as you expect it to do (with a little bit of luck). But consider a simple server/client application where the messages are not seperated by a small delay: server = runTCPServer (..) $ \ad -> appSource ad $$ Data.Conduit.List.mapM_ (liftIO . print) client = runTCPClient (..) $ \ad -> (yield "msg1" >> yield "msg2") $$ appSink ad The server will simply print out "msg1msg2" as one message, not as two separate messages. Even worse, if your network connection is bad or your chunks are getting too big for buffering, you might end up with something like: "msg1ms" "g2" `await` is not reliable in that regard because the network source is not consistent and non-deterministic. My libraray makes sure that every "yield" from the client corresponds to exactly one (not more or less) "await" at the server. There are more benefits when using my library. For example consider a client which first sends an authorization message, then a couple of hashes from different files and then maybe some timestamps on some other files. Writing that server is straight forward: client = runTCPClient (..) $ \ad -> send1 ad $$ yield authenticationMsg sendList ad $$ mapM_ yield [file1hash, file2hash, file3hash] sendList ad $$ mapM_ yield [timestamp1, timestamp2] server = runTCPServer (..) $ \ad -> (next,[authmsg]) <- receive ad $$ Data.Conduit.List.consume (next,hashes) <- receive next $$ Data.Conduit.List.consume (next,timestamps) <- receive next $$ Data.Conduit.List.consume close next Each `receive` corresponds to exactly one `send`. Without this library you have to manually check/verify/associate each message by hand in one big loop, whereas this library allows you to split your application into logical conduit "groups" which are straight forward to use. - Nils
Thanks for explanation.
However I'm still confused as if I'll use one of protocols (protobuf, json,
sereal message, binary message) all of them has
iteration interface i.e. function of type (ByteString -> Result a) that
returns
data Result = Error Text ByteString | NeedMore (ByteString -> Result a) |
HaveResult a
all of them can be used in conduits using sinkParser or same approach and
so there will be
no problem in sending and receiving messages even if they are split, and
that will be done in a logical block.
I see that there will be problems if I'll use raw data and I see that your
library solves it, am I right?
On 25 February 2013 15:53, Nils
Hi Alexander,
Am 25.02.2013 06:52, schrieb Alexander V Vershilov:
Can you describe if there any difference with latest conduit API
(yield, await) that can be used to write functions in a very similar style, but without using exeternal packages:
I have indeed written this library for the 0.5.6 API, so things might have changed a bit in the 1.0 API, but I'd be surprised if the fundamental flaws of this approach would have been fixed.
runTCPServer settings app = appSource ap $$ go =$= CL.mapM_ encode =$ appSink app
where go = forever $ do bp <- decode <$> await
`await` works on a conduit of strict `ByteString` chunks. The size/length of each `ByteString` is dependent on your network connection, so if you have a fast internet connection and your bytestrings are sufficiently separated because you're waiting for a response between each message your program might actually work as you expect it to do (with a little bit of luck). But consider a simple server/client application where the messages are not seperated by a small delay:
server = runTCPServer (..) $ \ad -> appSource ad $$ Data.Conduit.List.mapM_ (liftIO . print) client = runTCPClient (..) $ \ad -> (yield "msg1" >> yield "msg2") $$ appSink ad
The server will simply print out "msg1msg2" as one message, not as two separate messages. Even worse, if your network connection is bad or your chunks are getting too big for buffering, you might end up with something like:
"msg1ms" "g2"
`await` is not reliable in that regard because the network source is not consistent and non-deterministic. My libraray makes sure that every "yield" from the client corresponds to exactly one (not more or less) "await" at the server.
There are more benefits when using my library. For example consider a client which first sends an authorization message, then a couple of hashes from different files and then maybe some timestamps on some other files. Writing that server is straight forward:
client = runTCPClient (..) $ \ad -> send1 ad $$ yield authenticationMsg sendList ad $$ mapM_ yield [file1hash, file2hash, file3hash] sendList ad $$ mapM_ yield [timestamp1, timestamp2]
server = runTCPServer (..) $ \ad -> (next,[authmsg]) <- receive ad $$ Data.Conduit.List.consume (next,hashes) <- receive next $$ Data.Conduit.List.consume (next,timestamps) <- receive next $$ Data.Conduit.List.consume close next
Each `receive` corresponds to exactly one `send`. Without this library you have to manually check/verify/associate each message by hand in one big loop, whereas this library allows you to split your application into logical conduit "groups" which are straight forward to use.
- Nils
-- Alexander
Of course you don't have to use my library if your library already solves the same problem. Am 25.02.2013 13:10, schrieb Alexander V Vershilov:
data Result = Error Text ByteString | NeedMore (ByteString -> Result a) | HaveResult a
I still see an issue with this if you're receiving multiple messages in one `ByteString` chunk. A type like `HaveResult a ByteString` which also returns the leftover `ByteString` after the first result has been found could solve this, but you'd still have to re-yield this bytestring by hand and you'll possible end up with more calls to "decode" than necessary. Of course, with json you could avoid this problem by wrapping all messages into a json-list and sending this list as one big message. But that way you lose almost all advantages of a conduit - you cannot map over those list iems, you cannot process each item one by one, you cannot fold or filter them. When using this approach I don't really see the advantage over using the plain old "connectTo/hGetContents" method? Anyway, my library is independent of the "bytestring encoding" you want to use with it. I think it's the most general solution and avoids having to re-implement the same solution for each and every "protocol design" over and over again.
One question - and sorry that I didn't get a chance to try some examples myself - but can bi-directional conversations be carried between server/client in interleaving fashion using your library? Something like (simplified): myClient = receive >>= send . compute I have previously accomplished this using TBMChan and cereal-conduit to achieve the message delineation, but your library would certainly remove a layer of complexity if interleaved bi-directional talk is possible. - Ozgun On Monday, February 25, 2013 at 8:09 AM, Nils wrote:
Of course you don't have to use my library if your library already solves the same problem.
Am 25.02.2013 13:10, schrieb Alexander V Vershilov:
data Result = Error Text ByteString | NeedMore (ByteString -> Result a) | HaveResult a
I still see an issue with this if you're receiving multiple messages in one `ByteString` chunk. A type like `HaveResult a ByteString` which also returns the leftover `ByteString` after the first result has been found could solve this, but you'd still have to re-yield this bytestring by hand and you'll possible end up with more calls to "decode" than necessary.
Of course, with json you could avoid this problem by wrapping all messages into a json-list and sending this list as one big message. But that way you lose almost all advantages of a conduit - you cannot map over those list iems, you cannot process each item one by one, you cannot fold or filter them. When using this approach I don't really see the advantage over using the plain old "connectTo/hGetContents" method?
Anyway, my library is independent of the "bytestring encoding" you want to use with it. I think it's the most general solution and avoids having to re-implement the same solution for each and every "protocol design" over and over again.
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org (mailto:Haskell-Cafe@haskell.org) http://www.haskell.org/mailman/listinfo/haskell-cafe
participants (3)
-
Alexander V Vershilov -
Nils -
Ozgun Ataman