[Conduit] weird action of leftover.

older
[haskell.org Summer of Code 2013]...

Magicloud Magiclouds

8 Apr 2013 8 Apr '13

5:44 a.m.

Say I have code like below. If I comment the leftover in main, I got (Just "L1\n", Just "L2\n", Just "L3\n", Just "L4\n"). But if I did not comment the leftover, then I got (Just "L1\n", Just "L1\n", Just "", Just "L2\n"). Why is not it (Just "L1\n", Just "L1\n", Just "L2\n", Just "L3\n")? takeLine :: (Monad m) => Consumer ByteString m (Maybe ByteString) takeLine = do mBS <- await case mBS of Nothing -> return Nothing Just bs -> case DBS.elemIndex _lf bs of Nothing -> return $ Just bs Just i -> do let (l, ls) = DBS.splitAt (i + 1) bs leftover ls return $ Just l main = do m <- runResourceT $ sourceFile "test.simple" $$ (do a <- takeLine leftover $ fromJust a b <- takeLine c <- takeLine d <- takeLine return (a, b, c, d)) print m -- 竹密岂妨流水过山高哪阻野云飞 And for G+, please use magiclouds#gmail.com.

Attachments:

attachment.html (text/html — 1.5 KB)

Show replies by date

Michael Snoyman

9 Apr 9 Apr

4:12 a.m.

It's a bug in your implementation of takeLine I believe. It doesn't take into account that lines can span multiple chunks. When you call takeLine the first time, you get "L1\n". leftover puts a chunk with exactly those contents back. When you call takeLine the second time, it gets the chunk "L1\n", and your splitAt gives you back "L1\n" and "". The "" is then leftover, and the next call to takeLine gets it. Your takeLine needs to include logic saying "there's no newline in this chunk at all, let's get the next chunk and try that." You can look at the source to lines[1] for an example of the concept. Michael [1] http://haddocks.fpcomplete.com/fp/7.4.2/20130313-1/conduit/src/Data-Conduit-... On Mon, Apr 8, 2013 at 8:44 AM, Magicloud Magiclouds < magicloud.magiclouds@gmail.com> wrote:

...

Say I have code like below. If I comment the leftover in main, I got (Just "L1\n", Just "L2\n", Just "L3\n", Just "L4\n"). But if I did not comment the leftover, then I got (Just "L1\n", Just "L1\n", Just "", Just "L2\n"). Why is not it (Just "L1\n", Just "L1\n", Just "L2\n", Just "L3\n")?

takeLine :: (Monad m) => Consumer ByteString m (Maybe ByteString) takeLine = do mBS <- await case mBS of Nothing -> return Nothing Just bs -> case DBS.elemIndex _lf bs of Nothing -> return $ Just bs Just i -> do let (l, ls) = DBS.splitAt (i + 1) bs leftover ls return $ Just l

main = do m <- runResourceT $ sourceFile "test.simple" $$ (do a <- takeLine leftover $ fromJust a b <- takeLine c <- takeLine d <- takeLine return (a, b, c, d)) print m

-- 竹密岂妨流水过山高哪阻野云飞

And for G+, please use magiclouds#gmail.com.

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Magicloud Magiclouds

4:48 a.m.

Thank you for the reply. I've learnt the code of "lines". So it is because how ByteString works, that the conduit is not a stream of bytes, but chunks, right? On Tue, Apr 9, 2013 at 12:12 PM, Michael Snoyman wrote:

...

It's a bug in your implementation of takeLine I believe. It doesn't take into account that lines can span multiple chunks. When you call takeLine the first time, you get "L1\n". leftover puts a chunk with exactly those contents back. When you call takeLine the second time, it gets the chunk "L1\n", and your splitAt gives you back "L1\n" and "". The "" is then leftover, and the next call to takeLine gets it.

Your takeLine needs to include logic saying "there's no newline in this chunk at all, let's get the next chunk and try that." You can look at the source to lines[1] for an example of the concept.

Michael

[1] http://haddocks.fpcomplete.com/fp/7.4.2/20130313-1/conduit/src/Data-Conduit-...

On Mon, Apr 8, 2013 at 8:44 AM, Magicloud Magiclouds < magicloud.magiclouds@gmail.com> wrote:

...
Say I have code like below. If I comment the leftover in main, I got (Just "L1\n", Just "L2\n", Just "L3\n", Just "L4\n"). But if I did not comment the leftover, then I got (Just "L1\n", Just "L1\n", Just "", Just "L2\n"). Why is not it (Just "L1\n", Just "L1\n", Just "L2\n", Just "L3\n")?

takeLine :: (Monad m) => Consumer ByteString m (Maybe ByteString) takeLine = do mBS <- await case mBS of Nothing -> return Nothing Just bs -> case DBS.elemIndex _lf bs of Nothing -> return $ Just bs Just i -> do let (l, ls) = DBS.splitAt (i + 1) bs leftover ls return $ Just l

main = do m <- runResourceT $ sourceFile "test.simple" $$ (do a <- takeLine leftover $ fromJust a b <- takeLine c <- takeLine d <- takeLine return (a, b, c, d)) print m

-- 竹密岂妨流水过山高哪阻野云飞

And for G+, please use magiclouds#gmail.com.

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

-- 竹密岂妨流水过山高哪阻野云飞 And for G+, please use magiclouds#gmail.com.

Michael Snoyman

4:53 a.m.

Yes, that's a fair explanation. On Tue, Apr 9, 2013 at 7:48 AM, Magicloud Magiclouds < magicloud.magiclouds@gmail.com> wrote:

...

Thank you for the reply. I've learnt the code of "lines". So it is because how ByteString works, that the conduit is not a stream of bytes, but chunks, right?

On Tue, Apr 9, 2013 at 12:12 PM, Michael Snoyman wrote:

...
It's a bug in your implementation of takeLine I believe. It doesn't take into account that lines can span multiple chunks. When you call takeLine the first time, you get "L1\n". leftover puts a chunk with exactly those contents back. When you call takeLine the second time, it gets the chunk "L1\n", and your splitAt gives you back "L1\n" and "". The "" is then leftover, and the next call to takeLine gets it.

Your takeLine needs to include logic saying "there's no newline in this chunk at all, let's get the next chunk and try that." You can look at the source to lines[1] for an example of the concept.

Michael

[1] http://haddocks.fpcomplete.com/fp/7.4.2/20130313-1/conduit/src/Data-Conduit-...

On Mon, Apr 8, 2013 at 8:44 AM, Magicloud Magiclouds < magicloud.magiclouds@gmail.com> wrote:

...
Say I have code like below. If I comment the leftover in main, I got (Just "L1\n", Just "L2\n", Just "L3\n", Just "L4\n"). But if I did not comment the leftover, then I got (Just "L1\n", Just "L1\n", Just "", Just "L2\n"). Why is not it (Just "L1\n", Just "L1\n", Just "L2\n", Just "L3\n")?

takeLine :: (Monad m) => Consumer ByteString m (Maybe ByteString) takeLine = do mBS <- await case mBS of Nothing -> return Nothing Just bs -> case DBS.elemIndex _lf bs of Nothing -> return $ Just bs Just i -> do let (l, ls) = DBS.splitAt (i + 1) bs leftover ls return $ Just l

main = do m <- runResourceT $ sourceFile "test.simple" $$ (do a <- takeLine leftover $ fromJust a b <- takeLine c <- takeLine d <- takeLine return (a, b, c, d)) print m

-- 竹密岂妨流水过山高哪阻野云飞

And for G+, please use magiclouds#gmail.com.

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

-- 竹密岂妨流水过山高哪阻野云飞

And for G+, please use magiclouds#gmail.com.

4474

Age (days ago)

4475

Last active (days ago)

List overview

Download

3 comments

2 participants

participants (2)

Magicloud Magiclouds
Michael Snoyman