reading file content with conduit
Hello, I have a strange behaviour with my code when reading data from a file saved by sinkFile. I only see the first record of the file, although the file seems to contain several records. Any hints ? -- built in a sandbox with GHC 8.2.2, base 4.10.1.0, binary 0.8.5.1, bytestring 0.8.10.2, conduit 1.3.1 {-# LANGUAGE DeriveGeneric #-} import qualified Data.ByteString as DB import qualified Data.ByteString.Lazy as DBL import qualified Data.Binary as DBI import GHC.Generics (Generic) import Conduit import Data.Int (Int64) data MySubRec = MySubRec { sr1 :: Float, sr2 :: Float } deriving (Generic, Show) data MyRec = MyRec { r1 :: Int64, r2 :: String, r3 :: [MySubRec], r4 :: [MySubRec] } deriving (Generic, Show) instance DBI.Binary MySubRec instance DBI.Binary MyRec es = MySubRec { sr1 =1.0, sr2 =1000.5 } myList = repeat es e1 = MyRec{ r1=1, r2="e1", r3=take 2 myList, r4=take 1 myList} e2 = MyRec{ r1=2, r2="e2", r3=take 2 myList, r4=take 1 myList} myData = concat $ repeat [e1,e2] dataToBs :: Monad m => ConduitT MyRec DB.ByteString m () dataToBs = do d <- await case d of Just bs -> do yield $ DBL.toStrict $ DBI.encode bs dataToBs _ -> return () bsToData :: Monad m => ConduitT DB.ByteString MyRec m () bsToData = do d <- await case d of Just bs -> do yield $ DBI.decode $ DBL.fromStrict bs bsToData _ -> return () main = do runConduitRes $ yieldMany (take 10 myData) .| dataToBs .| sinkFile "/tmp/res.bin" runConduitRes $ sourceFile "/tmp/res.bin" .| bsToData .| mapM_C (liftIO . putStrLn . show)
Hello Fabien, in your example, the problem is that sourceFile gives you a bytestring chunk that contains more than one record and decoding function does not return a list of values, instead return only one. So you need to pass the leftover data to the next decoding round. You may consider using https://hackage.haskell.org/package/binary-conduit package that does that (see https://hackage.haskell.org/package/binary-conduit-1.3.1/docs/src/Data.Condu...) -- Best regards, Alexander. On Fri, 2 Nov 2018 at 15:09, Fabien R <theedge456@free.fr> wrote:
Hello, I have a strange behaviour with my code when reading data from a file saved by sinkFile. I only see the first record of the file, although the file seems to contain several records.
Any hints ?
-- built in a sandbox with GHC 8.2.2, base 4.10.1.0, binary 0.8.5.1, bytestring 0.8.10.2, conduit 1.3.1 {-# LANGUAGE DeriveGeneric #-} import qualified Data.ByteString as DB import qualified Data.ByteString.Lazy as DBL import qualified Data.Binary as DBI import GHC.Generics (Generic) import Conduit import Data.Int (Int64)
data MySubRec = MySubRec { sr1 :: Float, sr2 :: Float } deriving (Generic, Show)
data MyRec = MyRec { r1 :: Int64, r2 :: String, r3 :: [MySubRec], r4 :: [MySubRec] } deriving (Generic, Show)
instance DBI.Binary MySubRec instance DBI.Binary MyRec
es = MySubRec { sr1 =1.0, sr2 =1000.5 } myList = repeat es e1 = MyRec{ r1=1, r2="e1", r3=take 2 myList, r4=take 1 myList} e2 = MyRec{ r1=2, r2="e2", r3=take 2 myList, r4=take 1 myList} myData = concat $ repeat [e1,e2]
dataToBs :: Monad m => ConduitT MyRec DB.ByteString m () dataToBs = do d <- await case d of Just bs -> do yield $ DBL.toStrict $ DBI.encode bs dataToBs _ -> return () bsToData :: Monad m => ConduitT DB.ByteString MyRec m () bsToData = do d <- await case d of Just bs -> do yield $ DBI.decode $ DBL.fromStrict bs bsToData _ -> return () main = do runConduitRes $ yieldMany (take 10 myData) .| dataToBs .| sinkFile "/tmp/res.bin" runConduitRes $ sourceFile "/tmp/res.bin" .| bsToData .| mapM_C (liftIO . putStrLn . show) _______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.
-- Alexander
Thanks Alexander, The package fixed the problem. I thought that, since a conduit is driven by downstream, if bsToData requested a record, sourceFile would only send the corresponding Bytestrings. -- Fabien
Hello Fabien, your expectations are correct, but in order to make this really happen your consumer function should be aware of the conduit pipeline in order to consume only the required amount of data. That may happen automatically in two cases: 1. your function consumes an entire chunk 2. your function can work in an iterative way and can return unprocessed data or continuation that may consume more data (the case of iterative API in binary) A nice example of the function that is related to your use case and aware of the conduit pipeline: https://hackage.haskell.org/package/conduit-extra-1.3.0/docs/Data-Conduit-At... If you pass a parser in sinkParser, it will consume only the required amount of data. -- Alexander On Sun, 4 Nov 2018 at 11:48, Fabien R <theedge456@free.fr> wrote:
Thanks Alexander, The package fixed the problem. I thought that, since a conduit is driven by downstream, if bsToData requested a record, sourceFile would only send the corresponding Bytestrings.
-- Fabien _______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.
-- Alexander
participants (2)
-
Alexander V Vershilov -
Fabien R