Help with enumerator pipeline

31 Aug 2011

      I've just recently started learning how to use the enumerator library. I'm
designing a utility that parses a file where each line is a JSON object.
I've designed it to look like:

source enumerator (enumHandle pretty much) ->
chunk by lines (enumeratee) ->
parse a line into an Object (enumeratee) ->
filter objects based on a criteria (enumeratee) ->
limit some keys from each object (enumeratee) ->
encode the object into a lazy bytestring (enumeratee) ->
output the file to stdout (iteratee)

I'm having difficulties with the types, particularly composing the
enumeratees in the middle. Someone in #haskell said that's a good case for
...
=> from Control.Monad but that seems to not like it if an enumeratee
changes the type between the input and output.
Here's the relevant types:

type Object = Map Text Value

pipeline :: MonadIO m => (Enumerator ByteString IO ()) -> [Text] -> [Filter]
-> Iteratee a m ()
pipeline s rfs fs = s $$ splitLines    >=>
                         parseLine            >=>
                         (filterObjects fs)   >=>
                         (restrictFields rfs) >=>
                         encoder              $$
                         output
splitLines :: Monad m => Enumeratee ByteString ByteString m b
parseLine :: Monad m => Enumeratee ByteString Object m b
filterObjects :: Monad m => [Filter] -> Enumeratee Object Object m b
restrictObjects :: Monad m => [Text] -> Enumeratee Object Object m b
encoder :: Monad m => Enumeratee Object LBS.ByteString m b
output :: MonadIO m => Iteratee LBS.ByteString m ()

I'm using >=> because I'd prefer to compose them left-right for readability

Here's the error I get

 Couldn't match expected type `ByteString'
                with actual type `M.Map Text Value'
    Expected type: Data.Enumerator.Step ByteString m b1
                   -> Iteratee ByteString m b2
      Actual type: Enumeratee ByteString Object m0 b0
    In the first argument of `(>=>)', namely `parseLine'
    In the second argument of `(>=>)', namely
      `parseLine
     >=> (filterObjects fs) >=> (restrictFields rfs) >=> encoder'

Any ideas?

-- 
Michael Xavier
http://www.michaelxavier.net
LinkedIn http://www.linkedin.com/pub/michael-xavier/13/b02/a26

Michael Xavier

David McBride

Felipe Almeida Lessa

Michael Xavier

Ertugrul Soeylemez

Michael Xavier

Michael Snoyman

Michael Xavier

tags

participants (5)