I'd mentioned in my previous email that I was concerned that "consume" would traverse the entire list for each new record pulled from the database. I'll try to start using the combinators instead; do get started with, it was easier to just use the raw datatypes to more easily see what was happening.

Would this version of consume perhaps be better:

consume :: Monad m => Iteratee e a m [a]
consume = liftI $ step id where
    step acc chunk =
        case chunk of
            Chunks [] -> Continue $ returnI . step acc
            Chunks xs -> Continue $ returnI . (step $ acc . (xs ++))
            EOF -> Yield (acc []) EOF

Michael

On Thu, Aug 19, 2010 at 8:40 AM, John Millikin <jmillikin@gmail.com> wrote:
I'm glad to hear you found it useful!

For implementing selectList, have you considered using the "consume"
iteratee? I suspect it would simplify your code to something like:

selectList a b c d = do
   res <- run $ select a b c d ==<< consume
   case res of
       Left e -> error e
       Right x -> return x

You might also want to look at the iteratee combinators (returnI,
yield, continue), which would reduce some boilerplate, eg:

"Iteratee $ return $ Continue k" becomes "continue k"
"Iteratee $ return $ Yield x EOF" becomes "yield x EOF"

On Wed, Aug 18, 2010 at 22:24, Michael Snoyman <michael@snoyman.com> wrote:
> John,
> This package looks very promising. I used iteratee for the yaml package, but
> I had many of the concerns that you have mentioned below. Version 0.2 of
> persistent is going to have some form of an enumerator interface for getting
> the results of a query, and I eventually decided that iteratee was
> introducing too much complexity to be a good candidate. However, I was able
> to port the package[1] over to enumerator in about half an hour; I
> especially benefited from your example applications.
>
> The only concern that I had was the possible inefficiency of representing
> all chunks as a list. In the case of persistent, the enumerator will
> *always* generate a one-lengthed list, and the most common operation is
> selectList, which returns all results as a list. If I used your consume
> function, I believe there would be a *lot* of list traversals. Instead,
> selectList[2] uses ([a] -> [a]) for building up the result internally. I
> haven't really thought the issue through fully, so I can recommend anything
> better. Perhaps more importantly, the simplification introduced by just
> dealing with lists is well received.
> Keep up the good work, I look forward to seeing more about enumerator.
> Michael
> [1] http://github.com/snoyberg/persistent/tree/enumerator
> [2] http://github.com/snoyberg/persistent/blob/enumerator/Database/Persist/Base.hs#L322