On Fri, Apr 29, 2011 at 6:32 AM, John Lato <jwlato@gmail.com> wrote:
> If you do this, the user needs to take care to order the iteratees so that
> the last iteratee has small leftovers. Consider:
>
> manyToOne [consumeALot, return ()]
>
> In this case, the entire stream consumed by the first iteratee will need to
> be retained and passed on by manyToOne. In many cases, the user may not
> know how much each iteratee will consume, which can make these semantics
> problematic.
>
> Iteratee has 'enumPair', (renamed 'zip' in HEAD) which returns the leftovers
> from whichever iteratee consumes more. This avoids the problem of retaining
> extra data, and seems simpler to reason about. Although if you really need
> to consume a predictable amount of data, the safest is probably to run the
> whole thing in a 'take'.

My motivation is: in general it is difficult (impossible?) to choose
the iteratee that consumed more data because you don't know what the
data is. For example, if you give 'Chunks [a,b]' to two iteratees and
one of them returns 'Chunks [c]' and the other one returns 'Chunks
[d]', which one consumed more data? The answer is that it depends on
the types. If they are Ints, both consumed the same, if they are
ByteStrings, you would need to check if one is prefix of the other.
What if one returns 'Chunks [c]' and the other one returns 'Chunks
[d,e]'? If they are ByteStrings, should we compare 'c' against 'd ++
e'?

This situation results from the implementation in the enumerator package. In iteratee it doesn't arise with well-behaved* iteratees, because only one chunk is ever processed at a time. It's only necessary to check the length of the returned chunks to see which consumed more data.

By well-behaved, I mean that the chunk returned by an iteratee must be a tail of the provided input. In other words, it returns only unconsumed data from the stream and doesn't alter the stream. At least in the iteratee package, an iteratee which violates this rule is likely to result in undefined behavior (in general, not just this function).

So I thought it would be easier to program with an API that is
predictable and immune to changes in block sizes. If you don't want
leftovers, just use 'manyToOne [..., dropWhile (const True)]', which
guarantees that you won't leak.

Iteratees should be immune to changes in block sizes anyway, although it's been a while since I looked at the enumerator implementation so it could be different.

If you use 'manyToOne [..., dropWhile (const True)]', when does it terminate?

John