At Mon, 9 May 2011 17:55:17 +0100, John Lato wrote:

Felipe Almeida Lessa wrote:

> So, in the enumerator vs. iterIO challenge, the only big differences I see are: > >  a) iterIO has a different exception handling mechanism. >  b) iterIO can have pure iteratees that don't touch the monad. >  c) iterIO's iteratees can send control messages to ther enumerators. >  d) iterIO's enumerators are enumeratees, but enumerator's enumerators > are simpler. >  e) enumerator has fewer dependencies. >  f) enumerator uses conventional nomenclature. >  g) enumerator is Haskell 98, while iterIO needs many extensions (e.g. > MPTC and functional dependencies). >

'a' is important, but I think a lot of people underestimate the value of 'c', which is why a control system was implemented in 'iteratee'. ... it's relatively simple for an enumerator to return data to an iteratee using an IORef for example.

Would you just embed IORefs for the result into an Exception type? That's actually a pretty simple solution when you can do it. It's a bit harder for my setting, because I'm using this stuff in support of a research project that doesn't make the IO Monad available to most code. I'd like to write Inums/Enumeratees that work with both the IO Monad and my own weird monads. This is admittedly a fringe problem, so IORef is probably fine for most settings. But if there's any possible way you could do it with STRefs, that would be really cool... After further thought, though, I'm still not 100% satisfied with iterIO's control mechanism. Someone earlier in this thread pointed out that my SSL module doesn't support STARTTLS particularly conveniently. I read that and decided to go add a function to make STARTTLS really convenient. What I came up with ended up using MVars to communicate the switch from the enumerator to the iteratee and was ugly enough that I did not commit it. What you really want is the ability to send both upstream and downstream control messages. Right now, I'd say iterIO has better support for upstream control messages, while iteratee has better support for downstream messages, since iteratee can just embed an Exception in a Stream. (I'm assuming you could have something like a 'Flush' exception to cause output to be flushed by an Iteratee that was for some reason buffering some.) I'm curious how this works in practice, though. What is the convention for Enumeratees receiving Exceptions they don't know about in the Stream? Are they supposed to throw the exceptions upwards (which wouldn't help), or propagate them downwards. And how do they synchronize exceptions? Suppose you have a pipeline with an Enumeratee transcoding utf8 bytes to Chars, and another implementing text compression or something that requires buffering: ByteString +--------------+ [Char] +----------+ [Char] ----------> | UTF8-DECODER | ----------> | BUFFER | --------> +--------------+ +----------+ Now say a Stream with EOF (Just Flush) arrives at the UTF8-DECODER in the middle of a multi-byte character. Do you defer the Flush until the character is complete, or let it skip ahead to the end of the previous character and immediately send it to the next state? Or, worse, propagate it back up as an uncaught exception?

 And adding support to keep track of the stream position would be a pretty simple (and possibly desirable) change.

Can you explain how iteratee could keep track of the stream position? I'm not saying it's impossible, just that it's a challenging puzzle to make the types come out and I'd love to see the solution. Somehow you would need to pass the onCont continuation to itself to preserve it, and then type a gets in the way because it's possibly no longer the right type. In other words, you could try something like: {-# LANGUAGE Rank2Types #-} data Iteratee s m a = Iteratee (forall r. (a -> Stream s -> m r) -> OnCont s m a r -> m r) data OnCont s m a r = OnCont (OnCont s m a r -> (Stream s -> Iteratee s m a) -> Maybe SomeException -> m r) But now I have no way of using or unpacking the OnCont in an Iteratee that doesn't return type a, and in general a control handler has no idea what type the iteratee that threw the exception has--it's in fact likely a different type from whatever enclosing function is wrapped by a catch call. Even if you do solve the type problem, another problem is that you don't know how many times you need to call the continuation function before you stop getting buffered data and start actually causing IO to happen. Part of the reason iterIO doesn't have this problem is that iterIO's Chunk structure (which is vaguely equivalent to iteratee's Stream) is a Monoid, so it's really easy to save up multiple chunks of residual and "ungotten" data. Every Iter is passed all buffered data of its input type in its entirety (and the inner pipeline stages can actually un-transcode data to make this true across data types). But that's also what makes downstream control messages are harder, because there's no way to represent exceptions at particular points in the input stream, just an EOF marker at the very end.

I like the MonadTrans implementation a lot...

Thanks, David