Stream processing
Hello everybody, I'm trying to formulate the stream processing problem, which doesn't seem to be solved fully by the currently existing patterns. I'm experimenting with a new idea, but I want to make sure that I don't miss any defining features of the problem, so here is my list. A stream processing abstraction should: * have a categorically proven design (solved by iteratees, pipes), * be composable (solved by all of them), * be reasonably easy to understand and work with (solved by conduit, pipes), * support leftovers (solved by conduit and to some degree by iteratees), * be reliable in the presence of async exceptions (solved by conduit, pipes-safe), * hold on to resources only as long as necessary (solved by conduit and to some degree by pipes-safe), * ideally also allow upstream communication (solved by pipes and to some degree by conduit). * be fast (solved by all of them). Anything else you would put in that list? Greets, Ertugrul -- Not to be or to be and (not to be or to be and (not to be or to be and (not to be or to be and ... that is the list monad.
It's quite old at this point, but you may be interested in reading the
initial motivations for creating conduit when the iteratee pattern (and
enumerator library in particular) already existed:
https://github.com/snoyberg/conduit/blob/master/README.md#general-goal
I would say the only real component missing from your list is being able to
structure significantly more complicated control flows, such as the use
case of combining a web server and web client into a web proxy. That was
probably the example which finally pushed me to start thinking seriously
about an enumerator replacement. In conduit, this use case is addressed by
connect-and-resume, which essentially allows you to escape the inversion of
control normally introduced by the conduit pattern.
On Fri, May 10, 2013 at 5:56 PM, Ertugrul Söylemez
Hello everybody,
I'm trying to formulate the stream processing problem, which doesn't seem to be solved fully by the currently existing patterns. I'm experimenting with a new idea, but I want to make sure that I don't miss any defining features of the problem, so here is my list. A stream processing abstraction should:
* have a categorically proven design (solved by iteratees, pipes),
* be composable (solved by all of them),
* be reasonably easy to understand and work with (solved by conduit, pipes),
* support leftovers (solved by conduit and to some degree by iteratees),
* be reliable in the presence of async exceptions (solved by conduit, pipes-safe),
* hold on to resources only as long as necessary (solved by conduit and to some degree by pipes-safe),
* ideally also allow upstream communication (solved by pipes and to some degree by conduit).
* be fast (solved by all of them).
Anything else you would put in that list?
Greets, Ertugrul
-- Not to be or to be and (not to be or to be and (not to be or to be and (not to be or to be and ... that is the list monad.
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
On 05/11/2013 11:04 AM, Michael Snoyman wrote:
It's quite old at this point, but you may be interested in reading the initial motivations for creating conduit when the iteratee pattern (and enumerator library in particular) already existed:
https://github.com/snoyberg/conduit/blob/master/README.md#general-goal
I would say the only real component missing from your list is being able to structure significantly more complicated control flows, such as the use case of combining a web server and web client into a web proxy. That was probably the example which finally pushed me to start thinking seriously about an enumerator replacement. In conduit, this use case is addressed by connect-and-resume, which essentially allows you to escape the inversion of control normally introduced by the conduit pattern.
Speaking of escaping inversion of control, I'm going to take a moment to shamelessly plug this post of mine showing that you can escape the pipe monad using a trick uncannily similar to `io-streams`: http://www.haskellforall.com/2013/04/pipes-and-io-streams.html It's not equivalent to `conduit`'s connect and resume, though, which is very different.
Michael Snoyman
In conduit, this use case is addressed by connect-and-resume, which essentially allows you to escape the inversion of control normally introduced by the conduit pattern.
Indeed, but this isn't even an extra feature in my case. It follows
naturally from the automaton design. Thanks for pointing it out,
though. I need to think about how to expose this feature.
Greets,
Ertugrul
--
Key-ID: E5DD8D11 "Ertugrul Soeylemez
participants (3)
-
Ertugrul Söylemez -
Gabriel Gonzalez -
Michael Snoyman