On Sat, Mar 10, 2012 at 4:21 AM, Mario Blažević
I like your design, it seems to strike a good balance between elegance and practicality. The only thing missing for the latter is a deeper support for chunking. Of course, that would probably destroy some of the elegance [1]. I don't think that problem has been solved in any of the enumerator/iteratee/pipe/wire/conduit libraries so far.
Chunking is supported but not by primitive constructs. The way you implement chunked streams is to simply use some form of "container" representing a chunk as your input/output type. Of course, that means that the abstraction is now operating at the level of chunks instead of elements, which may be inconvenient, but I doubt that there exists a way to "lift" element operations to chunks in an efficient and general way. Another issue is how to deal with unconsumed input. For that, there is a ChunkPipe type (in pipes-extra) with a specialized monad instance that threads unconsumed input along. You can see an example of ChunkPipe in action in this prototype http server by Jeremy Shaw: http://src.seereason.com/pipes-http-2/pipes-http-2/. Note that this is based on a old version of pipes-core, however.
Did you consider adding some stream-splitting and merging pipes, like those in the SCC package [2] or those described in the last Monad.Reader issue [3]? Your arrow-like combinators seem well thought out, but they don't go very far.
I'm not sure why you say that they don't go very far. I looked at Splitter and Join in Monad.Reader 19, and they seem equivalent to 'splitP' and 'joinP' in pipes-core. There shouldn't be any problem implementing all the other combinators there in terms of monoidal primitives (e.g. 'not' is just 'swap'). There is also a 'zip' combinator in pipes-extra, that synchronizes two Producers on their respective outputs. Can you elaborate on use cases that seem to be missing? BR, Paolo