Re: [Haskell-cafe] ANNOUNCE: pipes-core 0.0.1

On 12-03-10 05:16 AM, Paolo Capriotti wrote:

On Sat, Mar 10, 2012 at 4:21 AM, Mario Blažević wrote:

...

I like your design, it seems to strike a good balance between elegance and practicality. The only thing missing for the latter is a deeper support for chunking. Of course, that would probably destroy some of the elegance [1]. I don't think that problem has been solved in any of the enumerator/iteratee/pipe/wire/conduit libraries so far.

Chunking is supported but not by primitive constructs. The way you implement chunked streams is to simply use some form of "container" representing a chunk as your input/output type.

Of course, that means that the abstraction is now operating at the level of chunks instead of elements, which may be inconvenient, but I doubt that there exists a way to "lift" element operations to chunks in an efficient and general way.

Another issue is how to deal with unconsumed input. For that, there is a ChunkPipe type (in pipes-extra) with a specialized monad instance that threads unconsumed input along. You can see an example of ChunkPipe in action in this prototype http server by Jeremy Shaw: http://src.seereason.com/pipes-http-2/pipes-http-2/. Note that this is based on a old version of pipes-core, however.

The only sane way I've found to deal with chunks is to move the responsibility into the glue logic, which would be your (>+>) and (>>) combinators. The upstream argument of (>+>) could then produce chunks of any size it finds suitable, while the downstream argument would specify exactly how much input it needs without having to worry about the upstream chunk boundaries. How these requests are phrased is an open question. I've developed the incremental-parser package specifically for this purpose. Other approaches are possible, but I'm convinced that chunking should not be left to individual components. The chunk type probably shouldn't be reflected even in their types.

...

Did you consider adding some stream-splitting and merging pipes, like those in the SCC package [2] or those described in the last Monad.Reader issue [3]? Your arrow-like combinators seem well thought out, but they don't go very far.

I'm not sure why you say that they don't go very far. I looked at Splitter and Join in Monad.Reader 19, and they seem equivalent to 'splitP' and 'joinP' in pipes-core.

The main purpose of the Splitter type is to act as a conditional, sending each input item *either* into the left or into the right output, marking is as either true or false. Your splitterP sends each item to *both* left and right output. It's a tee, not a split. If your point is that the Splitter a m r type is isomorphic to Pipe a (Either a a) m r, that is true. There is a benefit to the abstraction, though. Once you introduce chunking into the picture, however, the Splitter type can be changed under the hood to send an entire chunk to its left or right output. The corresponding efficient chunked Pipe type would be Pipe [a] (Either [a] [a]) m r, which is not at all the same as Pipe [a] [Either a a] m r -- if you have to pack every single item of the input chunk into an Either value, you've lost all performance benefits of chunking. The former type is efficient but I'm not sure if it would allow you to abstract the chunking logic out of the individual components.

There shouldn't be any problem implementing all the other combinators there in terms of monoidal primitives (e.g. 'not' is just 'swap').

I agree, with the chunking reservation above. Anyway, consider adding the Boolean combinators to the library. I find them quite intuitive.