On Fri, 2010-03-26 at 13:29 +0000, Duncan Coutts wrote:
On Fri, 2010-03-26 at 02:11 +0000, Maciej Piechotka wrote:
Hello,
I have written patch (attached) which introduce new API for zlib - personally I wrote it to have easy implementation of compression/decompression in iteratee (example file - no yet working -attached).
Earlier version of patch I tried to sent to maintainer of zlib. However I received no response (possibly I get into spam folder by helpful spamassassin or something like that). I definitely don't want to fork this library - especially that it is in haskell platform.
Ivan is right :-)
I understand. However given the last update of zlib it could be that the address is no longer valid.
I've discussed iterator approaches with zlib before. Tibbe and I discussed it in the context of web servers that want precise control over outputting blocks.
I'm not happy however that something that is essentially pure (compression/decompression) should have to have be in IO just to fit the iterator pattern. That seems wrong. I could tolerate ST however.
Well. Iteration can be made pure again iff: 1. It will be possible to copy the z_stream. Currently it is not possible with zlib. 2. It is deal with great care in iteration. It is possible to use safely unsafePerformIO and seq. Iteratee can be pure as it have no copy of the context. Existsing compress/decompress functions as well are 'safe' for the same reason. 3. It is implemented in pure haskell.
Secondly we cannot have the zlib package depend on the iteratee package. Though we can expose interfaces that allow you to construct the iterator stuff.
Hmm. Have you seen the patch? It does not change any dependencies. It just adds 6 functions and 3 data structures for API (+ helper functions). It DOES NOT remove any currently existing function (or even change them). it is just additional interface defined. It could be a separate package if it hasn't use Stream. I have never imagined to add dependency of iteratee.
I think that it is possible for you to construct the impure interface you want on top of the existing pure interface, using threads and MVars. It is a general pattern, to transform a lazy [a] -> [a] function into an imperative "push block / pull block" interface. You connect up the lazy function to an input channel and an output MVar (running in a new thread). Then externally you push blocks into the input channel and wait for blocks on the output channel. If benchmarks prove that to have too high an overhead then we might consider doing something lower level.
1. That was my first approach. It was so horribly complicated that I gave up. Especially if you wanted any level of error handling. 2. To fully handle the iterator API the decompress would have to be: ... -> (L.ByteString, L.ByteString) Where first is decompressed string and second a rest of input. 3. If anyone is going to write such function - I'm fine. I just chosen what I believe it is a simpler and more correct approach.
Another alternative that I would be satisfied with would be a lower level interface using the ST monad. That should allow the existing pure interface to be implemented safely on top, and hopefully would allow the kind of impure interface you're after.
As the function have side effects I have chosen IO. The patch was intended as a low-level interface for specific applications. If application ensure that computation is done in proper way it may convert it into ST/pure easily. I think that adding unsafePerformIO/unsafeIOToST misses the point. If computation is pure it can probably use compress/decompress. If not then IO/ST monad does not impose additional restrictions
Duncan
Regards PS. I would like to underline that it DOES NOT add dependency on iteratee or any other package and it DOES NOT intend to replace/change an existing API. It is a simply new even-lower-level API for special situation like iteratee (web servers etc.).