Hi all, I've just released stm-conduit [1] on Hackage. This package introduces conduits to the wonderful world of concurrency. My package solves the common problem of constant bottleneck switching loaders have. This is when, for example, we stream XML from the disk and then parse the XML in one conduit pipeline. While it streams a file from the disk, the process is clearly IO bound, and while it parses the XML, the process is CPU bound. By putting each task on its own thread, the disk IO doesn't need to wait for the CPU to parse a document before loading the next file. By using stm-based conduits, we have full resource utilization. The way it does this is by creating a source and sink for TChans, letting us stream data between conduits and channels. There are more examples in the docs. Check it out! Regards, - clark [1] http://hackage.haskell.org/package/stm-conduit
Hello. Thu, Feb 09, 2012 at 01:32:41PM -0500, Clark Gaebel wrote
Hi all,
I've just released stm-conduit [1] on Hackage. This package introduces conduits to the wonderful world of concurrency.
My package solves the common problem of constant bottleneck switching loaders have. This is when, for example, we stream XML from the disk and then parse the XML in one conduit pipeline. While it streams a file from the disk, the process is clearly IO bound, and while it parses the XML, the process is CPU bound. By putting each task on its own thread, the disk IO doesn't need to wait for the CPU to parse a document before loading the next file. By using stm-based conduits, we have full resource utilization.
The way it does this is by creating a source and sink for TChans, letting us stream data between conduits and channels. There are more examples in the docs.
Check it out!
Regards, - clark
A day ago I've make analogical library in utils for my project, code was 90% same. Thanks for putting such a library on hackage -- Best regards, Alexander.
Happy to help! I'm new to this whole "package on hackage" thing, so any feedback would be great. - clark On Thu, Feb 9, 2012 at 2:19 PM, Alexander V Vershilov < alexander.vershilov@gmail.com> wrote:
Hello.
Hi all,
I've just released stm-conduit [1] on Hackage. This package introduces conduits to the wonderful world of concurrency.
My package solves the common problem of constant bottleneck switching loaders have. This is when, for example, we stream XML from the disk and then
XML in one conduit pipeline. While it streams a file from the disk, the
is clearly IO bound, and while it parses the XML, the process is CPU bound. By putting each task on its own thread, the disk IO doesn't need to wait for the CPU to parse a document before loading the next file. By using stm-based conduits, we have full resource utilization.
The way it does this is by creating a source and sink for TChans, letting us stream data between conduits and channels. There are more examples in
Thu, Feb 09, 2012 at 01:32:41PM -0500, Clark Gaebel wrote parse the process the docs.
Check it out!
Regards, - clark
A day ago I've make analogical library in utils for my project, code was 90% same. Thanks for putting such a library on hackage
-- Best regards, Alexander.
Your package uses TMChans which AFAIK are unbounded. That means that if the writer is faster than the reader, then everything will be kept into memory. This means that using TMChans you may no longer say that your program uses a constant amount of memory. Actually, you lose a lot of your space reasoning since, being concurrent processes, you can't guarantee almost anything wrt. progress of the reader. This doesn't mean that your package is broken, this means that it has a caveat that should be stated on the docs. Congrats on your Hackage debut! Cheers! =) -- Felipe.
Actually, that is a moderately fatal flaw. I just uploaded 0.2.2 which
addresses this by highly recommending using bounded channels, as well as
adding sources/sinks for them.
Thanks for catching that!
- clark
On Thu, Feb 9, 2012 at 2:29 PM, Felipe Almeida Lessa wrote: Your package uses TMChans which AFAIK are unbounded. That means that
if the writer is faster than the reader, then everything will be kept
into memory. This means that using TMChans you may no longer say that
your program uses a constant amount of memory. Actually, you lose a
lot of your space reasoning since, being concurrent processes, you
can't guarantee almost anything wrt. progress of the reader. This doesn't mean that your package is broken, this means that it has
a caveat that should be stated on the docs. Congrats on your Hackage debut! Cheers! =) --
Felipe.
On 2/9/12 2:29 PM, Felipe Almeida Lessa wrote:
Your package uses TMChans which AFAIK are unbounded. That means that if the writer is faster than the reader, then everything will be kept into memory. This means that using TMChans you may no longer say that your program uses a constant amount of memory. Actually, you lose a lot of your space reasoning since, being concurrent processes, you can't guarantee almost anything wrt. progress of the reader.
Of course, you're free to use TBMChans instead, which are bounded :) -- Live well, ~wren
I added that after reading his feedback, and seeing the flaw in using
TMChans.
On Sun, Feb 12, 2012 at 1:47 AM, wren ng thornton
On 2/9/12 2:29 PM, Felipe Almeida Lessa wrote:
Your package uses TMChans which AFAIK are unbounded. That means that if the writer is faster than the reader, then everything will be kept into memory. This means that using TMChans you may no longer say that your program uses a constant amount of memory. Actually, you lose a lot of your space reasoning since, being concurrent processes, you can't guarantee almost anything wrt. progress of the reader.
Of course, you're free to use TBMChans instead, which are bounded :)
-- Live well, ~wren
______________________________**_________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/**mailman/listinfo/haskell-cafehttp://www.haskell.org/mailman/listinfo/haskell-cafe
participants (4)
-
Alexander V Vershilov -
Clark Gaebel -
Felipe Almeida Lessa -
wren ng thornton