Re: [Haskell-cafe] Conduit and pipelined protocol processing using a threadpool

28 Nov 2012

      On Wed, 2012-11-28 at 09:17 +0200, Michael Snoyman wrote:
...
On Tue, Nov 27, 2012 at 7:25 PM, Nicolas Trangez
 wrote:
        Michael,
On Tue, 2012-11-27 at 17:14 +0200, Michael Snoyman wrote:
        > I think the stm-conduit package[1] may be helpful for this
        use case.
        > Each time you get a new command, you can fork a thread and
        give it the
        > TBMChan to write to, and you can use sourceTBMChan to get a
        source to
        > send to the client.
That's +- what I had in mind. I did find stm-conduit before
        and did try
        to get the thing working using it, but these attempts failed.
I attached an example which might clarify what I intend to do.
        I'm aware
        it contains several potential bugs (leaking threads etc), but
        that's
        beside the question ;-)
If only I could figure out what to put on the 3 lines of
        comment I left
        in there...
Thanks for your help,
Nicolas
The issue is that you're trying to put everything into a single
Conduit, which forces reading and writing to occur in a single thread
of execution. Since you want your writing to be triggered by a
separate event (data being available on the Chan), you're running into
limitations.
The reason network-conduit provides a Source for the incoming data and
a Sink for outgoing data is specifically to address your use case. You
want to take the data from the Source and put it into the Chan in one
thread, and take the data from the other Chan and put it into the Sink
in a separate thread. Something like:
myApp appdata = do
    chan1 <- ...
    chan2 <- ...
    replicateM_ 5 $ forkIO $ worker chan1 chan2
    forkIO $ appSource appdata $$ sinkTBMChan chan1
    sourceTBMChan chan2 $$ appSink appdata
You'll also want to make sure to close chan1 and chan2 to make sure
that your threads stop running.
Thanks, I +- figured that out last night. Only thing left is managing
the handshake before forking off the workers, if possible (otherwise
things become very messy IMHO), but ResumableSources etc might be of use
here.

Maybe there's a library somewhere in here...

Thanks a bunch,

Nicolas