On 03/02/2013 16:06, Felipe Almeida Lessa wrote:
I guess you could use the Flush datatype [1] depending on how your data is generated.
Thank you for this suggestion. I tried to do exactly this by modifying my bulk Redis source so that it can timeout and send empty lists [1]. Then I wrote a few helpers conduits[2], such as : concatFlush :: (Monad m) => Integer -> Conduit [a] m (Flush a) which will convert a stream of [a] into a stream of (Flush a), sending Flush whenever it encounters and empty list or it send a tunable amount of data downstream. I finally modified my examples [3]. I realized then it would be nice to have fmap for conduits (but I am not sure how to write such a type signature). Suggestions are welcome ! [1] https://github.com/bartavelle/hslogstash/commit/663bf8f5e6058b476c9ed9b5c9cf... [2] https://github.com/bartavelle/hslogstash/blob/master/Data/Conduit/Misc.hs [3] https://github.com/bartavelle/hslogstash/blob/master/examples/RedisToElastic...