As a newbie to Haskell, I found your thorough analysis very
interesting. Thanks for the great read! I have a few questions
regarding some of your comments, see below:
Jules Bean
E,F. Progressive GET pSynGET :: URL -> ((Bool,ByteString) -> IO ()) -> IO () pAsynGET :: URL -> ((Bool,ByteString) -> IO ()) -> IO (MVar ())
(This is a particular simple case of Oleg's iteratees, I think) Download the data at whatever speed is convenient. As data arrives, feed it to the 'callback' provided. The ByteString is the new chunk of data, the 'Bool' is just supposed to indicate whether or not this is the final chunk.
Incidentally there are more complex options than (Bool,Bytestring) -> IO (). A simple and obvious change is to add a return value. Another is a 'state monad by hand', as in (Bool,Bytestring) -> s -> s, and change the final return value of the type to IO s, which allows the callback to accumulate summary information and still be written as pure code.
I want to be sure that I understand the implications of the callback function returning an IO action as originally proposed versus it being a pure function. It would seem to me that if it were a pure callback the usefulness would be limited as I would not be able to take the data read from the network and immediately write it out to a file. Is this correct? And if the above is correct, is there a way to define the callback such that one does not have to hardcode the IO monad in the return type so you can have the best of both worlds?
Other options allow the 'callback' to request early termination, by layering in an 'Either' type in there.
I believe the ability to request early termination is important, and was one of the nice features of Oleg's left-fold enumerators. It would be a shame if the API did not offer this capability.
Another more sophisticated option, I think, is the higher rank
MonadTrans t => URL -> ((forall m. Monad m) => (Bool,ByteString) -> t m) -> t IO ()
...which, unless I've made a mistake, allows you to write in 'any monad which can be expressed as a transformer', by transforming it over IO, but still contains the implicit promise that the 'callback' does no IO. For example t = StateT reduces to the earlier s -> s example, in effect, with a slightly different data layout.
I don't fully understand this, but would this prevent one from calling IO actions as it was receiving the chunks in the callback (such as writing it to a file immediately)?