Re: [Haskell-cafe] lazy-vs-strict, recursive IO (was: memory needed for SAX parsing XML)

(changed the thread title hoping to get some others to weigh in on the recursive I/O scheme presented here)

From: Daniil Elovkov John Lato wrote:

...

...

Another (additional) approach would be to encapsulate unsafeInterleaveIO within some routine and not let it go out into the wild.

lazilyDoWithIO :: IO a -> (a -> b) -> IO b

It would use unsafeInterleave internally but catch all IO errors within itself.

I wonder if this is a reasonable idea? Maybe it's already done? So the topic is shifting...

doWithIO :: NFData b => IO a -> (a -> b) -> IO b doWithIO m f = liftM (\a -> let b = f a in b `deepseq` b) m

It works (just stick it in a "try" block for error handling), but you need to write a lot of NFData instances.  You also need to be careful that b is some sort of reduced structure, or you can end up forcing the whole file (or other data) into memory.

I meant a different thing. In your example there is no unsafeInterleave at all. I think you mean that 'm' argument is supposed to be an unsafeInterleaved io action, like getContents, and deepseq'ing saves us from it hanging somewhere for a long time. Ok

Yes, that's what I intended. I understand now that it isn't what you meant.

But I meant to have a routine that lets us use ordinary io actions in a lazy way, and restricting that 'hanging' within bounds of this routine.

But having thought about it a little more I understood that it's impossible.

I think it's possible provided you have unsafeInterleaveIO and deepseq. As you discovered, to force evaluation through an unsafeInterleaveIO you need to fully evaluate data, and deepseq is the way to do so.

Lazy io works now because unsafeInterleaveIO is sticked into getContents itself, and is called repeatedly (via recursion). Or at least I can think of this implementation, haven't looked into it.

I realised that calling unsafeInterleaveIO for an ordinary io action will not make it run lazily. It will, still, run all-at-once, just not now, but later.

Yes, exactly. unsafeInterleaveIO needs to be used recursively for getContents to function properly.

So, to cope with it, I can think of exposing a little structure of an io action. Normally it's completely opaque. But if we knew where its recursion point lies, then we could control its course of execution.

So, if we had something like   type RecIO a = IO a -> IO a and io actions were like   getContents :: RecIO [Char]   getContents rec = do       c <- readOneChar       rest <- rec       return (c:rest)

then we could either run them   normally :: RecIO a -> IO a   normally r = r (normally r) or   lazily :: RecIO a -> IO a   lazily r = unsafeInterleavIO $ r (lazily r)

And   lazilyDoWithIO :: RecIO a -> (a -> b) -> IO b   lazilyDoWithIO m f = do     a <- lazily m     return $ f a

Hmm, but then, we would have to take special care to not let it out of this function anyway... So here we come to deepSeq'ing you proposed.

I think my understanding matches yours at this point, except I'm not sure I follow RecIO. Your getContents, for example, seems to only read one character. I think you mean something like this? recGetContents :: RecIO String recGetContents rec = do c <- getChar ise <- isEOF if ise then return [c] else liftM (c:) rec normally :: RecIO a -> IO a normally = fix lazily :: RecIO a -> IO a lazily rec = let f = unsafeInterleaveIO (rec f) in f This works, but I'm not sure that it accomplishes anything. (it doesn't work in ghci because EOF isn't sent through the terminal, but if you test for another character it works fine).

And anyway, instead of re-writing pure functions to become iteratees we will have to re-write io functions to adopt continuation passing style.

Initially it looked better to me :)

Re-writing pure functions to be iteratees doesn't have to be hard; many of the provided iteratees have the same names as standard functions so a lot of the work can be done by just changing imports.

But with this approach we can run lazily any io action that has the form of RecIO. Also, we can interleave normally and lazily based on the time of day and other conditions :)

Originally I didn't see much use to this scheme, but I'm starting to change my mind. It would be nice for users to be able to control the evaluation of functions like getContents, hGetContents, etc., and this looks like it could be easier to maintain.

...

It also doesn't help with other IO effects, e.g. writing output.  I consider this one of the nicest features of iteratee-based processing.

Can you clarify what's the problem with writing? I think I just haven't switched from the topic of gluing code.

Because as for gluing code, type signature for io writing d -> IO () is perfectly sufficient.

The problem isn't with writing output per se, but with sequencing I/O effects with lazy data processing. If you wanted to write to a log file after processing each 100kbytes of a file, for example, doing so with bytestrings would require manually chunking the string, creating output, etc., which then hampers both modularity and readability. Adding this sort of monadic side-effect while data is only partially evaluated is pretty simple with iteratees. Actually, this recursive IO combinator idea you've sketched out would be somewhat helpful in this regard too. John