On Tue, 2012-04-17 at 10:32 +0300, Eyal Lotem wrote:

I built a preliminary prototype of "cabal update" enhancements:

A) Using a Lazy I/O bytestring with the HTTP download, so that it can do a streaming-decompress on-the-fly (This gives a 30% speedup on my machine)

That's interesting. I'm surprised it's that much. So you're decompressing and writing the decompressed .tar to disk? But not unpacking the .tar on the fly right? Is that 30% figure just of the time to write the uncompressed file to disk, or also of the download and unpack? Or are we just talking about the index file here? I guess I typically assume that it makes sense to write the compressed file to disk, and decompresses and unpack on the fly, less disk I/O. That said, for the index we do keep the uncompressed version on disk, so doing the decompression on the fly there would make sense.

B) Using the bytestring-progress library to show a useful progress report on the whole process. Unfortunately, due to the PVP, the version constraints are (as usual) over-restrictive, so they need to be relaxed. Until the fix is put on hackage, you have to install bytestring-progress from https://github.com/Peaker/bytestring-progress

Interesting. I'm a bit sceptical of it's subtle side effects. I wonder if we can do it in a slightly nicer way. Actually, something that would help is the new interfaces in the zlib binding I've been working on. That'd let you push blocks in one at a time and keep track of the original length, and calculate progress.

C) It's quick&dirty (Proof of concept), because: C.1) No version constraints on bytestring-progress C.2) Using hSetBuffering on stdout and ANSI terminal codes, which may be problematic in different kinds of terminals C.3) The only way to disable it is setting verbosity=quiet. It probably deserves its own option

I hope it's a useful demonstration of the utility of this, though. It makes "cabal update" something I'd dread a little less to run :)

Yes, thanks. BTW, the other way to make that better is to do an incremental index update, and so simply download much less data. If you're interested in hacking on that, lemme know.

Also, I was wondering why there's no monad for the passing of Verbosity around to so many functions. Then I could easily add more options that go around to control things like whether to show progress bars...

There should be, but you actually want rather more than that otherwise the saving of the verbosity params would be offset by rather a lot of liftIO $ ... What it needs is a shell monad of some kind. I've done a few test designs. It's a major overhaul, but should be done. Duncan