On Tue, Jun 17, 2008 at 07:55:51AM +0100, Neil Mitchell wrote:
Hi,
Hello!
David:
Is there any reason not to just write all lazy fields of variable size in a deferred manner?
I completely hadn't though of this! You will loose a bit of time, for example reading fields which were previously not deferred will require a file seek, but the effort/thought/efficiency trade-off is good. I think I will go with this.
Actually, you ought to be able to pretty easily remove this tradeoff by introducing a strict read function as a new method in your class. So anyone who wants to strictly read lazy data can use that function instead of the lazy one. The writing of data is unaffected... or rather, it *is* affected, but only in the sense that writing deferred data cannot be as lazy as the writing of undeferred data. e.g. when writing a deferred list, you can't output the first word until you've already read through the entire list in order to find out how long it is. That could actually be quite a problem for code that relies on laziness to handle massive amounts of data. :( Lazy reading seems to require strict writing, while lazy writing requires strict reading! So perhaps you want a pair of read functions, one strict, one lazy, and a pair of write functions. That doesn't provide any sort of fine-grained control, but at least allows streaming in either direction. You could use the same data format for either strictly or lazily pickled data as long as you have a reserved value for the length-to-be-skipped, so the lazy reader could tell if it reaches data that whose length it doesn't know. David