On Tue, 2007-02-13 at 22:43 -0500, Jefferson Heard wrote:
It was suggested that I might derive some performance benefit from using lazy bytestrings in my tokenizer instead of regular strings. Here's the code that I've tried. Note that I've hacked the "basic" wrapper code in the Lazy version, so the code should be all but the same. The only thing I had to do out of the ordinary was write my own 'take' function instead of using the substring function provided by Data.Lazy.ByteString.Char8. The take function I used was derived from the one GHC uses in GHC.List and produces about the same code.
If you use the latest darcs version of alex and the "basic-bytestring" wrapper then you don't have to write any of your own take functions. Actually, lazy ByteString are still not as optimised as I would like. There are still too many indirections. That's something I'm working on at the moment. Duncan