2010/10/26 Stephen Tetley <stephen.tetley@gmail.com>

Hello

I would change you Alex specification to this:

$digit = 0-9 -- digits
$alpha = [a-zA-Z] -- alphabetic characters
$eol = [\r\n]
$any = [^$eol]

tokens :-

$eol { tok $ \_ -> Eol }
$any+ { tok $ \s -> Str s }

The complementation operator (^) works of character sets so I don't
expect your original formulation to work:

$any = [^\r\n]

(maybe it should, but I never liked the Alex syntax...)

You can test alex scanners like this:

demo01 = alexScanTokens "happy?\n"
demo02 = readFile "sample" >>= print . alexScanTokens

Note - your sample file is using extended characters so it fails for
me with Alex 2.3.2. I'm now sure how capable the current version of
Alex is or whether better Unicode support can be enabled with flags.

Regards

Stephen
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe