lexer works fine, problem is in happy parser. 2010/10/26 Stephen Tetley

Hello

I would change you Alex specification to this:

$digit = 0-9 -- digits $alpha = [a-zA-Z] -- alphabetic characters $eol = [\r\n] $any = [^$eol]

tokens :-

$eol { tok $ \_ -> Eol } $any+ { tok $ \s -> Str s }

The complementation operator (^) works of character sets so I don't expect your original formulation to work:

$any = [^\r\n]

(maybe it should, but I never liked the Alex syntax...)

You can test alex scanners like this:

demo01 = alexScanTokens "happy?\n" demo02 = readFile "sample" >>= print . alexScanTokens

Note - your sample file is using extended characters so it fails for me with Alex 2.3.2. I'm now sure how capable the current version of Alex is or whether better Unicode support can be enabled with flags.

Regards

Stephen _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Great thanks! All work right now. 2010/10/26 Stephen Tetley

The lexer was wrong - but it was the lexer function not the lexer spec - try the one below.

Note that you have to take 'len' chars from the original input. Previously you were taking the whole of the "rest-of--input":

lexer :: (TheToken -> P a) -> P a lexer f input@(_,_,instr) = case alexScan input 0 of AlexEOF -> f Eof input AlexError (pos, _, _) -> Failed $ showPos pos ++ ": lexical error" AlexSkip input' len -> lexer f input' AlexToken (pos, c, str) len act -> let (Token newpos thetok) = act pos (take len instr) in f thetok (newpos, c, str) _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe