8 Mar
2011
8 Mar
'11
10:54 a.m.
I'd join comments in with tokens so each token has a comment - possibly the empty string, then the parser can decide what to do with the comment part of token - e.g retaining it for functions, ignoring it for everything else. You may have to write a two-pass lexer to do this.