February 2007 - C2hs - Haskell.org

patch applied (c2hs): Dump perfect attribute order compatability with the previous parser
by Duncan Coutts 20 Feb '07

20 Feb '07

Tue Feb 20 01:40:55 PST 2007 Duncan Coutts <duncan.coutts(a)worc.ox.ac.uk> * Dump perfect attribute order compatability with the previous parser We can't preserve this anyway so no need for keeping the code that tries to do so. M ./c2hs/c/CParser.y -11 +7

1 0

patch applied (c2hs): Oops, missing ','
by Duncan Coutts 20 Feb '07

20 Feb '07

Tue Feb 20 01:39:40 PST 2007 Duncan Coutts <duncan.coutts(a)worc.ox.ac.uk> * Oops, missing ',' M ./c2hs/c/CParserMonad.hs -1 +1

1 0

patch applied (c2hs): Ignore all __attribute__ expressions in the lexer
by Duncan Coutts 20 Feb '07

20 Feb '07

Mon Feb 19 14:45:49 PST 2007 Duncan Coutts <duncan.coutts(a)worc.ox.ac.uk> * Ignore all __attribute__ expressions in the lexer GNU __attribute__ expressions take the form of: __attribute__ '(' '(' ... ')' ')' where the stuff in the middle may also have nested '(' ')' brackets. So upon getting an __attribute__ token, the lexer now skips tokens counting bracket nesting depth and stops when it's seen the full attribute args. So the parser now never sees attributes which is good because they greatly complicate the grammar of declarations. On the other hand it's bad because later on we might need the info from some attributes. So we'll likely have to revisit this issue. M ./c2hs/c/CLexer.x -2 +14

1 0

patch applied (c2hs): Add shadowTypedef operation to parser monad
by Duncan Coutts 19 Feb '07

19 Feb '07

Mon Feb 19 12:35:13 PST 2007 Duncan Coutts <duncan.coutts(a)worc.ox.ac.uk> * Add shadowTypedef operation to parser monad Local vars in inner scopes can shadow type names so we need to be able to adjust the current typedef identifier set accordingly (ie delete identifiers from the typedef ident set). For example: typedef int X; /* X declared as a type name */ void foo() { int X; X = X +1 } /* X redeclared as a local var */ M ./c2hs/c/CParserMonad.hs -1 +11

1 0

patch applied (c2hs): Rename vars and reorder exports in ParserMonad module
by Duncan Coutts 19 Feb '07

19 Feb '07

Mon Feb 19 12:33:58 PST 2007 Duncan Coutts <duncan.coutts(a)worc.ox.ac.uk> * Rename vars and reorder exports in ParserMonad module M ./c2hs/c/CParserMonad.hs -12 +12

1 0

patch applied (c2hs): Add scope history and enterScope/leaveScope
by Duncan Coutts 19 Feb '07

19 Feb '07

Mon Feb 12 11:00:42 PST 2007 Duncan Coutts <duncan.coutts(a)worc.ox.ac.uk> * Add scope history and enterScope/leaveScope entering a scope pushes the current tyident set into a history stack leaving a scope pops that stack. M ./c2hs/c/CParserMonad.hs -1 +15

1 0

patch applied (c2hs): Rename parser state fields, change get/setInput type and tidy imports
by Duncan Coutts 19 Feb '07

19 Feb '07

Mon Feb 12 10:59:51 PST 2007 Duncan Coutts <duncan.coutts(a)worc.ox.ac.uk> * Rename parser state fields, change get/setInput type and tidy imports M ./c2hs/c/CLexer.x -12 +15 M ./c2hs/c/CParser.y -2 +3 M ./c2hs/c/CParserMonad.hs -24 +24

1 0

patch applied (c2hs): Split tokens and parser monad out of Lexer.x into their own modules
by Duncan Coutts 19 Feb '07

19 Feb '07

Mon Feb 12 04:20:37 PST 2007 Duncan Coutts <duncan.coutts(a)worc.ox.ac.uk> * Split tokens and parser monad out of Lexer.x into their own modules This allows the parser monad to be made a little more abstract. M ./c2hs/c/CLexer.x -382 +4 A ./c2hs/c/CParserMonad.hs A ./c2hs/c/CTokens.hs

1 0

a new C parser
by Duncan Coutts 19 Feb '07

19 Feb '07

All, I posted some time ago about a plan to make a new C parser that would correctly deal with the typedef problem and then to try some automated testing by making a gcc wrapper. I've now mostly done that. Currently I can parse almost all of the Linux kernel, glibc, GNU binutils, and the C code produced by ghc when it compiles itself. I can also parse most of the artificial nasty typedef examples: typedef int A, B(A); I think I correctly deal with scope of typedef names and typedef names being shadowed by local variable declarations: typedef int A; A main () { A A; -- declares local var A as having type A A = A + 1; -- A now refers to the local var, not the type; } A x; -- end of scope, A reverts to referring to the typedef name. A slightly more tricky example of the same thing is where function parameters redeclare names: typedef int A; A main (A A) { A = A + 1; -- A now refers to the local var, not the type; } An example I don't deal with yet is: typedef int A; int A = sizeof(A); that is we shadow the typename A with the local var A, so the A in the initialiser should now refer to the local var not the typename. However currently the parser only does the shadowing after the ';', not after the '=' as would be required. This is because the production rule looks like: declaring_list : declaration_specifier declarator initializer_opt {% do doDeclIdent $1 $2 return $ CDecl $1 [($2, $3, Nothing)]) } where as we'd really like to say: declaring_list : declaration_specifier declarator {% doDeclIdent $1 $2 } initializer_opt { CDecl $1 [($2, $4, Nothing)]) } that is to modify the typedef set immediately after the declarator and before parsing the initializer. This is exactly how people do it with YACC but unfortunately happy does not support this form where actions are interspersed in the production. It's probably possible to refactor the grammar to do this with happy, though it's not totally obvious to me how. Dealing with '__attribute__'s: ============================== GNU C attributes are a real pain. They seem to be able to appear almost anywhere in a declaration. However adding them to the grammar has to be done very carefully to avoid introducing ambiguities (ie shift/reduce or reduce/reduce conflicts). So for the moment I've taken the approach of not parsing them at all but recognising and ignoring attributes in the lexer and not passing them on to the parser at all. This effectively treats attributes as whitespace and they can appear anywhere. This isn't a long term solution because eventually we'll need to recognise some attributes because they can affect structure layout and we need that for accurately calculating the size of types and offsets of structure members. C99 extensions: =============== I'm not sure I've got 100% coverage of C99 features, but at least I cover the ones used in the code I've seen so far. These are just the ones that I can remember adding: * mixed declarations and statements in a compound statement * compound literals * for statements with declarations: for (int n = 0; n < m; n++) {}; * wide character character and string literals * unsigned long long numeric literals * _Bool and _Complex basic types GNU extensions: =============== Again, just some of the GNU extensions that I can remember adding to the new parser compared to the extensions the previous parser recognised: * anonymous structs and unions inside other structs/unions. * thread local storage qualifier * old style and array range compound literal member designator * case ranges * computed gotos and address of labels * conditional operator with missing then part: "x ? : y" * empty structs * allow redundant ';' in several places * asm expressions * several built-in 'functions' that take type names as parameters Various of these extensions require extensions in the AST. Testing: ======== I've made a cc-wrapper program that you can use as if it were gcc. It calls gcc with the same args as it was called with. Then if it can figure out what the gcc args meant and it looks like gcc was being asked to compile a .c file then it tries to pre-process and parse the same .c file. It outputs results to a log file and puts detailed reports on parse failures into separate files (in a directory specified via an environment variable). So (in theory at least) one ought to be able to compile vast amounts of C code and find out which extensions are really used and break the parser. At the moment the it doesn't check that it correctly parses the code, only that it parses it at all. Extra checks as I mentioned before could include checking if the sizes of types and offsets of structure members computed by c2hs match those computed by gcc. Another test would be to pretty-print the C code again and see if gcc can still parse it and if it produces the same object code (that'd require fully parsing attributes). As I said, I've been testing this on the linux kernel (using allyesconfig) glibc and ghc. Last time I tried it against linux it failed on 7 out of 4406 .c files. All those failures were due to the same bug which I think I've now fixed. In glibc, a few files fail to parse due to their use of nested function declarations which is a rarely used GNU extension that I've not implemented yet. Plan: ===== So there are a few things to do, one is to continue testing large amounts of C code. I intend to include cc-wrapper in c2hs (or perhaps do it as a c2hs mode via a flag) so that users can do their own testing and provide a way to give detailed bug reports. I haven't checked yet that the new C parser doesn't break things when doing the full .chs -> .hs translations. Also, I've not extended the guts of c2hs to understand any of these new language extensions so trying to bind things which really use them may not work. Then I need to start merging the new parser into the darcs version of c2hs. I'll try and do this as a series of small understandable patches. If anyone wants to play with the code I've got right now rather than wait for things to get merged then do say. I could put up a tarball. Duncan

3 3

Re: [Haskell-cafe] c2hs failed to process __extension__
by Duncan Coutts 18 Feb '07

18 Feb '07

On Mon, 2007-01-29 at 21:25 +0900, 向井淳 wrote: > c2hs fails to process __extension__? > > From the changelog, c2hs now can understand __extension. > Is this an exceptional case? Did you try the very latest darcs version of c2hs? Yesterday I added a couple patches that try to cope with more uses of __extension. Duncan

1 1