Hi Niklas, 1) My most desired feature would be a syntax tree that does not pluck

pluck comments out and make me treat them separately. It looks much easier to me to have a fully descriptive tree and (filter . concatMap) / traverse them out in some way than getting a list of comments and having to insert them back in the right places myself. Is that possible?

Sadly not - it's theoretically impossible. The fact that you can put comments literally wherever, means that it's impossible to treat them as nodes of the AST. E.g. f {- WHERE -} x = -- WOULD -- THESE do -- COMMENTS a {- END -} <- g x -- UP return {- ? -} a What would be theoretically possible is to define a restricted language that allows comments only in certain well-defined places (cf haddock), and ignores any others. That's a lot of work though, and it's not clear how big the gain is. :-\ A different solution could be to improve the support, through better helper functions, for handling a syntax tree and a list of comments together. That's something I think could be worthwhile.

2) Have you considered downloading the all-of-Hackage tarball and running haskell-src-exts over it to get a benchmark of how much HSE can already parse of the Haskell code out there?

Considered, yes. Done, no. Would love to see the results :-). The crew at OdHac (Roman, Erik, Simon) ensured that the current version handles all of 'base', which is a good start. Cheers, Niklas

+1 When I worked on the font-lock support for haskell-mode, the irony of trying to approximate the classification that the hugs/ghc/whatnot parser was already doing wasn't lost on me. I still would like to tap into more of the knowledge generated and lost in the compiler: - A list of all tokens (source position, span, classification). Comments are conventionally treated as whitespace, but it's not very hard to capture them before dropping them. - All identifiers can be classified into def and use with enough lexical scope information to get from use to def and from def to all uses. Take this one step further and type information can be stored with the def. - Tokens should be mapped to the underlying AST if possible, and vice versa. I'm sure there's more, but with this, one could build an awesome editor, code navigator. I think it's possible, but I don't have time to work on it, alas. I'd like suggestions as to to realize this. Tommy -- ponding hacking the parser combinators to keep this information. On Aug 20, 2013, at 03:00 , Niklas Hambüchen wrote:

On 20/08/13 18:19, Niklas Broberg wrote:

...

Sadly not - it's theoretically impossible. The fact that you can put comments literally wherever, means that it's impossible to treat them as nodes of the AST. E.g.

f {- WHERE -} x = -- WOULD -- THESE do -- COMMENTS a {- END -} <- g x -- UP return {- ? -} a

Oh, I see what you mean.

I guess what I mean instead is:

* A lex list that contains *everything*, including comments and white space

* A full syntax tree of which each node points to (indexes) a position in the lex list to get the precise original position; comments in between two nodes can then be determined and more easily played with because they are between their positions in the lex list

* An abstract syntax tree that has whitespace and comments discarded (what HSE has now)

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

This is not using haskell-src-exts, but the Haskell Refactorer has a structure to keep a parallel tree of tokens indexed by SrcSpan, which attempts to allocate comments to the appropriate point. See https://github.com/alanz/HaRe/blob/master/src/Language/Haskell/Refact/Utils/.... It does not make use of the AST itself, so may be usable with haskell-src-exts Alan On Tue, Aug 20, 2013 at 11:19 AM, Niklas Broberg wrote:

Hi Niklas,

1) My most desired feature would be a syntax tree that does not pluck

...

pluck comments out and make me treat them separately. It looks much easier to me to have a fully descriptive tree and (filter . concatMap) / traverse them out in some way than getting a list of comments and having to insert them back in the right places myself. Is that possible?

Sadly not - it's theoretically impossible. The fact that you can put comments literally wherever, means that it's impossible to treat them as nodes of the AST. E.g.

f {- WHERE -} x = -- WOULD -- THESE do -- COMMENTS a {- END -} <- g x -- UP return {- ? -} a

What would be theoretically possible is to define a restricted language that allows comments only in certain well-defined places (cf haddock), and ignores any others. That's a lot of work though, and it's not clear how big the gain is. :-\

A different solution could be to improve the support, through better helper functions, for handling a syntax tree and a list of comments together. That's something I think could be worthwhile.

...

2) Have you considered downloading the all-of-Hackage tarball and running haskell-src-exts over it to get a benchmark of how much HSE can already parse of the Haskell code out there?

Considered, yes. Done, no. Would love to see the results :-). The crew at OdHac (Roman, Erik, Simon) ensured that the current version handles all of 'base', which is a good start.

Cheers, Niklas

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

On 20/08/13 09:48, Niklas Hambüchen wrote:

Nice!

I hope that haskell-suite will eventually become awesome and solve most of our automation-on-Haskell-code needs.

Two questions:

1) My most desired feature would be a syntax tree that does not pluck pluck comments out and make me treat them separately. It looks much easier to me to have a fully descriptive tree and (filter . concatMap) / traverse them out in some way than getting a list of comments and having to insert them back in the right places myself. Is that possible?

+1 for this. There was a small discussion relevant to this on café recently, if anyone is interested: http://comments.gmane.org/gmane.comp.lang.haskell.cafe/106768

2) Have you considered downloading the all-of-Hackage tarball and running haskell-src-exts over it to get a benchmark of how much HSE can already parse of the Haskell code out there?

Thanks!

-- Mateusz K.

On 20/08/13 11:02, JP Moresmau wrote:

BuildWrapper has some code that tries to link back the comments to the declaration from the AST generated by haskell-src-exts and the comments. See https://github.com/JPMoresmau/BuildWrapper/blob/master/src/Language/Haskell/.... The unit tests provide some samples: https://github.com/JPMoresmau/BuildWrapper/blob/master/test/Language/Haskell.... Maybe this can help you.

JP It certainly look like I might be able to learn from this.

Thank you. -- Mateusz K.

On 20/08/13 11:56, Dag Odenhall wrote:

Good stuff!

Is there any way, or plans for a way, to parse a file based on its LANGUAGE pragmas? Last I checked e.g. HSP simply enabled all extensions when parsing, which can cause code to be parsed incorrectly in some cases.

Can you give any examples of such cases? I had recently been asked about this and could not come up with much at all. -- Mateusz K.

HSE parses based on pragmas by default. This can be configured through the ParseMode [1]. But your question regards HSP, Haskell Server Pages, which indeed just enables most extensions by default. Right now there's no way to configure that, but it shouldn't be hard for a skilled programmer to fix. Patches most welcome. :-) Cheers, Niklas [1] http://hackage.haskell.org/packages/archive/haskell-src-exts/1.13.5/doc/html... On 20 Aug 2013 12:57, "Dag Odenhall" wrote:

Good stuff!

Is there any way, or plans for a way, to parse a file based on its LANGUAGE pragmas? Last I checked e.g. HSP simply enabled all extensions when parsing, which can cause code to be parsed incorrectly in some cases.

On Tue, Aug 20, 2013 at 10:15 AM, Niklas Broberg

...

wrote:

...

Fellow Haskelleers,

I'm pleased to announce the release of haskell-src-exts-1.14.0!

* On hackage: http://hackage.haskell.org/package/haskell-src-exts * Via cabal: cabal install haskell-src-exts * git repo: https://github.com/haskell-suite/haskell-src-extshttp://code.haskell.org/haskell-src-exts

There are two primary reasons for this release, and a number of smaller ones.

The first primary reason is technical: haskell-src-exts 1.14 revamps the Extension datatype, among other things to allow turning extensions on and off (similar to what Cabal allows). We also introduce the concept of a Language, separate from a set of extensions. This is the only backwards-incompatible change in this release.

The second reason is structural: haskell-src-exts is now part of a larger context -- the Haskell Suite. The package has a new home on github (see above), alongside its new cool friends: haskell-names and haskell-packages. There is also a really nice issue tracker there - please help me fill it, or better yet, empty it!

What this release does *not* cover is support for the extensions added to GHC in recent time (with the exceptions of CApiFFI and InterruptibleFFI). Work is in progress on many of these, and there will be another major release not far off in the future.

This release owes many thanks to Roman Cheplyaka in particular, as well as Erik Hesselink, Simon Meier and David Fox. Thanks a lot!

Complete changelog:

1.13.6 --> 1.14.0 ===============

* Modernize the Extension datatype in L.H.E.Extension, following the lead of Cabal, to allow negative and positive extension modifiers (turning features on and off). You need to worry about backwards-incompatible changes if any of the following pertains to you: 1) If you use the Extension datatype programmatically - it has changed significantly (see documentation). 2) The ParseMode record now has one more field (baseLanguage :: Language), which might give you a type error. 3) The behavior of the (extensions :: [Extension]) field has changed, which could bite you if you pass custom extensions in the ParseMode. Previously, the ParseMode defaulted to the list of extensions accepted by Haskell2010, and if you set the list explicitly you would override this. Now, the defaults are { baseLanguage = Haskell2010, extensions = [] }, and explicitly setting a list of extensions will be interpreted on top of Haskell2010. See further the documentation for L.H.E.Extension.

* Add support for the 'capi' calling convention. It is enabled with the CApiFFI extension. It's been included since GHC 7.4, and advertised since 7.6.

* Add support for the 'interruptible' FFI safety annotation, enabled with the InterruptibleFFI extension.

* Give better error message when lexing newline fails. In particular, fix the bug when the parser would crash if the file didn't end with a newline.

* Support unboxed tuple expressions and patterns.

* Fix bug in lexing of primitive integer literals in hex or octal notation.

* Disallow negative primitive word literals (such as W# (-0x8000000000000000##)).

* Allow phase control for SPECIALIZE pragma.

* Derive Foldable and Traversable instances for all annotated AST types.

* Fix bug with pretty-printing WARNING and DEPRECATED pragmas.

Cheers, Niklas

-- You received this message because you are subscribed to the Google Groups "Haskell Server Pages" group. To unsubscribe from this group and stop receiving emails from it, send an email to haskell-server-pages+unsubscribe@googlegroups.com. To post to this group, send email to haskell-server-pages@googlegroups.com. Visit this group at http://groups.google.com/group/haskell-server-pages. For more options, visit https://groups.google.com/groups/opt_out.

-- You received this message because you are subscribed to the Google Groups "Haskell Server Pages" group. To unsubscribe from this group and stop receiving emails from it, send an email to haskell-server-pages+unsubscribe@googlegroups.com. To post to this group, send email to haskell-server-pages@googlegroups.com . Visit this group at http://groups.google.com/group/haskell-server-pages. For more options, visit https://groups.google.com/groups/opt_out.