#9628: Add Annotations to the AST to simplify source to source conversions -------------------------------------+------------------------------------- Reporter: alanz | Owner: alanz Type: feature | Status: new request | Milestone: Priority: normal | Version: 7.8.3 Component: Compiler | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Unknown Unknown/Multiple | Blocked By: Type of failure: | Related Tickets: None/Unknown | Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Changes (by alanz): * owner: => alanz -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9628#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9628: Add Annotations to the AST to simplify source to source conversions -------------------------------------+------------------------------------- Reporter: alanz | Owner: alanz Type: feature | Status: new request | Milestone: Priority: normal | Version: 7.8.3 Component: Compiler | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Unknown Unknown/Multiple | Blocked By: Type of failure: | Related Tickets: None/Unknown | Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Comment (by simonpj): Whoa! There appear to be two quite separate things going on here: * In [wiki:GhcAstAnnotations] the issue is attaching a `SrcSpan` to keywords * In comment:1 the issue appears to be about attaching an annotation of client-specified type to every node in the tree. These seem quite orthogonal to me. For the latter I would suggest looking at the `Located` type, instead of what you suggest in comment:1. The `Located` type is wrapped around almost every node in the tree, and if you want to add some ubiquitous annotation type, it would be the place to do so. Simon -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9628#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9628: Add Annotations to the AST to simplify source to source conversions -------------------------------------+------------------------------------- Reporter: alanz | Owner: alanz Type: feature | Status: new request | Milestone: Priority: normal | Version: 7.8.3 Component: Compiler | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Unknown Unknown/Multiple | Blocked By: Type of failure: | Related Tickets: None/Unknown | Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Comment (by simonpj): Yes, that's right. Looks much nicer to me. The syntax tree is ''already'' heavily decorated with `SrcSpans`. Just parameterise over that, and you can decorate with something else instead. An alternative would be to insist that there was ''always'' a `SrcSpan`, plus perhaps something else: {{{ data GenLocated l e = L SrcSpan l e type Located e = GenLocated () e }}} That's tiresome because there are lots of `()` values, but does mean you can always find a `SrcSpan`. One would have to explore use cases. Simon -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9628#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9628: Add Annotations to the AST to simplify source to source conversions -------------------------------------+------------------------------------- Reporter: alanz | Owner: alanz Type: feature | Status: new request | Milestone: Priority: normal | Version: 7.8.3 Component: Compiler | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Unknown Unknown/Multiple | Blocked By: Type of failure: | Related Tickets: None/Unknown | Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Comment (by alanz): Once the parameter is added to the hsSyn structures, there are two ways to work it forward. 1. Work it in as a parameter throughout all the compiler passes 2. Hard code it to SrcSpan for renamer onwards Option 1 allows tool writers to add annotations and then submit for further processing, but sprinkles the parameter variable throughout the code base. Option 2 provided the most value for tool developers. Which road? I am leaning towards working it all the way through. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9628#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9628: Add Annotations to the AST to simplify source to source conversions -------------------------------------+------------------------------------- Reporter: alanz | Owner: alanz Type: feature | Status: new request | Milestone: Priority: normal | Version: 7.8.3 Component: Compiler | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Unknown Unknown/Multiple | Blocked By: Type of failure: | Related Tickets: None/Unknown | Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Comment (by alanz): Question: can I rip `hooks` out of `DynFlags`, to be managed separately? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9628#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9628: Add Annotations to the AST to simplify source to source conversions -------------------------------------+------------------------------------- Reporter: alanz | Owner: alanz Type: feature | Status: new request | Milestone: Priority: normal | Version: 7.8.3 Component: Compiler | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Unknown Unknown/Multiple | Blocked By: Type of failure: | Related Tickets: None/Unknown | Test Case: | Blocking: | Differential Revisions: D246 | -------------------------------------+------------------------------------- Changes (by alanz): * differential: => D246 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9628#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9628: Add Annotations to the AST to simplify source to source conversions -------------------------------------+------------------------------------- Reporter: alanz | Owner: alanz Type: feature | Status: new request | Milestone: Priority: normal | Version: 7.9 Component: Compiler | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Unknown Unknown/Multiple | Blocked By: Type of failure: | Related Tickets: None/Unknown | Test Case: | Blocking: | Differential Revisions: D297 | -------------------------------------+------------------------------------- Changes (by alanz): * differential: D246 => D297 * version: 7.8.3 => 7.9 Old description:

Two different kinds of annotation are proposed to be added to HsSyn

1. Additional elements to explictly track the location of all syntactic elements not currently tracked. In particular, locations of reserved words and punctuation.

2. A top-level type parameter that allows a tool-writer to thread an application-specific annotation through the syntax tree.

These two could potentially be merged, if the default annotation parameter is capable of tracking the first category of information.

Discussion of the feature is at GhcAstAnnotations

New description: Add Annotations to HsSyn to explicitly track the locations of all non- blank source code elements to allow tools to parse a Haskell file, modify the AST and then produce an updated version of the source preserving the layout for unchanged parts. Discussion of the feature is at GhcAstAnnotations Note: an early effort was at Phab:D297, but this was abandoned as unworkable. -- -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9628#comment:13 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9628: Add Annotations to the AST to simplify source to source conversions -------------------------------------+------------------------------------- Reporter: alanz | Owner: alanz Type: feature | Status: new request | Milestone: Priority: normal | Version: 7.9 Component: Compiler | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Unknown Unknown/Multiple | Blocked By: Type of failure: | Related Tickets: None/Unknown | Test Case: | Blocking: | Differential Revisions: D297 | -------------------------------------+------------------------------------- Comment (by alanz): I have updated the Wiki page to reflect the current status, including an overview of the current design https://ghc.haskell.org/trac/ghc/wiki/GhcAstAnnotations#design I have also captured the summary status in GhcApiWorkInProgress. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9628#comment:15 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9628: Add Annotations to the AST to simplify source to source conversions -------------------------------------+------------------------------------- Reporter: alanz | Owner: alanz Type: feature | Status: new request | Milestone: Priority: normal | Version: 7.9 Component: Compiler | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Unknown Unknown/Multiple | Blocked By: Type of failure: | Related Tickets: None/Unknown | Test Case: | Blocking: | Differential Revisions: D297 | -------------------------------------+------------------------------------- Comment (by NeilMitchell): Alan and I debated this point, and I was the one who suggested the many data types approach (although I wouldn't go as far as saying I advocate it - I'm somewhat neutral). My concern was that if you do the dynamic approach you have to document each one of those strings, and the documentation has to be correct or the users won't have a clue what is going on, and if you update it all your clients will break and no one will realise. The documentation will basically be as long as the data types, but won't be statically checked on either side, and will very likely end up being wrong. Maybe a hybrid approach is to have location information for keywords like "of", "case", "where", "(" and ")". You can have a fixed static enumeration of keywords, so you have type safety, and every keyword is optionally in every AST. The number of keywords is relatively small, and which keyword is contained within each AST node doesn't require documenting (the lexical structure makes it pretty obvious). You end up with no requirement for fragile docs, one data type with a fairly small number of entries, and reasonable safety for both the producer and consumer. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9628#comment:17 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9628: Add Annotations to the AST to simplify source to source conversions -------------------------------------+------------------------------------- Reporter: alanz | Owner: alanz Type: feature | Status: new request | Milestone: Priority: normal | Version: 7.9 Component: Compiler | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Unknown Unknown/Multiple | Blocked By: Type of failure: | Related Tickets: None/Unknown | Test Case: | Blocking: | Differential Revisions: D297 | -------------------------------------+------------------------------------- Comment (by alanz): I share Neil's concern about the fragility of the string lookup. A user of this feature would have to keep track of the string key, as well as the shape of the returned structure used for the particular annotation, and would not know until runtime failure that there was a problem. I think the hybrid approach where there is a specific enumerated type for the keyword/feature being looked up could be workable, but requires the API user to consult the parser.y.pp file to see exactly what fits in where. From a user of the feature point of view, having a straightforward mapping from the constructor name to the annotation name which can then be interrogated via e.g. ghci is definitely beneficial. This does require an explosion of types, but this is pretty much a once- off exercise. Is there a code size problem if many classes are derived for each one? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9628#comment:19 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9628: Add Annotations to the AST to simplify source to source conversions -------------------------------------+------------------------------------- Reporter: alanz | Owner: alanz Type: feature | Status: new request | Milestone: Priority: normal | Version: 7.9 Component: Compiler | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Unknown Unknown/Multiple | Blocked By: Type of failure: | Related Tickets: None/Unknown | Test Case: | Blocking: | Differential Revisions: D297 | -------------------------------------+------------------------------------- Comment (by simonpj): I don't have data, but people are already complaining about the amount of code generated by data type declarations #9669. Have you counted how many data constructors there are in `HsSyn`? It's a LOT. It just feels like a sledgehammer to crack a nut. Simon -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9628#comment:21 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9628: Add Annotations to the AST to simplify source to source conversions -------------------------------------+------------------------------------- Reporter: alanz | Owner: alanz Type: feature | Status: new request | Milestone: Priority: normal | Version: 7.9 Component: Compiler | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Unknown Unknown/Multiple | Blocked By: Type of failure: | Related Tickets: None/Unknown | Test Case: | Blocking: | Differential Revisions: D297 | -------------------------------------+------------------------------------- Comment (by alanz): The multiple annotation approach seems promising, first stab results in Annotation types {{{ #!haskell data AnnModule = AnnModule SrcSpan deriving (Eq,Data,Typeable,Show) data AnnWhere = AnnWhere SrcSpan deriving (Eq,Data,Typeable,Show) -- Should this be AnnBraces? The tokens are ITocurly / ITccurly data AnnCurlies = AnnCurlies (SrcSpan,SrcSpan) deriving (Eq,Data,Typeable,Show) data AnnSemi = AnnSemi SrcSpan deriving (Eq,Data,Typeable,Show) data AnnComma = AnnComma SrcSpan deriving (Eq,Data,Typeable,Show) -- | Pragma declaration, e.g. '{-# SOURCE' '#-}' data AnnPragma = AnnPragma (SrcSpan,SrcSpan) deriving (Eq,Data,Typeable,Show) }}} Helper functions in parser {{{ #!haskell -- | Given a list of @Maybe annotation@, add the @Just@ ones to the -- given location ams :: Located a -> [Maybe (SrcSpan -> P ())] -> P (Located a) ams a@(L l _) bs = (mapM_ (\a -> a l) $ catMaybes bs) >> return a mj :: (Outputable a,Typeable a,Show a,Eq a) => (SrcSpan -> a) -> Located e -> Maybe (SrcSpan -> P ()) mj c l = Just (\s -> addAnnotation s (c (gl $l))) mm :: (Outputable a,Typeable a,Show a,Eq a) => (SrcSpan -> a) -> Maybe SrcSpan -> Maybe (SrcSpan -> P ()) mm c Nothing = Nothing mm c (Just l) = Just (\s -> addAnnotation s (c l)) mm2 :: (Outputable a,Typeable a,Show a,Eq a) => ((SrcSpan,SrcSpan) -> a) -> Maybe (SrcSpan,SrcSpan) -> Maybe (SrcSpan -> P ()) mm2 c Nothing = Nothing mm2 c (Just (l1,l2)) = Just (\s -> addAnnotation s (c (l1,l2))) }}} Example rule in parser {{{ module :: { Located (HsModule RdrName) } : maybedocheader 'module' modid maybemodwarning maybeexports 'where' body {% fileSrcSpan >>= \ loc -> ams (L loc (HsModule (Just $3) $5 (fst $ snd $7) (snd $ snd $7) $4 $1 ) ) [mj AnnModule $3, mj AnnWhere $6 ,mm2 AnnCurlies (fst $ fst $7),mm AnnSemi (snd $ fst $7)] } }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9628#comment:23 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9628: Add Annotations to the AST to simplify source to source conversions -------------------------------------+------------------------------------- Reporter: alanz | Owner: alanz Type: feature | Status: new request | Milestone: Priority: normal | Version: 7.9 Component: Compiler | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Unknown Unknown/Multiple | Blocked By: Type of failure: | Related Tickets: None/Unknown | Test Case: | Blocking: | Differential Revisions: D297 | -------------------------------------+------------------------------------- Comment (by alanz): That could work, and it is a lot simpler. The stuff for paired SrcSpans just doubles everything up at the moment, as I work it through. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9628#comment:25 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9628: Add Annotations to the AST to simplify source to source conversions -------------------------------------+------------------------------------- Reporter: alanz | Owner: alanz Type: feature | Status: new request | Milestone: Priority: normal | Version: 7.9 Component: Compiler | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Unknown Unknown/Multiple | Blocked By: Type of failure: | Related Tickets: None/Unknown | Test Case: | Blocking: | Differential Revisions: D297 | -------------------------------------+------------------------------------- Comment (by alanz): I updated the work in progress on D297 with this approach, and it feels much better, and is less invasive. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9628#comment:27 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9628: Add Annotations to the AST to simplify source to source conversions -------------------------------------+------------------------------------- Reporter: alanz | Owner: alanz Type: feature | Status: new request | Milestone: Priority: normal | Version: 7.9 Component: Compiler | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Unknown Unknown/Multiple | Blocked By: Type of failure: | Related Tickets: None/Unknown | Test Case: | Blocking: | Differential Revisions: D297 | -------------------------------------+------------------------------------- Comment (by simonpj): I suggest doing so only if the two can differ. In the case of `String` there can be string gaps, thus {{{ foo :: String foo = "blah blah\ \more blah blah\ \and more" }}} and I guess you want to have all that layout reproduced. Fine. But for integers like `3234242329423`, I don't see how the displayed form could differ. For `Words` perhaps there is binary/hex forms? Regardless, I'm not against this, but very keen that the reasons for keeping the two are documented on a per-literal basis, as I have begun to do above. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9628#comment:29 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9628: Add Annotations to the AST to simplify source to source conversions -------------------------------------+------------------------------------- Reporter: alanz | Owner: alanz Type: feature | Status: new request | Milestone: Priority: normal | Version: 7.9 Component: Compiler | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Unknown Unknown/Multiple | Blocked By: Type of failure: | Related Tickets: None/Unknown | Test Case: | Blocking: | Differential Revisions: D297 | -------------------------------------+------------------------------------- Comment (by alanz): I have updated the GhcAstAnnotations page to reflect the current design, as well as to add a section outlining an approach to comments, which is https://ghc.haskell.org/trac/ghc/wiki/GhcAstAnnotations#comments The comments have been added in Phab:D297, causing the file `ApiAnnotations` to be moved into `Lexer.x` -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9628#comment:31 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9628: Add Annotations to the AST to simplify source to source conversions -------------------------------------+------------------------------------- Reporter: alanz | Owner: alanz Type: feature | Status: new request | Milestone: Priority: normal | Version: 7.9 Component: Compiler | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Unknown Unknown/Multiple | Blocked By: Type of failure: | Related Tickets: None/Unknown | Test Case: | Blocking: | Differential Revisions: D297 | -------------------------------------+------------------------------------- Comment (by alanz): == Summary of AZ understanding of the landscape == === Pros and cons of current approach === Pros Very non-invasive Programs that don't need the information don't touch it; this seems like the strongest advantage Can also bring out comments, tied to smallest enclosing SrcSpan having an annotation. The comments can be in Haddock mode, to aid tool processing. This makes use of the standard Opt_Haddock and Opt_KeepRawTokenStream settings. Cons Parser.y.pp is more complex Actually using the annotations is more complex === Pros and cons of embedded annotations as per Richard Eisenberg === Pros The things you want are right there; no extra mappings to carry around, no lookups, no AnnColon2 stuff. Seems simple and uniform with the existing source locations. Eg. if a keyword 'do' could so be mdo, you might have a Located Bool to indicate. But if it's just do then we'd get Located (), and then we drop it altogether. The in-tree story amounts to not dropping it. Simpler parser cons Breaks lots of existing code, both in GHC and tools that make use of the AST. === Other thoughts === In terms of HaRe, the AST mainipulation will happen via ghc-exactprint. For this, the annotations are used as a starting point for processing. The mechanism intended in ghc-exactprint is to convert the absolute locations to relative locations, much as combinators in a pretty printer. This way when tooling modifies the AST, the output will honour the original layout as much as possible, given changes. Examples Lifting a declaration will preserve the layout of the original but dedented appropriately. Changing the name of an identifier, resulting in a longer or shorter name, just shifts things around naturally. {{{ foo x = x + y where fn a = a + 2 y = fn x }}} becomes {{{ foo xlong = xlong + y where fn a = a + 2 y = fn xlong }}} * The annotations can be easily accessed via a traversal of the appropriate kind. * There is a middle ground, where each AST element that has annotations tied to it in the current system has a single additional field containing [(SrcSpan,Ann]). Equally, the annotations retrieval function can provide all annotations attached to a given SrcSpan, via the current mechanism. * The most natural way to access the annotations is indirectly via a library such as HaRe or ghc-exactprint, as indicated by e.g. hlint, yi and IHaskell developers. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9628#comment:33 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9628: Add Annotations to the AST to simplify source to source conversions -------------------------------------+------------------------------------- Reporter: alanz | Owner: alanz Type: feature | Status: new request | Milestone: Priority: normal | Version: 7.9 Component: Compiler | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Unknown Unknown/Multiple | Blocked By: Type of failure: | Related Tickets: None/Unknown | Test Case: | Blocking: | Differential Revisions: D297 | -------------------------------------+------------------------------------- Comment (by simonpj): Fair enough in general. I've added some comments on Phab -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9628#comment:35 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9628: Add Annotations to the AST to simplify source to source conversions -------------------------------------+------------------------------------- Reporter: alanz | Owner: alanz Type: feature | Status: new request | Milestone: Priority: normal | Version: 7.9 Component: Compiler | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Unknown Unknown/Multiple | Blocked By: Type of failure: | Related Tickets: None/Unknown | Test Case: | Blocking: | Differential Revisions: D297 | -------------------------------------+------------------------------------- Comment (by alanz): I have updated Phab:D426, Phab:D438 and Phab:D412 to rebase against current master, and to work in the feedback received. They are once more ready for review. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9628#comment:37 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9628: Add Annotations to the AST to simplify source to source conversions -------------------------------------+------------------------------------- Reporter: alanz | Owner: alanz Type: feature | Status: new request | Milestone: Priority: normal | Version: 7.9 Component: Compiler | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Unknown Unknown/Multiple | Blocked By: Type of failure: | Related Tickets: None/Unknown | Test Case: | Blocking: | Differential Revisions: D297 | -------------------------------------+------------------------------------- Comment (by alanz): I have updated ​Phab:D426, ​Phab:D438 and ​Phab:D412 to include documentation about what annotations can be expected against each AST element, and also rebases against current master. The process of documentation showed up a number of shortcomings where annotations were missing or duplicated. I have also changed the `getAnnotation` result from `Maybe SrcSpan` to `[SrcSpan]` and allow multiple annotations of the same type on a given AST element. This means that the horrible `AnnColon2` and similar are now gone. The users of the annotations should be able to work out what is what from the surrounding context. They are once more ready for final review and hopefully merge for 7.10 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9628#comment:39 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9628: Add Annotations to the AST to simplify source to source conversions -------------------------------------+------------------------------------- Reporter: alanz | Owner: alanz Type: feature | Status: new request | Milestone: Priority: normal | Version: 7.9 Component: Compiler | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Unknown Unknown/Multiple | Blocked By: Type of failure: | Related Tickets: None/Unknown | Test Case: | Blocking: | Differential Revisions: D297 | -------------------------------------+------------------------------------- Comment (by Austin Seipp ): In [changeset:"803fc5db31f084b73713342cdceaed5a9c664267/ghc"]: {{{ #!CommitTicketReference repository="ghc" revision="803fc5db31f084b73713342cdceaed5a9c664267" Add API Annotations Summary: The final design and discussion is captured at https://ghc.haskell.org/trac/ghc/wiki/GhcAstAnnotations This is a proof of concept implementation of a completely separate annotation structure, populated in the parser,and tied to the AST by means of a virtual "node-key" comprising the surrounding SrcSpan and a value derived from the specific constructor used for the node. The key parts of the design are the following. == The Annotations == In `hsSyn/ApiAnnotation.hs` ```lang=haskell type ApiAnns = (Map.Map ApiAnnKey SrcSpan, Map.Map SrcSpan [Located Token]) type ApiAnnKey = (SrcSpan,AnnKeywordId) -- --------------------------------------------------------------------- -- | Retrieve an annotation based on the @SrcSpan@ of the annotated AST -- element, and the known type of the annotation. getAnnotation :: ApiAnns -> SrcSpan -> AnnKeywordId -> Maybe SrcSpan getAnnotation (anns,_) span ann = Map.lookup (span,ann) anns -- |Retrieve the comments allocated to the current @SrcSpan@ getAnnotationComments :: ApiAnns -> SrcSpan -> [Located Token] getAnnotationComments (_,anns) span = case Map.lookup span anns of Just cs -> cs Nothing -> [] -- | Note: in general the names of these are taken from the -- corresponding token, unless otherwise noted data AnnKeywordId = AnnAs | AnnBang | AnnClass | AnnClose -- ^ } or ] or ) or #) etc | AnnComma | AnnDarrow | AnnData | AnnDcolon .... ``` == Capturing in the lexer/parser == The annotations are captured in the lexer / parser by extending PState to include a field In `parser/Lexer.x` ```lang=haskell data PState = PState { .... annotations :: [(ApiAnnKey,SrcSpan)] -- Annotations giving the locations of 'noise' tokens in the -- source, so that users of the GHC API can do source to -- source conversions. } ``` The lexer exposes a helper function to add an annotation ```lang=haskell addAnnotation :: SrcSpan -> Ann -> SrcSpan -> P () addAnnotation l a v = P $ \s -> POk s { annotations = ((AK l a), v) : annotations s } () ``` The parser also has some helper functions of the form ```lang=haskell type MaybeAnn = Maybe (SrcSpan -> P ()) gl = getLoc gj x = Just (gl x) ams :: Located a -> [MaybeAnn] -> P (Located a) ams a@(L l _) bs = (mapM_ (\a -> a l) $ catMaybes bs) >> return a ``` This allows annotations to be captured in the parser by means of ``` ctypedoc :: { LHsType RdrName } : 'forall' tv_bndrs '.' ctypedoc {% hintExplicitForall (getLoc $1)

...

ams (LL $ mkExplicitHsForAllTy $2 (noLoc []) $4) [mj AnnForall $1,mj AnnDot $3] } | context '=>' ctypedoc {% ams (LL $ mkQualifiedHsForAllTy $1 $3) [mj AnnDarrow $2] } | ipvar '::' type {% ams (LL (HsIParamTy (unLoc $1) $3)) [mj AnnDcolon $2] } | typedoc { $1 }

== Parse result ==

```lang-haskell
data HsParsedModule = HsParsedModule {
    hpm_module    :: Located (HsModule RdrName),
    hpm_src_files :: [FilePath],
       -- ^ extra source files (e.g. from #includes).  The lexer collects
       -- these from '# <file> <line>' pragmas, which the C preprocessor
       -- leaves behind.  These files and their timestamps are stored in
       -- the .hi file, so that we can force recompilation if any of
       -- them change (#3589)
    hpm_annotations :: ApiAnns
  }

-- | The result of successful parsing.
data ParsedModule =
  ParsedModule { pm_mod_summary   :: ModSummary
               , pm_parsed_source :: ParsedSource
               , pm_extra_src_files :: [FilePath]
               , pm_annotations :: ApiAnns }

This diff depends on D426 Test Plan: sh ./validate Reviewers: austin, simonpj, Mikolaj Reviewed By: simonpj, Mikolaj Subscribers: Mikolaj, goldfire, thomie, carter Differential Revision: https://phabricator.haskell.org/D438 GHC Trac Issues: #9628 }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9628#comment:41 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#9628: Add Annotations to the AST to simplify source to source conversions -------------------------------------+------------------------------------- Reporter: alanz | Owner: alanz Type: feature | Status: closed request | Milestone: Priority: normal | Version: 7.9 Component: Compiler | Keywords: Resolution: fixed | Architecture: Unknown/Multiple Operating System: | Difficulty: Unknown Unknown/Multiple | Blocked By: Type of failure: | Related Tickets: None/Unknown | Test Case: | Blocking: | Differential Revisions: D297 | -------------------------------------+------------------------------------- Changes (by alanz): * status: new => closed * resolution: => fixed -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9628#comment:43 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler