TTG hsSyn for Batch and Interactive Parsing - ghc-devs - Haskell.org

newer
Re: Add HeapView functionality...

TTG hsSyn for Batch and Interactive Parsing

older
Phabricator reboot

Alan & Kim Zimmerman

7 May 2018 7 May '18

11:16 a.m.

I want to be able to run the GHC parser in one of two modes, batch which functions as now, and interactive which will (eventually) be incremental. In addition, the hsSyn AST for each will have different TTG[1] annotations, so that it can better support IDE usage. I think this can be done by changing the types in HsExtension to introduce a 'Process' type as follows data Pass = Parsed Process | Renamed | Typechecked deriving (Data) data Process = Batch | Interactive deriving (Show, Data) We then rename the pass synonyms so that batch is the default type GhcPs = GhcPass ('Parsed 'Batch) type GhcPsI = GhcPass ('Parsed 'Interactive) I have attached a simple proof of concept file, which emulates parsing and renaming. Is this an appropriate approach to take? Alan [1] Trees That Grow https://ghc.haskell.org/trac/ghc/wiki/ ImplementingTreesThatGrow

Attachments:

attachment.html (text/html — 1.2 KB)
ParserTypes.hs (text/x-haskell — 2.2 KB)

Reply

Sign in to reply online Use email software

Show replies by date

Simon Peyton Jones

8 May 8 May

4:54 a.m.

At first blush, “running the parser in two modes” and “changing the Pass” type don’t match up in my mind. One seems quite local (how to run the parser). The other seems more pervasive. Can you say more about your proposed design, perhaps even on a wiki page? Simon From: ghc-devs On Behalf Of Alan & Kim Zimmerman Sent: 07 May 2018 16:17 To: ghc-devs Subject: TTG hsSyn for Batch and Interactive Parsing I want to be able to run the GHC parser in one of two modes, batch which functions as now, and interactive which will (eventually) be incremental. In addition, the hsSyn AST for each will have different TTG[1] annotations, so that it can better support IDE usage. I think this can be done by changing the types in HsExtension to introduce a 'Process' type as follows data Pass = Parsed Process | Renamed | Typechecked deriving (Data) data Process = Batch | Interactive deriving (Show, Data) We then rename the pass synonyms so that batch is the default type GhcPs = GhcPass ('Parsed 'Batch) type GhcPsI = GhcPass ('Parsed 'Interactive) I have attached a simple proof of concept file, which emulates parsing and renaming. Is this an appropriate approach to take? Alan [1] Trees That Grow https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow

Reply

Sign in to reply online Use email software

Alan & Kim Zimmerman

4:02 p.m.

I have started a wiki page at https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow/IdeSupport On 8 May 2018 at 10:54, Simon Peyton Jones wrote:

At first blush, “running the parser in two modes” and “changing the Pass” type don’t match up in my mind. One seems quite local (how to run the parser). The other seems more pervasive.

Can you say more about your proposed design, perhaps even on a wiki page?

Simon

*From:* ghc-devs *On Behalf Of *Alan & Kim Zimmerman *Sent:* 07 May 2018 16:17 *To:* ghc-devs *Subject:* TTG hsSyn for Batch and Interactive Parsing

I want to be able to run the GHC parser in one of two modes, batch which functions as now, and interactive which will (eventually) be incremental.

In addition, the hsSyn AST for each will have different TTG[1] annotations, so that it can better support IDE usage.

I think this can be done by changing the types in HsExtension to introduce a 'Process' type as follows

data Pass = Parsed Process | Renamed | Typechecked deriving (Data)

data Process = Batch | Interactive deriving (Show, Data)

We then rename the pass synonyms so that batch is the default

type GhcPs = GhcPass ('Parsed 'Batch) type GhcPsI = GhcPass ('Parsed 'Interactive)

I have attached a simple proof of concept file, which emulates parsing and renaming.

Is this an appropriate approach to take?

Alan

[1] Trees That Grow https://ghc.haskell.org/trac/ghc/wiki/ ImplementingTreesThatGrow

Reply

Sign in to reply online Use email software

Simon Peyton Jones

9 May 9 May

4:15 a.m.

Thanks. I am absolutely behind this objective: I propose to move the API Annotations to where they belong, inside the AST. Indeed I thought that was always part of the TTG plan. But I don’t understand what this has to do with interactive vs batch parsing. Why don’t you unconditionally retain API-annotation info? How would GhcPs be used differently to GhcPsI? You might want to answer by clarifying on the wiki page, so that it is a persistent record of the design debugged in dialogue by email. Simon From: Alan & Kim Zimmerman Sent: 08 May 2018 21:02 To: Simon Peyton Jones Cc: ghc-devs Subject: Re: TTG hsSyn for Batch and Interactive Parsing I have started a wiki page at https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow/IdeSupport On 8 May 2018 at 10:54, Simon Peyton Jones mailto:simonpj@microsoft.com> wrote: At first blush, “running the parser in two modes” and “changing the Pass” type don’t match up in my mind. One seems quite local (how to run the parser). The other seems more pervasive. Can you say more about your proposed design, perhaps even on a wiki page? Simon From: ghc-devs mailto:ghc-devs-bounces@haskell.org> On Behalf Of Alan & Kim Zimmerman Sent: 07 May 2018 16:17 To: ghc-devs mailto:ghc-devs@haskell.org> Subject: TTG hsSyn for Batch and Interactive Parsing I want to be able to run the GHC parser in one of two modes, batch which functions as now, and interactive which will (eventually) be incremental. In addition, the hsSyn AST for each will have different TTG[1] annotations, so that it can better support IDE usage. I think this can be done by changing the types in HsExtension to introduce a 'Process' type as follows data Pass = Parsed Process | Renamed | Typechecked deriving (Data) data Process = Batch | Interactive deriving (Show, Data) We then rename the pass synonyms so that batch is the default type GhcPs = GhcPass ('Parsed 'Batch) type GhcPsI = GhcPass ('Parsed 'Interactive) I have attached a simple proof of concept file, which emulates parsing and renaming. Is this an appropriate approach to take? Alan [1] Trees That Grow https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow

Reply

Sign in to reply online Use email software

Alan & Kim Zimmerman

4:12 p.m.

I have updated the Wiki. On 9 May 2018 at 10:15, Simon Peyton Jones wrote:

Thanks.

I am absolutely behind this objective:

I propose to move the API Annotations to where they belong, inside the AST.

Indeed I thought that was always part of the TTG plan.

*But I don’t understand what this has to do with interactive vs batch parsing. *Why don’t you unconditionally retain API-annotation info? How would GhcPs be used differently to GhcPsI?

You might want to answer by clarifying on the wiki page, so that it is a persistent record of the design debugged in dialogue by email.

Simon

*From:* Alan & Kim Zimmerman *Sent:* 08 May 2018 21:02 *To:* Simon Peyton Jones *Cc:* ghc-devs *Subject:* Re: TTG hsSyn for Batch and Interactive Parsing

I have started a wiki page at https://ghc.haskell.org/trac/ghc/wiki/ ImplementingTreesThatGrow/IdeSupport

On 8 May 2018 at 10:54, Simon Peyton Jones wrote:

At first blush, “running the parser in two modes” and “changing the Pass” type don’t match up in my mind. One seems quite local (how to run the parser). The other seems more pervasive.

Can you say more about your proposed design, perhaps even on a wiki page?

Simon

*From:* ghc-devs *On Behalf Of *Alan & Kim Zimmerman *Sent:* 07 May 2018 16:17 *To:* ghc-devs *Subject:* TTG hsSyn for Batch and Interactive Parsing

I want to be able to run the GHC parser in one of two modes, batch which functions as now, and interactive which will (eventually) be incremental.

In addition, the hsSyn AST for each will have different TTG[1] annotations, so that it can better support IDE usage.

I think this can be done by changing the types in HsExtension to introduce a 'Process' type as follows

data Pass = Parsed Process | Renamed | Typechecked deriving (Data)

data Process = Batch | Interactive deriving (Show, Data)

We then rename the pass synonyms so that batch is the default

type GhcPs = GhcPass ('Parsed 'Batch) type GhcPsI = GhcPass ('Parsed 'Interactive)

I have attached a simple proof of concept file, which emulates parsing and renaming.

Is this an appropriate approach to take?

Alan

[1] Trees That Grow https://ghc.haskell.org/trac/ghc/wiki/ ImplementingTreesThatGrow

Reply

Sign in to reply online Use email software

Simon Peyton Jones

10 May 10 May

10:31 a.m.

Thanks. But I still don’t see the connection with “interactive”. Why should maintaining API annotations have anything to do with interactivity? Maybe data Process = WithApiAnnotations | WithoutApiAnnotations I understand you want two different variants of the syntax tree, but I don’t understand what functions might produce or consume them. In particular does the parser produce (HsSyn (GhcPs WithApiAnnotations)) or without? Simon From: Alan & Kim Zimmerman Sent: 09 May 2018 21:12 To: Simon Peyton Jones Cc: ghc-devs Subject: Re: TTG hsSyn for Batch and Interactive Parsing I have updated the Wiki. On 9 May 2018 at 10:15, Simon Peyton Jones mailto:simonpj@microsoft.com> wrote: Thanks. I am absolutely behind this objective: I propose to move the API Annotations to where they belong, inside the AST. Indeed I thought that was always part of the TTG plan. But I don’t understand what this has to do with interactive vs batch parsing. Why don’t you unconditionally retain API-annotation info? How would GhcPs be used differently to GhcPsI? You might want to answer by clarifying on the wiki page, so that it is a persistent record of the design debugged in dialogue by email. Simon From: Alan & Kim Zimmerman mailto:alan.zimm@gmail.com> Sent: 08 May 2018 21:02 To: Simon Peyton Jones mailto:simonpj@microsoft.com> Cc: ghc-devs mailto:ghc-devs@haskell.org> Subject: Re: TTG hsSyn for Batch and Interactive Parsing I have started a wiki page at https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow/IdeSupport On 8 May 2018 at 10:54, Simon Peyton Jones mailto:simonpj@microsoft.com> wrote: At first blush, “running the parser in two modes” and “changing the Pass” type don’t match up in my mind. One seems quite local (how to run the parser). The other seems more pervasive. Can you say more about your proposed design, perhaps even on a wiki page? Simon From: ghc-devs mailto:ghc-devs-bounces@haskell.org> On Behalf Of Alan & Kim Zimmerman Sent: 07 May 2018 16:17 To: ghc-devs mailto:ghc-devs@haskell.org> Subject: TTG hsSyn for Batch and Interactive Parsing I want to be able to run the GHC parser in one of two modes, batch which functions as now, and interactive which will (eventually) be incremental. In addition, the hsSyn AST for each will have different TTG[1] annotations, so that it can better support IDE usage. I think this can be done by changing the types in HsExtension to introduce a 'Process' type as follows data Pass = Parsed Process | Renamed | Typechecked deriving (Data) data Process = Batch | Interactive deriving (Show, Data) We then rename the pass synonyms so that batch is the default type GhcPs = GhcPass ('Parsed 'Batch) type GhcPsI = GhcPass ('Parsed 'Interactive) I have attached a simple proof of concept file, which emulates parsing and renaming. Is this an appropriate approach to take? Alan [1] Trees That Grow https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow

Reply

Sign in to reply online Use email software

Alan & Kim Zimmerman

5:35 p.m.

I have updated the Wiki with the clearer names, and noted that a single parser definition would still be used, as at present, but would only keep the extra info if it was requested to. The naming around interactive and batch is to anticipate a further step I would like to take, to make the parser fully incremental, in the sense that it would process as input the prior parse tree and a list of changes to the source, and then generate a fresh parse tree, with the changed nodes marked. This mode would be tightly coupled to an external too like haskell-ide-engine, to manage the bookkeeping around this. My thinking for this is to use the approach presented in the paper "Efficient and Flexible Incremental Parsing" by Wagner and Graham[1]. The plan is to modify `happy`, so that we can reuse the existing GHC Parser.y with minor modifications. This is the same approach as used in the library tree-sitter[2], which is a very active project on github. WIP is at [3], but it is very early stage. Regards Alan [1] https://pdfs.semanticscholar.org/4d22/fab95c78b3c23fa9dff88fb82976ed c213c2.pdf [2] https://github.com/tree-sitter/tree-sitter [3] https://github.com/alanz/happy/tree/incremental

Reply

Sign in to reply online Use email software

Simon Peyton Jones

18 May 18 May

10:13 a.m.

This sounds fine. But I still don’t understand There would still be a single parser definition in Parser.y, which would make use of functions to add the additional info to the generated source tree, which would be NOPs if the information was not being kept. This is similar to what happens at present with the Api Annotations. What is the type of the parser? Does it produce parser :: String -> HsSyn (GhcPass (Parsed WithApiAnnotation) or parser :: String -> HsSyn (GhcPass (Parsed WithoutApiAnnotation) ? We can’t make the result type depend on DynFlags! (Yet)( parser :: DynFlags -> String -> HsSyn (GhcPass (Parsed (if … then WithApiAnnotations else WihoutsApiAnnotations) So I’m puzzled. Nomenclature. I’d say “NoApiAnnotations” rather than “WithoutApiAnnotations”. Also: do we have data to show that it’s not OK to always keep API annotations. That would be simpler, wouldn’t it. Incidentally the Haddock stuff, decorations of type (Maybe LHsDocString) somehow belong in this world too, don’t they? SImon From: Alan & Kim Zimmerman Sent: 10 May 2018 22:35 To: Simon Peyton Jones Cc: ghc-devs Subject: Re: TTG hsSyn for Batch and Interactive Parsing I have updated the Wiki with the clearer names, and noted that a single parser definition would still be used, as at present, but would only keep the extra info if it was requested to. The naming around interactive and batch is to anticipate a further step I would like to take, to make the parser fully incremental, in the sense that it would process as input the prior parse tree and a list of changes to the source, and then generate a fresh parse tree, with the changed nodes marked. This mode would be tightly coupled to an external too like haskell-ide-engine, to manage the bookkeeping around this. My thinking for this is to use the approach presented in the paper "Efficient and Flexible Incremental Parsing" by Wagner and Graham[1]. The plan is to modify `happy`, so that we can reuse the existing GHC Parser.y with minor modifications. This is the same approach as used in the library tree-sitter[2], which is a very active project on github. WIP is at [3], but it is very early stage. Regards Alan [1] https://pdfs.semanticscholar.org/4d22/fab95c78b3c23fa9dff88fb82976edc213c2.p...https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpdfs.semanticscholar.org%2F4d22%2Ffab95c78b3c23fa9dff88fb82976edc213c2.pdf&data=02%7C01%7Csimonpj%40microsoft.com%7C186cfaf7a10a4663ce2f08d5b6bdf0f7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636615849263176553&sdata=itxDiaFAuZY4MH1oZDKOudKLNcsqrm7cywA9vt4Y1DI%3D&reserved=0 [2] https://github.com/tree-sitter/tree-sitter https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftree-sitter%2Ftree-sitter&data=02%7C01%7Csimonpj%40microsoft.com%7C186cfaf7a10a4663ce2f08d5b6bdf0f7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636615849263176553&sdata=HmYE2FMO97ouuAR1vQHPfaEooI1oxtcU4aNWNg3E9T8%3D&reserved=0 [3] https://github.com/alanz/happy/tree/incremental https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Falanz%2Fhappy%2Ftree%2Fincremental&data=02%7C01%7Csimonpj%40microsoft.com%7C186cfaf7a10a4663ce2f08d5b6bdf0f7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636615849263332819&sdata=3RcrS%2FMH53aENrS%2FFmdKt%2Fc9JUjIeEq6h5xugF7HmhU%3D&reserved=0

Reply

Sign in to reply online Use email software

Alan & Kim Zimmerman

10:31 a.m.

On 18 May 2018 at 16:13, Simon Peyton Jones wrote:

We can’t make the result type depend on DynFlags! (Yet)(

* parser :: DynFlags -> String *

* -> HsSyn (GhcPass (Parsed (if … *

* then WithApiAnnotations*

* else WihoutsApiAnnotations)*

We could conceptually have parser :: DynFlags -> String -> Either (HsSyn (Parsed WithApiAnnotations)) (HsSyn (Parsed NoApiAnnotations)) The main point is that the next phase can make use of either of the variants. And it may be simplest to just always use the annotations, the ParsedSource is normally discarded after renaming anyway. Alan

Reply

Sign in to reply online Use email software

Simon Peyton Jones

10:57 a.m.

If we are always going to generate a parse tree with annotations from the parser, let’s not generate two! I’m fine with always generating the annotations, but we just need to check that it doesn’t have insupportable costs. Simon From: Alan & Kim Zimmerman Sent: 18 May 2018 15:31 To: Simon Peyton Jones Cc: ghc-devs Subject: Re: TTG hsSyn for Batch and Interactive Parsing On 18 May 2018 at 16:13, Simon Peyton Jones mailto:simonpj@microsoft.com> wrote: We can’t make the result type depend on DynFlags! (Yet)( parser :: DynFlags -> String -> HsSyn (GhcPass (Parsed (if … then WithApiAnnotations else WihoutsApiAnnotations) We could conceptually have parser :: DynFlags -> String -> Either (HsSyn (Parsed WithApiAnnotations)) (HsSyn (Parsed NoApiAnnotations)) The main point is that the next phase can make use of either of the variants. And it may be simplest to just always use the annotations, the ParsedSource is normally discarded after renaming anyway. Alan

Reply

Sign in to reply online Use email software

2559

Age (days ago)

2570

Last active (days ago)

Download

9 comments

2 participants

tags

participants (2)

Alan & Kim Zimmerman
Simon Peyton Jones