Extracting representation from GHC

Dear GHC Developers, I would like to ask your opinion on my ideas to make it easier to use development tools with GHC. In the past when working on a Haskell refactoring tool I relied on using the GHC API for parsing and type checking Haskell sources. I extracted the representation and performed analysis and transformation on it as it was needed. However using the refactorer would be easier if it could work with build tools. To do this, my idea is to instruct GHC with a compilation flag to give out its internal representation of the source code. Most build tools let the user to configure the GHC flags so the refactoring tool would be usable in any build infrastructure. I'm thinking of using the pre-existing plugin architecture and adding two new fields to the Plugin datastructure. One would be called with the parsed representation (HsParsedModule) when parsing succeeds, another with the result of the type checking (TcGblEnv) when type checking is finished. What do you think about this solution? Boldizsár (ps: My first idea was using frontend plugins, but I could not access the representation from there and --frontend flag changed GHC compilation mode.)

I have too wanted this in the past and made a post to a similar effect
on the mailing list 6 months ago.
https://mail.haskell.org/pipermail/ghc-devs/2017-July/014427.html
It references this proposal for a similar feature.
https://ghc.haskell.org/trac/ghc/wiki/FrontendPluginsProposal#no1
If you would be glad to implement it then feel free to add me as a reviewer.
Cheers,
Matt
On Fri, Jan 19, 2018 at 9:35 AM, Németh Boldizsár
Dear GHC Developers,
I would like to ask your opinion on my ideas to make it easier to use development tools with GHC.
In the past when working on a Haskell refactoring tool I relied on using the GHC API for parsing and type checking Haskell sources. I extracted the representation and performed analysis and transformation on it as it was needed. However using the refactorer would be easier if it could work with build tools.
To do this, my idea is to instruct GHC with a compilation flag to give out its internal representation of the source code. Most build tools let the user to configure the GHC flags so the refactoring tool would be usable in any build infrastructure. I'm thinking of using the pre-existing plugin architecture and adding two new fields to the Plugin datastructure. One would be called with the parsed representation (HsParsedModule) when parsing succeeds, another with the result of the type checking (TcGblEnv) when type checking is finished.
What do you think about this solution?
Boldizsár
(ps: My first idea was using frontend plugins, but I could not access the representation from there and --frontend flag changed GHC compilation mode.)
_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Hi,
IIRC you can already use hscFrontendHook in the DynFlags hooks to retrieve
TcGblEnv, and with a little bit of work, also HsParsedModule.
Regards,
Shao Cheng
On Fri, Jan 19, 2018, 5:41 PM Matthew Pickering
I have too wanted this in the past and made a post to a similar effect on the mailing list 6 months ago.
https://mail.haskell.org/pipermail/ghc-devs/2017-July/014427.html
It references this proposal for a similar feature.
https://ghc.haskell.org/trac/ghc/wiki/FrontendPluginsProposal#no1
If you would be glad to implement it then feel free to add me as a reviewer.
Cheers,
Matt
Dear GHC Developers,
I would like to ask your opinion on my ideas to make it easier to use development tools with GHC.
In the past when working on a Haskell refactoring tool I relied on using
GHC API for parsing and type checking Haskell sources. I extracted the representation and performed analysis and transformation on it as it was needed. However using the refactorer would be easier if it could work with build tools.
To do this, my idea is to instruct GHC with a compilation flag to give out its internal representation of the source code. Most build tools let the user to configure the GHC flags so the refactoring tool would be usable in any build infrastructure. I'm thinking of using the pre-existing plugin architecture and adding two new fields to the Plugin datastructure. One would be called with the parsed representation (HsParsedModule) when
On Fri, Jan 19, 2018 at 9:35 AM, Németh Boldizsár
wrote: the parsing succeeds, another with the result of the type checking (TcGblEnv) when type checking is finished.
What do you think about this solution?
Boldizsár
(ps: My first idea was using frontend plugins, but I could not access the representation from there and --frontend flag changed GHC compilation mode.)
_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

| To do this, my idea is to instruct GHC with a compilation flag to give | out its internal representation of the source code. Why can't you just use GHC as a library, and ask it to parse and typecheck the module and then look at what it gives you. Others are more used to the GHC API than me, though. S | -----Original Message----- | From: ghc-devs [mailto:ghc-devs-bounces@haskell.org] On Behalf Of | Németh Boldizsár | Sent: 19 January 2018 09:35 | To: ghc-devs@haskell.org | Subject: Extracting representation from GHC | | Dear GHC Developers, | | I would like to ask your opinion on my ideas to make it easier to use | development tools with GHC. | | In the past when working on a Haskell refactoring tool I relied on | using the GHC API for parsing and type checking Haskell sources. I | extracted the representation and performed analysis and transformation | on it as it was needed. However using the refactorer would be easier | if it could work with build tools. | | To do this, my idea is to instruct GHC with a compilation flag to give | out its internal representation of the source code. Most build tools | let the user to configure the GHC flags so the refactoring tool would | be usable in any build infrastructure. I'm thinking of using the pre- | existing plugin architecture and adding two new fields to the Plugin | datastructure. One would be called with the parsed representation | (HsParsedModule) when parsing succeeds, another with the result of the | type checking (TcGblEnv) when type checking is finished. | | What do you think about this solution? | | Boldizsár | | (ps: My first idea was using frontend plugins, but I could not access | the representation from there and --frontend flag changed GHC | compilation mode.) | | _______________________________________________ | ghc-devs mailing list | ghc-devs@haskell.org | https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.h | askell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc- | devs&data=02%7C01%7Csimonpj%40microsoft.com%7C9d78fd2d16994ade4d9008d5 | 5f2007c3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6365195135360471 | 87&sdata=voUEz%2BKTp0p3CtwP1Hx6xA3cXN0qoYONLPd9T7xRve8%3D&reserved=0

See some additions inline. BR Robin 2018-01-19 18:27 GMT+01:00 Simon Peyton Jones via ghc-devs < ghc-devs@haskell.org>:
| To do this, my idea is to instruct GHC with a compilation flag to give | out its internal representation of the source code.
Why can't you just use GHC as a library, and ask it to parse and typecheck the module and then look at what it gives you.
Last time I checked (GHC 8.2, for haskell-indexer), using the library is not equivalent to using GHC's Main. GHC's Main does tremendous amount of magic with flag parsing and state setup, and doesn't expose all the functionality for libraries to do the same.
AFAIR we saw two possible ways to get the AST out from a complicated setup (FFI, objects, packages, ...): 1) invoke GHC and use Frontend plugin (but Frontend plugin is/was more limited at the time - the gist in the below trac entry mentions that even the Frontend plugin didn't do everything Main does). 2) Refactor GHC Main and expose all the functionality to GHC API. I filed https://ghc.haskell.org/trac/ghc/ticket/14018 a while ago that's slightly related. By the way, you can click around http://stuff.codereview.me/ghc/#ghc/ghc/Main.hs?corpus=ghc-8.2.1-rc2&signature in the 'main' function to see all the magic. Others are more used to the GHC API than me, though.
S
| -----Original Message----- | From: ghc-devs [mailto:ghc-devs-bounces@haskell.org] On Behalf Of | Németh Boldizsár | Sent: 19 January 2018 09:35 | To: ghc-devs@haskell.org | Subject: Extracting representation from GHC | | Dear GHC Developers, | | I would like to ask your opinion on my ideas to make it easier to use | development tools with GHC. | | In the past when working on a Haskell refactoring tool I relied on | using the GHC API for parsing and type checking Haskell sources. I | extracted the representation and performed analysis and transformation | on it as it was needed. However using the refactorer would be easier | if it could work with build tools. | | To do this, my idea is to instruct GHC with a compilation flag to give | out its internal representation of the source code. Most build tools | let the user to configure the GHC flags so the refactoring tool would | be usable in any build infrastructure. I'm thinking of using the pre- | existing plugin architecture and adding two new fields to the Plugin | datastructure. One would be called with the parsed representation | (HsParsedModule) when parsing succeeds, another with the result of the | type checking (TcGblEnv) when type checking is finished. | | What do you think about this solution? | | Boldizsár | | (ps: My first idea was using frontend plugins, but I could not access | the representation from there and --frontend flag changed GHC | compilation mode.) | | _______________________________________________ | ghc-devs mailing list | ghc-devs@haskell.org | https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.h | askell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc- | devs&data=02%7C01%7Csimonpj%40microsoft.com%7C9d78fd2d16994ade4d9008d5 | 5f2007c3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6365195135360471 | 87&sdata=voUEz%2BKTp0p3CtwP1Hx6xA3cXN0qoYONLPd9T7xRve8%3D&reserved=0 _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

See also
https://github.com/google/haskell-indexer/blob/master/haskell-indexer-backen...
for an as-complete GHC API based setup as I could get. The comments
indicate possible deficiencies.
The test
https://github.com/google/haskell-indexer/blob/master/haskell-indexer-backen...
shows some cases that are covered (for example testTemplateHaskellCodeExecFFI
).
In practice one can run AST extraction with HscNoLink + HscInterpreted for
most targets, but for hairy ones (FFI invoked from TemplateHaskell in
certain ways, for examples) that will fail.
2018-01-19 19:46 GMT+01:00 Robin Palotai
See some additions inline. BR Robin
2018-01-19 18:27 GMT+01:00 Simon Peyton Jones via ghc-devs < ghc-devs@haskell.org>:
| To do this, my idea is to instruct GHC with a compilation flag to give | out its internal representation of the source code.
Why can't you just use GHC as a library, and ask it to parse and typecheck the module and then look at what it gives you.
Last time I checked (GHC 8.2, for haskell-indexer), using the library is not equivalent to using GHC's Main. GHC's Main does tremendous amount of magic with flag parsing and state setup, and doesn't expose all the functionality for libraries to do the same.
AFAIR we saw two possible ways to get the AST out from a complicated setup (FFI, objects, packages, ...): 1) invoke GHC and use Frontend plugin (but Frontend plugin is/was more limited at the time - the gist in the below trac entry mentions that even the Frontend plugin didn't do everything Main does). 2) Refactor GHC Main and expose all the functionality to GHC API.
I filed https://ghc.haskell.org/trac/ghc/ticket/14018 a while ago that's slightly related.
By the way, you can click around http://stuff.codereview.me/ghc/#ghc/ghc/ Main.hs?corpus=ghc-8.2.1-rc2&signature in the 'main' function to see all the magic.
Others are more used to the GHC API than me, though.
S
| -----Original Message----- | From: ghc-devs [mailto:ghc-devs-bounces@haskell.org] On Behalf Of | Németh Boldizsár | Sent: 19 January 2018 09:35 | To: ghc-devs@haskell.org | Subject: Extracting representation from GHC | | Dear GHC Developers, | | I would like to ask your opinion on my ideas to make it easier to use | development tools with GHC. | | In the past when working on a Haskell refactoring tool I relied on | using the GHC API for parsing and type checking Haskell sources. I | extracted the representation and performed analysis and transformation | on it as it was needed. However using the refactorer would be easier | if it could work with build tools. | | To do this, my idea is to instruct GHC with a compilation flag to give | out its internal representation of the source code. Most build tools | let the user to configure the GHC flags so the refactoring tool would | be usable in any build infrastructure. I'm thinking of using the pre- | existing plugin architecture and adding two new fields to the Plugin | datastructure. One would be called with the parsed representation | (HsParsedModule) when parsing succeeds, another with the result of the | type checking (TcGblEnv) when type checking is finished. | | What do you think about this solution? | | Boldizsár | | (ps: My first idea was using frontend plugins, but I could not access | the representation from there and --frontend flag changed GHC | compilation mode.) | | _______________________________________________ | ghc-devs mailing list | ghc-devs@haskell.org | https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.h | askell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc- | devs&data=02%7C01%7Csimonpj%40microsoft.com%7C9d78fd2d16994ade4d9008d5 | 5f2007c3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6365195135360471 <(513)%20536-0471> | 87&sdata=voUEz%2BKTp0p3CtwP1Hx6xA3cXN0qoYONLPd9T7xRve8%3D&reserved=0 _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
participants (5)
-
Matthew Pickering
-
Németh Boldizsár
-
Robin Palotai
-
Shao Cheng
-
Simon Peyton Jones