[GHC] #14709: Extend the plugin mechanism to access program representation
#14709: Extend the plugin mechanism to access program representation -------------------------------------+------------------------------------- Reporter: lazac | Owner: (none) Type: feature | Status: new request | Priority: normal | Milestone: Component: GHC API | Version: 8.2.2 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: https://phabricator.haskell.org/D4342| https://ghc.haskell.org/trac/ghc/wiki/ExtendedPluginsProposal -------------------------------------+------------------------------------- This would enable development tools to access the GHC representation in the pre-existing build environment. By the inversion of control, the tool developers don't need to decide which Haskell modules have to be processed and with what configuration, because the normal build procedure could be invoked by the user with plugin flags specifying what tools to invoke. The plan is described in more details at [https://ghc.haskell.org/trac/ghc/wiki/ExtendedPluginsProposal Extended Plugins Proposal] -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14709 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14709: Extend the plugin mechanism to access program representation -------------------------------------+------------------------------------- Reporter: lazac | Owner: (none) Type: feature request | Status: new Priority: normal | Milestone: Component: GHC API | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | https://phabricator.haskell.org/D4342 https://ghc.haskell.org/trac/ghc/wiki/ExtendedPluginsProposal| -------------------------------------+------------------------------------- Changes (by bgamari): * cc: ezyang, alanz (added) Comment: Hmmm, interesting. AddI guess ezyang and Alan Zimmerman as I know they have both thought a fair amount about tooling. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14709#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14709: Extend the plugin mechanism to access program representation -------------------------------------+------------------------------------- Reporter: lazac | Owner: (none) Type: feature request | Status: new Priority: normal | Milestone: Component: GHC API | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | https://phabricator.haskell.org/D4342 https://ghc.haskell.org/trac/ghc/wiki/ExtendedPluginsProposal| -------------------------------------+------------------------------------- Changes (by lazac): * cc: mpickering (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14709#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14709: Extend the plugin mechanism to access program representation -------------------------------------+------------------------------------- Reporter: lazac | Owner: (none) Type: feature request | Status: new Priority: normal | Milestone: Component: GHC API | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | https://phabricator.haskell.org/D4342 https://ghc.haskell.org/trac/ghc/wiki/ExtendedPluginsProposal| -------------------------------------+------------------------------------- Comment (by bgamari): The description on the wiki page, specifically of the plugin interface itself, seems a bit terse. Do you think you could lay out more precisely what these new plug-ins will be be able to do? Do you anticipate being able to modify the AST or is this strictly for tooling that wishes to I spect the AST? A few more words on why front-end plug-ins are insufficient would also be helpful; is the problem that you want your plugin to run during the course of normal compilation? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14709#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14709: Extend the plugin mechanism to access program representation -------------------------------------+------------------------------------- Reporter: lazac | Owner: (none) Type: feature request | Status: new Priority: normal | Milestone: Component: GHC API | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | https://phabricator.haskell.org/D4342 https://ghc.haskell.org/trac/ghc/wiki/ExtendedPluginsProposal| -------------------------------------+------------------------------------- Comment (by lazac): I can imagine tools that modify these representations as well. The main purpose of the plugins being able to change the input is that it seemed a bit arbitrary to limit the plugins in this way. This is also present in Edsko's version. Yes, basically I would like to be able to run my tool during a normal (enhanced) compilation. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14709#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14709: Extend the plugin mechanism to access program representation -------------------------------------+------------------------------------- Reporter: lazac | Owner: (none) Type: feature request | Status: new Priority: normal | Milestone: Component: GHC API | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | https://phabricator.haskell.org/D4342 https://ghc.haskell.org/trac/ghc/wiki/ExtendedPluginsProposal| -------------------------------------+------------------------------------- Comment (by bgamari):
I can imagine tools that modify these representations as well
Sure. The one thing I would really like to see considered is recompilation checking. Currently the existing plugin types are borderline unusable due to the lack of recompilation avoidance (see #7414). While you don't need to solve this issue in your patch (doing so will likely require deeper changes in GHC), we should try to think through which information the plugin will need to provide for the compiler to be able to assess whether recompilation is necessary and design the interface with this need in mind. This will avoid backwards-incompatible interface changes in the future. Writing down a few motivating examples of applications of the new plugin types may be a good place to start. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14709#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14709: Extend the plugin mechanism to access program representation -------------------------------------+------------------------------------- Reporter: lazac | Owner: (none) Type: feature request | Status: new Priority: normal | Milestone: Component: GHC API | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | https://phabricator.haskell.org/D4342 https://ghc.haskell.org/trac/ghc/wiki/ExtendedPluginsProposal| -------------------------------------+------------------------------------- Comment (by lazac): The plugin should be able to tell if it requires recompilation due to change in plugin arguments or external factors, and GHC should be aware when the plugin itself changes. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14709#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
Frontend plugins are not applicable, because their usage changes the major mode of the compiler. So if the tool developer wants to go on with
#14709: Extend the plugin mechanism to access program representation -------------------------------------+------------------------------------- Reporter: lazac | Owner: (none) Type: feature request | Status: new Priority: normal | Milestone: Component: GHC API | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | https://phabricator.haskell.org/D4342 https://ghc.haskell.org/trac/ghc/wiki/ExtendedPluginsProposal| -------------------------------------+------------------------------------- Comment (by ezyang): This doesn't really have anything to do with the relative merits/demerits of the proposal, but I just wanted to point out: the compilation procedure, he must replicate what GHC would do if the frontend plugin was not used. Furthermore, it can't be inserted into a normal build environment, since using the --frontend flag clashes with other mode flags like --make or --interactive. Out of the box these statements are true, but as I mention in http://blog.ezyang.com/2017/02/how-to-integrate-ghc-api-programs-with- cabal/ there are some relatively simple tricks you can play to solve this problem. As another example, https://github.com/ezyang/ghc-shake comes with a wrapper program which overrides the meaning of `--make` but otherwise keeps all other major modes the same. Of course, if the frontend plugin is replacing a mode like `--make`, it needs to understand all of the same arguments that `--make` would have understood, but that is what you would expect, no? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14709#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14709: Extend the plugin mechanism to access program representation -------------------------------------+------------------------------------- Reporter: lazac | Owner: (none) Type: feature request | Status: new Priority: normal | Milestone: Component: GHC API | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | https://phabricator.haskell.org/D4342 https://ghc.haskell.org/trac/ghc/wiki/ExtendedPluginsProposal| -------------------------------------+------------------------------------- Comment (by lazac): Your tutorial is very useful, I also tried it when I was experimenting of using frontend plugins for the same problem that I would like to solve with the source plugins. I think it is possible to extract the typechecked AST with a frontend plugin and a wrapper program setting up the frontend compiler hook. However for my tool I need other information, including loaded interfaces, renamed AST and splices evaluated. I think other tool developers will find the availability of those useful as well. From a design perspective, I would like to extend GHC in a way that doesn't alter the compilation process, and I think that normal plugins are better for that than frontend plugins. It is also generally easier to setup the build system to use a few compiler flags than to use a different compiler, and the user will not have problems if he has more than one GHC installed. Overall you are right in that the wiki page oversimplifies the matter, I will try to improve it. Thank for pointing out! -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14709#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14709: Extend the plugin mechanism to access program representation -------------------------------------+------------------------------------- Reporter: lazac | Owner: (none) Type: feature request | Status: new Priority: normal | Milestone: Component: GHC API | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | https://phabricator.haskell.org/D4342 https://ghc.haskell.org/trac/ghc/wiki/ExtendedPluginsProposal| -------------------------------------+------------------------------------- Comment (by lazac): Replying to [comment:5 bgamari]:
Writing down a few motivating examples of applications of the new plugin types may be a good place to start.
Some of these tools exist using external libraries, but by using GHC as the backend they may be applied to more complex projects. I'm mostly interested in editing and code transformation tools, so here is a few example: - **Auto-complete**: Records the names available in a scope using interface files and type checked AST. - **Semantics aware formatting**: Auto-formatter that uses fixity information to break up complex operator expressions. Uses parsed and type checked representation. - **Semantics aware syntax highlight**: Check interfaces for recording extra information about names and use this in an editor to enhance highlighting names. - **Code generator**: Generating pattern matches and expressions in a dependent-typed fashion. - **Auto-correct**: Automatic fix common errors. Uses `-fdefer-*` flags, extracts type-checked representation. Also checks interfaces files. - **Refactoring tool**: Building a project database of syntactic and semantic information to rewrite source code. Uses all versions of syntax tree, splices, interfaces. My suggestion for recompilation checking is: - `-fplugin` change -> recompile - The hash of used plugin changes -> recompile - `-fplugin-opt` change -> ask the plugin for each module whether recompilation is needed -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14709#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14709: Extend the plugin mechanism to access program representation -------------------------------------+------------------------------------- Reporter: lazac | Owner: (none) Type: feature request | Status: new Priority: normal | Milestone: Component: GHC API | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | https://phabricator.haskell.org/D4342 https://ghc.haskell.org/trac/ghc/wiki/ExtendedPluginsProposal| -------------------------------------+------------------------------------- Comment (by mpickering): These examples are useful for discussing the problems with recompilation but I don't think they should stop the progress of this patch. If we all agree that this is a good direction, which I think we should, we should consider each of the proposed plugins and decide whether each of them is desirable in turn. To summarise, here are the five proposed extensions. {{{ parsedResultAction :: [CommandLineOption] -> ModSummary -> HsParsedModule needsRenamedSyntax :: [CommandLineOption] -> ModSummary -> Hsc Bool typeCheckResultAction :: [CommandLineOption] -> ModSummary -> TcGblEnv spliceRunAction :: [CommandLineOption] -> LHsExpr GhcTc -> TcM (LHsExpr GhcTc) interfaceLoadAction :: forall lcl . [CommandLineOption] -> ModIface -> IfM lcl ModIface }}} I think that `parsedResultAction`, `typecheckResultAction` and `splitRunAction` are well motivated hooks which we should definitely include. I am indifferent towards `interfaceLoadAction`, I don't see a more direct place to implement this but feel the hook should instead run after the interface is type checked. I won't accept a patch which includes the `needsRenamedSyntax` hook which is ad-hoc. Can other interested parties please comment on these specific so we can move this patch forward in a reasonable time frame? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14709#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14709: Extend the plugin mechanism to access program representation -------------------------------------+------------------------------------- Reporter: lazac | Owner: (none) Type: feature request | Status: new Priority: normal | Milestone: Component: GHC API | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | https://phabricator.haskell.org/D4342 https://ghc.haskell.org/trac/ghc/wiki/ExtendedPluginsProposal| -------------------------------------+------------------------------------- Comment (by alanz): As a fellow tool-writer, in principle I am in favour of this patch. One thing that confuses me though, is the motivating examples are to extract information, but the plugins allow changes on the fly. I am concerned that we open a pandora's box of arbitrary transformations via plugins, that will make it ***harder*** for tool writers, as we now need to know what other things are happening under the hood. That said, I do believe it would be incredibly useful to be able to make changes to some part of a given AST, and push it down the pipeline to either validate a proposed change to the code, or to gather type information based on the change. And to do that modification is required. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14709#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14709: Extend the plugin mechanism to access program representation -------------------------------------+------------------------------------- Reporter: lazac | Owner: (none) Type: feature request | Status: new Priority: normal | Milestone: Component: GHC API | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | https://phabricator.haskell.org/D4342 https://ghc.haskell.org/trac/ghc/wiki/ExtendedPluginsProposal| -------------------------------------+------------------------------------- Comment (by bgamari): First: thanks, lazac, for the patch. GHC's plugin framework has long been waiting for someone to flesh it out and this patch looks like a great step in that direction. However, GHC's plugin interfaces are part of the compiler's public interface and as such will be supported for the foreseeable future. Consequently, we want to be careful that we don't end up introducing an interface that we will later come to regret. I discussed your patch with the Simons this morning and there was a general consensus that the proposal and the Haskell tooling community (some of whom have yet to comment here) would both benefit immensely from having the design in written form as a [[https://github.com/ghc-proposals /ghc-proposals|formal proposal]]. This would allow us to collect enough use-cases to better evaluate your proposed interfaces. I would be happy to lend a hand in developing, proof-reading, or otherwise facilitating this proposal if you like to help. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14709#comment:12 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14709: Extend the plugin mechanism to access program representation -------------------------------------+------------------------------------- Reporter: lazac | Owner: (none) Type: feature request | Status: new Priority: normal | Milestone: Component: GHC API | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | https://phabricator.haskell.org/D4342 https://ghc.haskell.org/trac/ghc/wiki/ExtendedPluginsProposal| -------------------------------------+------------------------------------- Comment (by simonpj): Yes, GHC's API and plugin interface exists to make it possible (and as easy as possible) to build new tools on top of GHC. It has evolved over time, but needs love, and thoughtful development. So thank you for offering some concrete suggestions. Clearly we want any new extensions to serve as wide a group of clients as possible. Sharing the proposals as a GHC proposal has proved to be a very effective way to improve ideas by discussing them with others. It would really help to have a couple of example use-cases to motivate each of the proposed plugin extensions. Thanks! -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14709#comment:13 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14709: Extend the plugin mechanism to access program representation -------------------------------------+------------------------------------- Reporter: lazac | Owner: (none) Type: feature request | Status: new Priority: normal | Milestone: Component: GHC API | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | https://phabricator.haskell.org/D4342 https://ghc.haskell.org/trac/ghc/wiki/ExtendedPluginsProposal| -------------------------------------+------------------------------------- Comment (by lazac): I created a formal propasal as it was requested. It is available in the pull request [https://github.com/ghc-proposals/ghc-proposals/pull/107]. Feel free to contribute to the proposal. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14709#comment:14 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14709: Extend the plugin mechanism to access program representation -------------------------------------+------------------------------------- Reporter: lazac | Owner: (none) Type: feature request | Status: new Priority: normal | Milestone: Component: GHC API | Version: 8.2.2 Resolution: | Keywords: GHCProposal Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | https://phabricator.haskell.org/D4342 https://ghc.haskell.org/trac/ghc/wiki/ExtendedPluginsProposal| -------------------------------------+------------------------------------- Changes (by RyanGlScott): * keywords: => GHCProposal -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14709#comment:15 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14709: Extend the plugin mechanism to access program representation
-------------------------------------+-------------------------------------
Reporter: lazac | Owner: (none)
Type: feature request | Status: new
Priority: normal | Milestone:
Component: GHC API | Version: 8.2.2
Resolution: | Keywords: GHCProposal
Operating System: Unknown/Multiple | Architecture:
| Unknown/Multiple
Type of failure: None/Unknown | Test Case:
Blocked By: | Blocking:
Related Tickets: | Differential Rev(s):
Wiki Page: | https://phabricator.haskell.org/D4342
https://ghc.haskell.org/trac/ghc/wiki/ExtendedPluginsProposal|
-------------------------------------+-------------------------------------
Comment (by Ben Gamari
#14709: Extend the plugin mechanism to access program representation -------------------------------------+------------------------------------- Reporter: lazac | Owner: (none) Type: feature request | Status: closed Priority: normal | Milestone: 8.6.1 Component: GHC API | Version: 8.2.2 Resolution: fixed | Keywords: GHCProposal Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | https://phabricator.haskell.org/D4342 https://ghc.haskell.org/trac/ghc/wiki/ExtendedPluginsProposal| -------------------------------------+------------------------------------- Changes (by bgamari): * status: new => closed * resolution: => fixed * milestone: => 8.6.1 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14709#comment:17 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
participants (1)
-
GHC