Hello On Thu, Aug 22, 2013 at 8:13 PM, Simon Peyton-Jones wrote:

Luite, Edsko, Thomas, Nicolas****

** **

You have all variously proposed improvements to the GHC API and/or the plug-in mechanism. I have been so swamped in the last few months that I have not had a chance to look carefully at your proposals, nor how they relate to each other.****

** **

We are now only three weeks away from wanting to do a feature freeze on GHC 7.8, and there are a lot of other things that we want to complete****

http://ghc.haskell.org/trac/ghc/wiki/Status/GHC-7.8****

(Mostly they have gestating for some time.)****

** **

So I’m hoping you’ll be ok with not putting these plugin-related changes into 7.8. I have the sense that they’d benefit from more discussion among the folk interested in plugins. Perhaps some of the ideas could be combined nicely; I don’t know. And the people who are going to write plugins are also probably up for building HEAD anyhow.****

(Exception: Luite, I think you have some fairly narrow, specific changes that would help GHCJS, and I’m probably fine with those if you care to send patches.)****

**

Does this mean that GHCJS patches will be accepted, but it has to use some other mechanism (not Hooks)? It would be really nice to see ghcjs support in the upcoming GHC.

**

Please say if you think there’s a really strong reason for putting stuff in the 7.8.****

** **

Thanks****

** **

Simon****

** **

*From:* ghc-devs [mailto:ghc-devs-bounces@haskell.org] *On Behalf Of *Luite Stegeman *Sent:* 21 August 2013 03:51 *To:* ghc-devs *Subject:* extending GHC plugins with Hooks****

** **

hi all,****

** **

Sorry for taking so long to get back with this. I'm proposing a somewhat general way for adding 'hooks' to the GHC API, where users can override parts of the default compiling pipeline.****

** **

Hooks are simply functions or actions that replace existing compiler functionality. This means that usually only one application can use a specific hook at a time. ****

** **

The obvious data structure to store the hooks is DynFlags. Unfortunately defining hooks in DynFlags directly would give birth to the mother of all import cycles, and it would also break the split-dll scheme on Windows. So here's the idea:****

** **

- Define each hook in the module where it's exported****

- For each hook make a 'phantom' DataType and an instance for the Hook type familiy****

- Add a TypeRep based map in DynFlags [0]****

- For each hooked function, check for existence of a hook in DynFlags, otherwise run the default. Example: https://github.com/ghcjs/ghcjs-build/blob/master/refs/patches/ghc-ghcjs.patc... ****

** **

Now this approach does have some disadvantages:****

- No clear integration with existing plugins (I've tried adding an onLoadPlugin field to Plugin, where the Plugin could update DynFlags when it's loaded, but it was a bit messy, and plugins would not be loaded in time for some hooks, particularly what Edsko needs)****

- More of GHC depends on type families****

- Decentralized hooks definitions feel a bit messy****

** **

So I'm open to suggestions for improvements (or replacements) of this scheme. I have some time the coming weeks to clean up or change the patch. ****

** **

We've been testing some hooks with GHCJS for a while, and so far they seem to provide what we need (but I'm going to doublecheck the coming weeks that we don't have missing functionality):****

** **

- Customizations for linking JavaScript code with our own library locations [1]****

- Hooking into the DriverPipeline so we can use the compilation manager [2] ****

- Desugaring customizations to remove some C-isms from the FFI code [3]*** *

- Typechecking foreign import javascript imports [4]****

- Override the built-in GHC.Prim so we can customize primop types [5]****

** **

I think it's easy to add those for Edsko and Thomas as well.****

** **

luite****

** **

[0] https://github.com/ghcjs/ghcjs-build/blob/master/refs/patches/ghc-ghcjs.patc... ****

[1] https://github.com/ghcjs/ghcjs/blob/master/src/Compiler/GhcjsHooks.hs#L44* ***

[2] https://github.com/ghcjs/ghcjs/blob/master/src/Compiler/GhcjsHooks.hs#L192 ****

https://github.com/ghcjs/ghcjs-build/blob/master/refs/patches/ghc-ghcjs.patc... ****

[3] https://github.com/ghcjs/ghcjs/blob/master/src/Gen2/Foreign.hs#L67****

https://github.com/ghcjs/ghcjs-build/blob/master/refs/patches/ghc-ghcjs.patc... ****

[4] https://github.com/ghcjs/ghcjs/blob/master/src/Gen2/Foreign.hs#L68****

https://github.com/ghcjs/ghcjs-build/blob/master/refs/patches/ghc-ghcjs.patc... ****

[5] https://github.com/ghcjs/ghcjs/blob/master/src/Compiler/GhcjsHooks.hs#L191 ****

https://github.com/ghcjs/ghcjs-build/blob/master/refs/patches/ghc-ghcjs.patc... ****

** **

(sorry the linked patch also contains other modifications required for GHCJS, that i'm going to submit separately)****

** **

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

-- Sincerely yours, -- Daniil Frumin

I’m sympathetic to supporting GHCJS. But, as described in my email, there seems to be several different proposals for extending plugins swirling around, and I’m not sure how they relate to each other. My suggestion (for you guys to lead ☺): · Identify the minimum necessary for GHCJS in 7.8 · Debate the remaining proposals under less time pressure Simon From: Carter Schonwald [mailto:carter.schonwald@gmail.com] Sent: 23 August 2013 19:36 To: Daniel F Cc: Simon Peyton-Jones; ghc-devs Subject: Re: extending GHC plugins with Hooks indeed, ghcjs is one of the things i'm most looking forward to! That the hooks machinery needed to nicely support GHCJS will enable many other use cases is also something i'm excited about too... would be a shame for neither to work with 7.8 .... -Carter On Fri, Aug 23, 2013 at 2:09 PM, Daniel F mailto:difrumin@gmail.com> wrote: Hello On Thu, Aug 22, 2013 at 8:13 PM, Simon Peyton-Jones mailto:simonpj@microsoft.com> wrote: Luite, Edsko, Thomas, Nicolas You have all variously proposed improvements to the GHC API and/or the plug-in mechanism. I have been so swamped in the last few months that I have not had a chance to look carefully at your proposals, nor how they relate to each other. We are now only three weeks away from wanting to do a feature freeze on GHC 7.8, and there are a lot of other things that we want to complete http://ghc.haskell.org/trac/ghc/wiki/Status/GHC-7.8 (Mostly they have gestating for some time.) So I’m hoping you’ll be ok with not putting these plugin-related changes into 7.8. I have the sense that they’d benefit from more discussion among the folk interested in plugins. Perhaps some of the ideas could be combined nicely; I don’t know. And the people who are going to write plugins are also probably up for building HEAD anyhow. (Exception: Luite, I think you have some fairly narrow, specific changes that would help GHCJS, and I’m probably fine with those if you care to send patches.) Does this mean that GHCJS patches will be accepted, but it has to use some other mechanism (not Hooks)? It would be really nice to see ghcjs support in the upcoming GHC. Please say if you think there’s a really strong reason for putting stuff in the 7.8. Thanks Simon From: ghc-devs [mailto:ghc-devs-bounces@haskell.orgmailto:ghc-devs-bounces@haskell.org] On Behalf Of Luite Stegeman Sent: 21 August 2013 03:51 To: ghc-devs Subject: extending GHC plugins with Hooks hi all, Sorry for taking so long to get back with this. I'm proposing a somewhat general way for adding 'hooks' to the GHC API, where users can override parts of the default compiling pipeline. Hooks are simply functions or actions that replace existing compiler functionality. This means that usually only one application can use a specific hook at a time. The obvious data structure to store the hooks is DynFlags. Unfortunately defining hooks in DynFlags directly would give birth to the mother of all import cycles, and it would also break the split-dll scheme on Windows. So here's the idea: - Define each hook in the module where it's exported - For each hook make a 'phantom' DataType and an instance for the Hook type familiy - Add a TypeRep based map in DynFlags [0] - For each hooked function, check for existence of a hook in DynFlags, otherwise run the default. Example: https://github.com/ghcjs/ghcjs-build/blob/master/refs/patches/ghc-ghcjs.patc... Now this approach does have some disadvantages: - No clear integration with existing plugins (I've tried adding an onLoadPlugin field to Plugin, where the Plugin could update DynFlags when it's loaded, but it was a bit messy, and plugins would not be loaded in time for some hooks, particularly what Edsko needs) - More of GHC depends on type families - Decentralized hooks definitions feel a bit messy So I'm open to suggestions for improvements (or replacements) of this scheme. I have some time the coming weeks to clean up or change the patch. We've been testing some hooks with GHCJS for a while, and so far they seem to provide what we need (but I'm going to doublecheck the coming weeks that we don't have missing functionality): - Customizations for linking JavaScript code with our own library locations [1] - Hooking into the DriverPipeline so we can use the compilation manager [2] - Desugaring customizations to remove some C-isms from the FFI code [3] - Typechecking foreign import javascript imports [4] - Override the built-in GHC.Prim so we can customize primop types [5] I think it's easy to add those for Edsko and Thomas as well. luite [0] https://github.com/ghcjs/ghcjs-build/blob/master/refs/patches/ghc-ghcjs.patc... [1] https://github.com/ghcjs/ghcjs/blob/master/src/Compiler/GhcjsHooks.hs#L44 [2] https://github.com/ghcjs/ghcjs/blob/master/src/Compiler/GhcjsHooks.hs#L192 https://github.com/ghcjs/ghcjs-build/blob/master/refs/patches/ghc-ghcjs.patc... [3] https://github.com/ghcjs/ghcjs/blob/master/src/Gen2/Foreign.hs#L67 https://github.com/ghcjs/ghcjs-build/blob/master/refs/patches/ghc-ghcjs.patc... [4] https://github.com/ghcjs/ghcjs/blob/master/src/Gen2/Foreign.hs#L68 https://github.com/ghcjs/ghcjs-build/blob/master/refs/patches/ghc-ghcjs.patc... [5] https://github.com/ghcjs/ghcjs/blob/master/src/Compiler/GhcjsHooks.hs#L191 https://github.com/ghcjs/ghcjs-build/blob/master/refs/patches/ghc-ghcjs.patc... (sorry the linked patch also contains other modifications required for GHCJS, that i'm going to submit separately) _______________________________________________ ghc-devs mailing list ghc-devs@haskell.orgmailto:ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs -- Sincerely yours, -- Daniil Frumin _______________________________________________ ghc-devs mailing list ghc-devs@haskell.orgmailto:ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

just a quick heads-up: After discussing with Thomas at ZuriHac I now have a review-ready patch (and a demo program that uses all the hooks) that should do everything Thomas, Edsko and I need, and is extensible if new things come up. I'm still in Switzerland though, and my internet connection is spotty, I'll try to submit it tonight or tomorrow morning luite On Tue, Aug 27, 2013 at 6:18 PM, Thomas Schilling wrote:

I think there are a couple of things that are worth exposing. Looking at Luite's patch, for example, there are a few that a basic high-level platform wouldn't solve. A few things an API might reasonably want to customise:

- Parse custom flags (when wrapping GHC)

- Hook into GHC's compilation manager or use one-shot mode and use Shake as a more flexible compilation manager

- Foreign function calling conventions (e.g., for GHCJS)

- Compilation ways (can't think of a good example here)

- The location of built-in / hardcoded identifiers (i.e., the PrelNames stuff)

- How modules are found (Finder) and where they are stored (e.g., network-backed store, in-memory only)

- Wrap/Adapt various stages of the renamer/typechecker/optimiser/codegen. Mostly for IDE tools (including tools to examine compiler transformations), but custom backends also need to replace the codegen bits

- Run custom stages in DriverPipeline or adapt existing ones (including not running them)

- Both Lambdachine and GHCJS implement a custom code generator. I think GHCJS is translating from STG. Lambdachine used to start from CorePrep, but due to the weird "unarise" stuff I had to switch to using STG which isn't too bad as it also seems more stable. A custom code generator requires disabling the "Asm" pass and all subsequent passes from DriverPipeline

- I'm not so sure about syntactic extensions. They might be better implemented using haskell-src-exts and then translated into plain Haskell

If we can get these things integrated nicely then it would be worthwhile to wait another year.

...

I’m sympathetic to supporting GHCJS. But, as described in my email,

On 27 August 2013 09:17, Simon Peyton-Jones wrote: there

...

seems to be several different proposals for extending plugins swirling around, and I’m not sure how they relate to each other.

My suggestion (for you guys to lead J):

· Identify the minimum necessary for GHCJS in 7.8

· Debate the remaining proposals under less time pressure

Simon

From: Carter Schonwald [mailto:carter.schonwald@gmail.com] Sent: 23 August 2013 19:36 To: Daniel F Cc: Simon Peyton-Jones; ghc-devs Subject: Re: extending GHC plugins with Hooks

indeed, ghcjs is one of the things i'm most looking forward to!

That the hooks machinery needed to nicely support GHCJS will enable many other use cases is also something i'm excited about too... would be a shame for neither to work with 7.8 ....

-Carter

On Fri, Aug 23, 2013 at 2:09 PM, Daniel F wrote:

Hello

On Thu, Aug 22, 2013 at 8:13 PM, Simon Peyton-Jones < simonpj@microsoft.com> wrote:

Luite, Edsko, Thomas, Nicolas

You have all variously proposed improvements to the GHC API and/or the plug-in mechanism. I have been so swamped in the last few months that I have not had a chance to look carefully at your proposals, nor how they relate to each other.

We are now only three weeks away from wanting to do a feature freeze on GHC 7.8, and there are a lot of other things that we want to complete

http://ghc.haskell.org/trac/ghc/wiki/Status/GHC-7.8

(Mostly they have gestating for some time.)

So I’m hoping you’ll be ok with not putting these plugin-related changes into 7.8. I have the sense that they’d benefit from more discussion among the folk interested in plugins. Perhaps some of the ideas could be combined nicely; I don’t know. And the people who are going to write plugins are also probably up for building HEAD anyhow.

(Exception: Luite, I think you have some fairly narrow, specific changes that would help GHCJS, and I’m probably fine with those if you care to send patches.)

Does this mean that GHCJS patches will be accepted, but it has to use some other mechanism (not Hooks)?

It would be really nice to see ghcjs support in the upcoming GHC.

Please say if you think there’s a really strong reason for putting stuff in the 7.8.

Thanks

Simon

From: ghc-devs [mailto:ghc-devs-bounces@haskell.org] On Behalf Of Luite Stegeman Sent: 21 August 2013 03:51 To: ghc-devs Subject: extending GHC plugins with Hooks

hi all,

Sorry for taking so long to get back with this. I'm proposing a somewhat general way for adding 'hooks' to the GHC API, where users can override parts of the default compiling pipeline.

Hooks are simply functions or actions that replace existing compiler functionality. This means that usually only one application can use a specific hook at a time.

The obvious data structure to store the hooks is DynFlags. Unfortunately defining hooks in DynFlags directly would give birth to the mother of all import cycles, and it would also break the split-dll scheme on Windows. So here's the idea:

- Define each hook in the module where it's exported

- For each hook make a 'phantom' DataType and an instance for the Hook type familiy

- Add a TypeRep based map in DynFlags [0]

- For each hooked function, check for existence of a hook in DynFlags, otherwise run the default. Example:

https://github.com/ghcjs/ghcjs-build/blob/master/refs/patches/ghc-ghcjs.patc...

...

Now this approach does have some disadvantages:

- No clear integration with existing plugins (I've tried adding an onLoadPlugin field to Plugin, where the Plugin could update DynFlags when it's loaded, but it was a bit messy, and plugins would not be loaded in

...

for some hooks, particularly what Edsko needs)

- More of GHC depends on type families

- Decentralized hooks definitions feel a bit messy

So I'm open to suggestions for improvements (or replacements) of this scheme. I have some time the coming weeks to clean up or change the

time patch.

...

We've been testing some hooks with GHCJS for a while, and so far they

...

to provide what we need (but I'm going to doublecheck the coming weeks

seem that

...

we don't have missing functionality):

- Customizations for linking JavaScript code with our own library locations [1]

- Hooking into the DriverPipeline so we can use the compilation manager [2]

- Desugaring customizations to remove some C-isms from the FFI code [3]

- Typechecking foreign import javascript imports [4]

- Override the built-in GHC.Prim so we can customize primop types [5]

I think it's easy to add those for Edsko and Thomas as well.

luite

[0]

https://github.com/ghcjs/ghcjs-build/blob/master/refs/patches/ghc-ghcjs.patc...

...

[1]

https://github.com/ghcjs/ghcjs/blob/master/src/Compiler/GhcjsHooks.hs#L44

...

[2]

https://github.com/ghcjs/ghcjs/blob/master/src/Compiler/GhcjsHooks.hs#L192

...

https://github.com/ghcjs/ghcjs-build/blob/master/refs/patches/ghc-ghcjs.patc...

...

[3] https://github.com/ghcjs/ghcjs/blob/master/src/Gen2/Foreign.hs#L67

https://github.com/ghcjs/ghcjs-build/blob/master/refs/patches/ghc-ghcjs.patc...

...

[4] https://github.com/ghcjs/ghcjs/blob/master/src/Gen2/Foreign.hs#L68

https://github.com/ghcjs/ghcjs-build/blob/master/refs/patches/ghc-ghcjs.patc...

...

[5]

https://github.com/ghcjs/ghcjs/blob/master/src/Compiler/GhcjsHooks.hs#L191

...

https://github.com/ghcjs/ghcjs-build/blob/master/refs/patches/ghc-ghcjs.patc...

...

(sorry the linked patch also contains other modifications required for GHCJS, that i'm going to submit separately)

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

-- Sincerely yours, -- Daniil Frumin

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

OK, that's fine. Thanks! Simon From: Nicolas Frisby [mailto:nicolas.frisby@gmail.com] Sent: 10 September 2013 19:18 To: Simon Peyton-Jones Cc: Luite Stegeman; Edsko de Vries; Thomas Schilling; ghc-devs Subject: Re: extending GHC plugins with Hooks My patch was extremely simple, so I'm asking for forgiveness instead of permission! https://github.com/ghc/ghc/commit/850490af1df426b306d898381a358a35425d16c7 The commit note includes a brief explanation of the benefits. The motivation originates with the HERMIT project at Univ. of Kansas: we'd like to help the user generate new top-level declarations in a module (eg a new datatype). Re-using the type-checker seems the simplest path towards robustness and feature completeness and this patch removes a simple but onerous obstacle. Is this OK? On Thu, Aug 22, 2013 at 11:13 AM, Simon Peyton-Jones mailto:simonpj@microsoft.com> wrote: Luite, Edsko, Thomas, Nicolas You have all variously proposed improvements to the GHC API and/or the plug-in mechanism. I have been so swamped in the last few months that I have not had a chance to look carefully at your proposals, nor how they relate to each other. We are now only three weeks away from wanting to do a feature freeze on GHC 7.8, and there are a lot of other things that we want to complete http://ghc.haskell.org/trac/ghc/wiki/Status/GHC-7.8 (Mostly they have gestating for some time.) So I'm hoping you'll be ok with not putting these plugin-related changes into 7.8. I have the sense that they'd benefit from more discussion among the folk interested in plugins. Perhaps some of the ideas could be combined nicely; I don't know. And the people who are going to write plugins are also probably up for building HEAD anyhow. (Exception: Luite, I think you have some fairly narrow, specific changes that would help GHCJS, and I'm probably fine with those if you care to send patches.) Please say if you think there's a really strong reason for putting stuff in the 7.8. Thanks Simon From: ghc-devs [mailto:ghc-devs-bounces@haskell.orgmailto:ghc-devs-bounces@haskell.org] On Behalf Of Luite Stegeman Sent: 21 August 2013 03:51 To: ghc-devs Subject: extending GHC plugins with Hooks hi all, Sorry for taking so long to get back with this. I'm proposing a somewhat general way for adding 'hooks' to the GHC API, where users can override parts of the default compiling pipeline. Hooks are simply functions or actions that replace existing compiler functionality. This means that usually only one application can use a specific hook at a time. The obvious data structure to store the hooks is DynFlags. Unfortunately defining hooks in DynFlags directly would give birth to the mother of all import cycles, and it would also break the split-dll scheme on Windows. So here's the idea: - Define each hook in the module where it's exported - For each hook make a 'phantom' DataType and an instance for the Hook type familiy - Add a TypeRep based map in DynFlags [0] - For each hooked function, check for existence of a hook in DynFlags, otherwise run the default. Example: https://github.com/ghcjs/ghcjs-build/blob/master/refs/patches/ghc-ghcjs.patc... Now this approach does have some disadvantages: - No clear integration with existing plugins (I've tried adding an onLoadPlugin field to Plugin, where the Plugin could update DynFlags when it's loaded, but it was a bit messy, and plugins would not be loaded in time for some hooks, particularly what Edsko needs) - More of GHC depends on type families - Decentralized hooks definitions feel a bit messy So I'm open to suggestions for improvements (or replacements) of this scheme. I have some time the coming weeks to clean up or change the patch. We've been testing some hooks with GHCJS for a while, and so far they seem to provide what we need (but I'm going to doublecheck the coming weeks that we don't have missing functionality): - Customizations for linking JavaScript code with our own library locations [1] - Hooking into the DriverPipeline so we can use the compilation manager [2] - Desugaring customizations to remove some C-isms from the FFI code [3] - Typechecking foreign import javascript imports [4] - Override the built-in GHC.Prim so we can customize primop types [5] I think it's easy to add those for Edsko and Thomas as well. luite [0] https://github.com/ghcjs/ghcjs-build/blob/master/refs/patches/ghc-ghcjs.patc... [1] https://github.com/ghcjs/ghcjs/blob/master/src/Compiler/GhcjsHooks.hs#L44 [2] https://github.com/ghcjs/ghcjs/blob/master/src/Compiler/GhcjsHooks.hs#L192 https://github.com/ghcjs/ghcjs-build/blob/master/refs/patches/ghc-ghcjs.patc... [3] https://github.com/ghcjs/ghcjs/blob/master/src/Gen2/Foreign.hs#L67 https://github.com/ghcjs/ghcjs-build/blob/master/refs/patches/ghc-ghcjs.patc... [4] https://github.com/ghcjs/ghcjs/blob/master/src/Gen2/Foreign.hs#L68 https://github.com/ghcjs/ghcjs-build/blob/master/refs/patches/ghc-ghcjs.patc... [5] https://github.com/ghcjs/ghcjs/blob/master/src/Compiler/GhcjsHooks.hs#L191 https://github.com/ghcjs/ghcjs-build/blob/master/refs/patches/ghc-ghcjs.patc... (sorry the linked patch also contains other modifications required for GHCJS, that i'm going to submit separately)