Re: Plugins: Accessing unexported bindings

Thanks for the explanation, Simon. I think Eric is spot on. Looking at some of the "ghc -v3" output, it appears GHC always runs the following passes before any plugins are run: *** Simplify: *** CorePrep: *** ByteCodeGen: *** Desugar: Just judging by the name, it could either be the first "Simplify" or the "Desugar" that gets rid of the dead-code at this point, I'm not sure which. It would help if this stage didn't remove dead-bindings. If doing so automatically is not the best option, requiring a user given pragma like 'KeepAlive' wouldn't be too onerous either. -Levent. On Mon, Dec 7, 2015 at 8:22 AM, Simon Peyton Jones wrote:

It would not be hard to stop the desugarer dropping dead bindings, if that was helpful.

S

| -----Original Message----- | From: Eric Seidel [mailto:eric@seidel.io] | Sent: 07 December 2015 15:44 | To: Simon Peyton Jones | Cc: Levent Erkok ; omeragacan@gmail.com; | ezyang@mit.edu; ghc-devs@haskell.org | Subject: Re: Plugins: Accessing unexported bindings | | The problem, as I recall, is that GHC does an initial bit of dead-code | elimination in the desugarer, before the plugins have a chance to run. | (I believe this is part of simpleOptPgm, but may be mistaken) | | I'm not sure why this is done in the desugarer, it seems to be out of | place there. | | On Mon, Dec 7, 2015, at 05:14, Simon Peyton Jones wrote: | > Plugins get to edit the entire core-to-core pipeline! There is no | magic. | > At least I don’t think so | > | > file:///Z:/tmp/users_guide/compiler-plugins.html | > | > S | > | > From: Levent Erkok [mailto:erkokl@gmail.com] | > Sent: 07 December 2015 13:11 | > To: Simon Peyton Jones | > Cc: Eric Seidel ; omeragacan@gmail.com; | > ezyang@mit.edu; ghc-devs@haskell.org | > Subject: Re: Plugins: Accessing unexported bindings | > | > That's a good point; keeping all annotated bindings alive seems to | be | > an overkill.. | > | > Regarding implementing "pass at the start." I'm not sure if plugin | > authors have any freedom as to decide when their plugin actually | runs. | > It seems GHC magically determines the order and runs them. Can you | > point me to some code/docs that tells me how to go "first" in that | > sense? (Or at least before the pass that drops dead code.) | > | > On Dec 7, 2015, at 4:45 AM, Simon Peyton Jones | > mailto:simonpj@microsoft.com> wrote: | > Indeed. How about this: if there's an ANN on a binder (any ANN), | then | > GHC should keep it alive. | > | > Really? It might be something like “don’t give warnings for this | > binding” or “don’t inline me” or something. To say *any* | annotation | > seems a bit brutal doesn’t it? Mind you I don’t have a better | idea. | > | > One thought: your plugin could add a pass right at the start, which | > marks everything you want as keep-alive. | > | > S | > | > From: Levent Erkok [mailto:erkokl@gmail.com] | > Sent: 07 December 2015 12:42 | > To: Simon Peyton Jones | > mailto:simonpj@microsoft.com> | > Cc: Eric Seidel mailto:eric@seidel.io>; | > omeragacan@gmail.commailto:omeragacan@gmail.com; | > ezyang@mit.edumailto:ezyang@mit.edu; | > ghc-devs@haskell.orgmailto:ghc-devs@haskell.org | > Subject: Re: Plugins: Accessing unexported bindings | > | > Indeed. How about this: if there's an ANN on a binder (any ANN), | then | > GHC should keep it alive. | > | > Is that something one of the core-developers can implement? Happy to | > open a ticket if that helps. | > | > On Dec 7, 2015, at 4:14 AM, Simon Peyton Jones | > mailto:simonpj@microsoft.com> wrote: | > If it's "dead" in this sense, it's already removed from ModGuts, no? | > | > Yes, if it’s dead it’s gone. That’s not too surprising, is it? | > | > So you need a way to keep it alive. Maybe we need a pragma for that. | Or | > how would you like to signal it in the source code? | > | > Simon | > | > From: Levent Erkok [mailto:erkokl@gmail.com] | > Sent: 07 December 2015 12:05 | > To: Simon Peyton Jones | > mailto:simonpj@microsoft.com> | > Cc: Eric Seidel mailto:eric@seidel.io>; | > omeragacan@gmail.commailto:omeragacan@gmail.com; | > ezyang@mit.edumailto:ezyang@mit.edu; | > ghc-devs@haskell.orgmailto:ghc-devs@haskell.org | > Subject: Re: Plugins: Accessing unexported bindings | > | > Thanks Simon.. But I remain utterly confused. As a "plugin" author, | > how do I get my hands on the Id associated with a top-level binder? | If | > it's "dead" in this sense, it's already removed from ModGuts, no? | > | > That is, by the time GHC runs my plugin, the Id has already | > disappeared for me to mark it "Local Exported." Is that not correct? | > | > On Dec 7, 2015, at 2:28 AM, Simon Peyton Jones | > mailto:simonpj@microsoft.com> wrote: | > In the mean time, I'm still looking for a solution that doesn't | > involve exporting such identifiers from modules. As Eric pointed | out, | > that seems to be the only current work-around for the time being. | > | > “Exported” in this context only means “keep alive”. It does not mean | > exported in the Haskell source code sense. I’ve just added this | > comment to Var.hs. | > | > So I think it does just what you want. | > | > Simon | > | > data ExportFlag -- See Note [ExportFlag on binders] | > = NotExported -- ^ Not exported: may be discarded as dead code. | > | Exported -- ^ Exported: kept alive | > | > {- Note [ExportFlag on binders] | > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | > An ExportFlag of "Exported" on a top-level binder says "keep this | > binding alive; do not drop it as dead code". This transititively | > keeps alive all the other top-level bindings that this binding | refers | > to. This property is persisted all the way down the pipeline, so | that | > the binding will be compiled all the way to object code, and its | > symbols will appear in the linker symbol table. | > | > However, note that this use of "exported" is quite different to the | > export list on a Haskell module. Setting the ExportFlag on an Id | does | > /not/ mean that if you import the module (in Haskell source code you | > will see this Id. Of course, things that appear in the export list | of | > the source Haskell module do indeed have their ExportFlag set. | > But many other things, such as dictionary functions, are kept alive | by | > having their ExportFlag set, even though they are not exported in | the | > source-code sense. | > | > We should probably use a different term for ExportFlag, like | > KeepAlive. | > | > From: ghc-devs [mailto:ghc-devs-bounces@haskell.org] On Behalf Of | > Levent Erkok | > Sent: 06 December 2015 20:32 | > To: Eric Seidel mailto:eric@seidel.io>; | > omeragacan@gmail.commailto:omeragacan@gmail.com; | > ezyang@mit.edumailto:ezyang@mit.edu | > Cc: ghc-devs@haskell.orgmailto:ghc-devs@haskell.org | > Subject: Re: Plugins: Accessing unexported bindings | > | > Omer, Eric, Ed: Thanks for the comments. | > | > Omer: I think Eric's observation is at play here. We're talking | about | > "dead-code," i.e., a binding that is neither exported, nor used by | any | > binding inside the module. Those seem to be getting dropped by the | > time user-plugins are run. Unfortunately, this is precisely what one | > would do with "properties" embedded in code. They serve as | > documentation perhaps, but are otherwise not needed by any other | > binding nor it makes sense to export them. | > | > Edward: Can you provide some more info into your solution? Sounds | like | > a chicken-egg issue to me: As a plugin author, I need the bindings | to | > access the Ids, and looks like I need the Ids to access the binders? | > | > A simple solution would be to simply keep all top-level bindings | > around while the plugin are running, but that obviously can lead to | > unnecessary work if the code is truly dead. A compromise could be | that | > the annotations can serve as entry points as well: I.e., if there's | an | > annotation on a top-level binder, then it should *not* be considered | > dead-code at least until after all the plugins are run. That would | > definitely simplify life. Would that be an acceptable alternative? | > | > In the mean time, I'm still looking for a solution that doesn't | > involve exporting such identifiers from modules. As Eric pointed | out, | > that seems to be the only current work-around for the time being. | > | > Thanks, | > | > -Levent. | > | > On Sun, Dec 6, 2015 at 11:08 AM, Eric Seidel | > mailto:eric@seidel.io> wrote: | > GHC should only drop un-exported bindings from the ModGuts if | they're | > also unused, ie *dead code*. | > | > The only way I know to get around this is to use the bindings | > somewhere, or just export them. | > | > On Sat, Dec 5, 2015, at 23:01, Levent Erkok wrote: | > > Hello, | > > | > > The mg_binds field of the ModGuts seem to only contain the | bindings | > > that are exported from the module being compiled. | > > | > > I guess GHC must be running user-plugins after it drops the | bindings | > > that are not exported, which makes perfect sense for most use | cases. | > > However, I'm working on a plugin where the end-programmer embeds | > > "properties" in the form of functions inside his/her code, which | are | > > not necessarily exported from the module under consideration. | > > | > > Is there a way to access all top-level bindings in a module from a | > > plugin, even if those bindings are not exported? | > > | > > Thanks, | > > | > > -Levent. | > > _______________________________________________ | > > ghc-devs mailing list | > > ghc-devs@haskell.orgmailto:ghc-devs@haskell.org | > > | https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fmail | > > .haskell.org%2fcgi-bin%2fmailman%2flistinfo%2fghc- | devs&data=01%7c01% | > > | 7csimonpj%40064d.mgd.microsoft.com%7cf6e3a9d4ad9f4e53a3ab08d2ff1d493 | > > | 4%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=Bv7lpsB%2fD88nSMuB7NY | > > | fBqR90%2bBq%2fwpJJ0JU9%2b6E4RI%3d<https://na01.safelinks.protection. | > > outlook.com/?url=http%3a%2f%2fmail.haskell.org%2fcgi- | bin%2fmailman%2 | > > flistinfo%2fghc- | devs&data=01%7c01%7csimonpj%40064d.mgd.microsoft.com | > > | %7cac4cbfe22e314080909908d2fe7c4ed8%7c72f988bf86f141af91ab2d7cd011db | > > 47%7c1&sdata=1z6DcZxjIAKj0PcsLeALphRLWJ3i%2fxvyaPtq0qo6elY%3d> | > _______________________________________________ | > ghc-devs mailing list | > ghc-devs@haskell.orgmailto:ghc-devs@haskell.org | > | https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fmail.h | > askell.org%2fcgi-bin%2fmailman%2flistinfo%2fghc- | devs&data=01%7c01%7csi | > | monpj%40064d.mgd.microsoft.com%7cf6e3a9d4ad9f4e53a3ab08d2ff1d4934%7c72 | > | f988bf86f141af91ab2d7cd011db47%7c1&sdata=Bv7lpsB%2fD88nSMuB7NYfBqR90%2 | > | bBq%2fwpJJ0JU9%2b6E4RI%3d<https://na01.safelinks.protection.outlook.co | > m/?url=http%3a%2f%2fmail.haskell.org%2fcgi- | bin%2fmailman%2flistinfo%2f | > ghc- | devs&data=01%7c01%7csimonpj%40064d.mgd.microsoft.com%7cac4cbfe22e3 | > | 14080909908d2fe7c4ed8%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=1z6 | > DcZxjIAKj0PcsLeALphRLWJ3i%2fxvyaPtq0qo6elY%3d> | >