Changes in GHC API modularity with the "Encode shape information in PMOracle" MR
Some time back, the `ghc-lib` project split into two targets : `ghc-lib-parser` for those projects that just need to produce syntax trees and `ghc-lib` (re-exporting `ghc-lib-parser` modules) having the remaining modules for projects that need to go on and distill parse trees to Core. The idea of course was to reduce build times for tools like hlint that only need parse trees. Roughly, `ghc-lib-parser` got about 200 files and `ghc-lib` 300. Today after landing `7915afc6bb9539a4534db99aeb6616a6d145918a`, "Encode shape information in `PmOracle`", `ghc-lib-parser` now needs 543 files and `ghc-lib` gets just 25. That may be just bad luck for `ghc-lib-parser` and the way it has to be but I thought I should at least mention the knock-on effect of this change on the modularity of the GHC API in case this consequence hasn't been considered? -- *Shayne Fletcher* Language Engineer */* +1 917 699 7663 *Digital Asset* <https://digitalasset.com/>, creators of *DAML <https://daml.com/>* -- This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html <http://www.digitalasset.com/emaildisclaimer.html>. If you are not the intended recipient, please delete this message.
Hi Shayne, Sorry to hear that! We didn't consider modularity at all and I would be happy to try to refactor in a way that would allow `ghc-lib-parser` to be properly separated again. I'm fairly certain that I didn't directly touch anything parser related, but apparently the new cyclic import of PmOracle within TcRnTypes (which is also exposed from `ghc-lib-parser`) pulled in the other half of GHC. I'll see if I can fix that tomorrow, if only by extracting a separate `Types`-style module. Cheers, Sebastian Am Mo., 16. Sept. 2019 um 22:04 Uhr schrieb Shayne Fletcher via ghc-devs < ghc-devs@haskell.org>:
Some time back, the `ghc-lib` project split into two targets : `ghc-lib-parser` for those projects that just need to produce syntax trees and `ghc-lib` (re-exporting `ghc-lib-parser` modules) having the remaining modules for projects that need to go on and distill parse trees to Core. The idea of course was to reduce build times for tools like hlint that only need parse trees.
Roughly, `ghc-lib-parser` got about 200 files and `ghc-lib` 300. Today after landing `7915afc6bb9539a4534db99aeb6616a6d145918a`, "Encode shape information in `PmOracle`", `ghc-lib-parser` now needs 543 files and `ghc-lib` gets just 25.
That may be just bad luck for `ghc-lib-parser` and the way it has to be but I thought I should at least mention the knock-on effect of this change on the modularity of the GHC API in case this consequence hasn't been considered?
-- *Shayne Fletcher* Language Engineer */* +1 917 699 7663 *Digital Asset* <https://digitalasset.com/>, creators of *DAML <https://daml.com/>*
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html. If you are not the intended recipient, please delete this message. _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Hi Sebastian, On Mon, Sep 16, 2019 at 5:23 PM Sebastian Graf <sgraf1337@gmail.com> wrote:
Hi Shayne,
Sorry to hear that! We didn't consider modularity at all and I would be happy to try to refactor in a way that would allow `ghc-lib-parser` to be properly separated again. I'm fairly certain that I didn't directly touch anything parser related, but apparently the new cyclic import of PmOracle within TcRnTypes (which is also exposed from `ghc-lib-parser`) pulled in the other half of GHC. I'll see if I can fix that tomorrow, if only by extracting a separate `Types`-style module.
That sounds awesome. Tremendous. Thank-you! Please feel free to reach out to me if there's anything I can do to help your analysis[*]! [*] For the record, the procedure for calculating the `ghc-lib-parser` modules is a little complicated by there needing to be some generated equivalents of `.hsc` files present for this to work but the procedure is at the end of the day just `ghc -M` invoked over `Parser.hs`.
Cheers, Sebastian
Fingers crossed and all the best! -- *Shayne Fletcher* Language Engineer */* +1 917 699 7663 *Digital Asset* <https://digitalasset.com/>, creators of *DAML <https://daml.com/>* -- This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html <http://www.digitalasset.com/emaildisclaimer.html>. If you are not the intended recipient, please delete this message.
This is precisely the problem I was worried about in April. https://mail.haskell.org/pipermail/ghc-devs/2019-April/017493.html Any fix should ensure adding a test to make sure it doesn't happen again. Cheers, Matt On Mon, Sep 16, 2019 at 10:38 PM Shayne Fletcher via ghc-devs <ghc-devs@haskell.org> wrote:
Hi Sebastian,
On Mon, Sep 16, 2019 at 5:23 PM Sebastian Graf <sgraf1337@gmail.com> wrote:
Hi Shayne,
Sorry to hear that! We didn't consider modularity at all and I would be happy to try to refactor in a way that would allow `ghc-lib-parser` to be properly separated again. I'm fairly certain that I didn't directly touch anything parser related, but apparently the new cyclic import of PmOracle within TcRnTypes (which is also exposed from `ghc-lib-parser`) pulled in the other half of GHC. I'll see if I can fix that tomorrow, if only by extracting a separate `Types`-style module.
That sounds awesome. Tremendous. Thank-you! Please feel free to reach out to me if there's anything I can do to help your analysis[*]!
[*] For the record, the procedure for calculating the `ghc-lib-parser` modules is a little complicated by there needing to be some generated equivalents of `.hsc` files present for this to work but the procedure is at the end of the day just `ghc -M` invoked over `Parser.hs`.
Cheers, Sebastian
Fingers crossed and all the best!
-- Shayne Fletcher Language Engineer / +1 917 699 7663 Digital Asset, creators of DAML
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html. If you are not the intended recipient, please delete this message._______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Hmm. The issue of build parallelism is kind-of orthogonal to .hs-boot files/circular imports. The impact of .hs-boot files on build parallelism depends on their announced imports; at least in my semantic model of them (which might be incorrect) they just break cycles. The degree of parallelism is visible as a property of this DAG where cycles are broken through insertion of .hs-boot nodes. The issue here is one of transitive dependencies, which is a property on the original cyclic dependency graph. So while we have the dependency graph HsExpr <--> TcRnTypes <--> PmOracle This is of no issue for build parallelism, because the .hs-boot files remove "feedback vertices" from the graph, rendering it acyclic: TcRnTypes.hs-boot <- HsExpr <- TcRnTypes <- PmOracle PmOracle.hs-boot <- TcRnTypes Yet a package including HsExpr must also include PmOracle to compile. The parallelism issue is best tackled by hadrian based profiles, on which we could compute try to come up with the optimal schedule and compare it across multiple CI runs. But the dependency issue here is rather specific to the set of modules that should be included in ghc-lib-parser, I'm afraid, and I don't see how to solve it in a general way other than pre-computing the set of transitive dependencies for specific blessed modules. Which might be a thing we want... One set for each major entry point to the compiler pipeline, for example. Am Di., 17. Sept. 2019 um 10:19 Uhr schrieb Matthew Pickering < matthewtpickering@gmail.com>:
This is precisely the problem I was worried about in April.
https://mail.haskell.org/pipermail/ghc-devs/2019-April/017493.html
Any fix should ensure adding a test to make sure it doesn't happen again.
Cheers,
Matt
On Mon, Sep 16, 2019 at 10:38 PM Shayne Fletcher via ghc-devs <ghc-devs@haskell.org> wrote:
Hi Sebastian,
On Mon, Sep 16, 2019 at 5:23 PM Sebastian Graf <sgraf1337@gmail.com>
wrote:
Hi Shayne,
Sorry to hear that! We didn't consider modularity at all and I would be
happy to try to refactor in a way that would allow `ghc-lib-parser` to be properly separated again.
I'm fairly certain that I didn't directly touch anything parser related, but apparently the new cyclic import of PmOracle within TcRnTypes (which is also exposed from `ghc-lib-parser`) pulled in the other half of GHC. I'll see if I can fix that tomorrow, if only by extracting a separate `Types`-style module.
That sounds awesome. Tremendous. Thank-you! Please feel free to reach out to me if there's anything I can do to help your analysis[*]!
[*] For the record, the procedure for calculating the `ghc-lib-parser` modules is a little complicated by there needing to be some generated equivalents of `.hsc` files present for this to work but the procedure is at the end of the day just `ghc -M` invoked over `Parser.hs`.
Cheers, Sebastian
Fingers crossed and all the best!
-- Shayne Fletcher Language Engineer / +1 917 699 7663 Digital Asset, creators of DAML
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html. If you are not the intended recipient, please delete this message._______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Shayne, out of curiosity, could you find out which of the three modules DsMonad, FamInst and TcSimplify lead to the blowup? If it's not too much of a hassle, that is. These are the only imports of PmOracle that aren't already exported from ghc-lib-parser. Cheers, Sebastian Am Mo., 16. Sept. 2019 um 22:38 Uhr schrieb Shayne Fletcher < shayne.fletcher@daml.com>:
Hi Sebastian,
On Mon, Sep 16, 2019 at 5:23 PM Sebastian Graf <sgraf1337@gmail.com> wrote:
Hi Shayne,
Sorry to hear that! We didn't consider modularity at all and I would be happy to try to refactor in a way that would allow `ghc-lib-parser` to be properly separated again. I'm fairly certain that I didn't directly touch anything parser related, but apparently the new cyclic import of PmOracle within TcRnTypes (which is also exposed from `ghc-lib-parser`) pulled in the other half of GHC. I'll see if I can fix that tomorrow, if only by extracting a separate `Types`-style module.
That sounds awesome. Tremendous. Thank-you! Please feel free to reach out to me if there's anything I can do to help your analysis[*]!
[*] For the record, the procedure for calculating the `ghc-lib-parser` modules is a little complicated by there needing to be some generated equivalents of `.hsc` files present for this to work but the procedure is at the end of the day just `ghc -M` invoked over `Parser.hs`.
Cheers, Sebastian
Fingers crossed and all the best!
-- *Shayne Fletcher* Language Engineer */* +1 917 699 7663 *Digital Asset* <https://digitalasset.com/>, creators of *DAML <https://daml.com/>*
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html. If you are not the intended recipient, please delete this message.
Seeing what I can do. On Tue, Sep 17, 2019 at 10:31 AM Sebastian Graf <sgraf1337@gmail.com> wrote:
Shayne,
out of curiosity, could you find out which of the three modules DsMonad, FamInst and TcSimplify lead to the blowup? If it's not too much of a hassle, that is. These are the only imports of PmOracle that aren't already exported from ghc-lib-parser.
Cheers, Sebastian
Am Mo., 16. Sept. 2019 um 22:38 Uhr schrieb Shayne Fletcher < shayne.fletcher@daml.com>:
Hi Sebastian,
On Mon, Sep 16, 2019 at 5:23 PM Sebastian Graf <sgraf1337@gmail.com> wrote:
Hi Shayne,
Sorry to hear that! We didn't consider modularity at all and I would be happy to try to refactor in a way that would allow `ghc-lib-parser` to be properly separated again. I'm fairly certain that I didn't directly touch anything parser related, but apparently the new cyclic import of PmOracle within TcRnTypes (which is also exposed from `ghc-lib-parser`) pulled in the other half of GHC. I'll see if I can fix that tomorrow, if only by extracting a separate `Types`-style module.
That sounds awesome. Tremendous. Thank-you! Please feel free to reach out to me if there's anything I can do to help your analysis[*]!
[*] For the record, the procedure for calculating the `ghc-lib-parser` modules is a little complicated by there needing to be some generated equivalents of `.hsc` files present for this to work but the procedure is at the end of the day just `ghc -M` invoked over `Parser.hs`.
Cheers, Sebastian
Fingers crossed and all the best!
-- *Shayne Fletcher* Language Engineer */* +1 917 699 7663 *Digital Asset* <https://digitalasset.com/>, creators of *DAML <https://daml.com/>*
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html. If you are not the intended recipient, please delete this message.
-- *Shayne Fletcher* Language Engineer */* +1 917 699 7663 *Digital Asset* <https://digitalasset.com/>, creators of *DAML <https://daml.com/>* -- This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html <http://www.digitalasset.com/emaildisclaimer.html>. If you are not the intended recipient, please delete this message.
Getting you some raw data to look at but first results indicate at least that `DsMonad` is pulling in `nativeGen`, `llvmGen`, `stgToCmm`, `simplCore`, `hieFile` and so on. On Tue, Sep 17, 2019 at 12:05 PM Shayne Fletcher <shayne.fletcher@daml.com> wrote:
Seeing what I can do.
On Tue, Sep 17, 2019 at 10:31 AM Sebastian Graf <sgraf1337@gmail.com> wrote:
Shayne,
out of curiosity, could you find out which of the three modules DsMonad, FamInst and TcSimplify lead to the blowup? If it's not too much of a hassle, that is. These are the only imports of PmOracle that aren't already exported from ghc-lib-parser.
Cheers, Sebastian
Am Mo., 16. Sept. 2019 um 22:38 Uhr schrieb Shayne Fletcher < shayne.fletcher@daml.com>:
Hi Sebastian,
On Mon, Sep 16, 2019 at 5:23 PM Sebastian Graf <sgraf1337@gmail.com> wrote:
Hi Shayne,
Sorry to hear that! We didn't consider modularity at all and I would be happy to try to refactor in a way that would allow `ghc-lib-parser` to be properly separated again. I'm fairly certain that I didn't directly touch anything parser related, but apparently the new cyclic import of PmOracle within TcRnTypes (which is also exposed from `ghc-lib-parser`) pulled in the other half of GHC. I'll see if I can fix that tomorrow, if only by extracting a separate `Types`-style module.
That sounds awesome. Tremendous. Thank-you! Please feel free to reach out to me if there's anything I can do to help your analysis[*]!
[*] For the record, the procedure for calculating the `ghc-lib-parser` modules is a little complicated by there needing to be some generated equivalents of `.hsc` files present for this to work but the procedure is at the end of the day just `ghc -M` invoked over `Parser.hs`.
Cheers, Sebastian
Fingers crossed and all the best!
-- *Shayne Fletcher* Language Engineer */* +1 917 699 7663 *Digital Asset* <https://digitalasset.com/>, creators of *DAML <https://daml.com/>*
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html. If you are not the intended recipient, please delete this message.
-- *Shayne Fletcher* Language Engineer */* +1 917 699 7663 *Digital Asset* <https://digitalasset.com/>, creators of *DAML <https://daml.com/>*
-- *Shayne Fletcher* Language Engineer */* +1 917 699 7663 *Digital Asset* <https://digitalasset.com/>, creators of *DAML <https://daml.com/>* -- This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html <http://www.digitalasset.com/emaildisclaimer.html>. If you are not the intended recipient, please delete this message.
Excellent work Sebastian. I confirm that the situation is resolved by https://gitlab.haskell.org/ghc/ghc/merge_requests/1743. On Mon, Sep 16, 2019 at 5:23 PM Sebastian Graf <sgraf1337@gmail.com> wrote:
Hi Shayne,
Sorry to hear that! We didn't consider modularity at all and I would be happy to try to refactor in a way that would allow `ghc-lib-parser` to be properly separated again. I'm fairly certain that I didn't directly touch anything parser related, but apparently the new cyclic import of PmOracle within TcRnTypes (which is also exposed from `ghc-lib-parser`) pulled in the other half of GHC. I'll see if I can fix that tomorrow, if only by extracting a separate `Types`-style module.
Cheers, Sebastian
Am Mo., 16. Sept. 2019 um 22:04 Uhr schrieb Shayne Fletcher via ghc-devs < ghc-devs@haskell.org>:
Some time back, the `ghc-lib` project split into two targets : `ghc-lib-parser` for those projects that just need to produce syntax trees and `ghc-lib` (re-exporting `ghc-lib-parser` modules) having the remaining modules for projects that need to go on and distill parse trees to Core. The idea of course was to reduce build times for tools like hlint that only need parse trees.
Roughly, `ghc-lib-parser` got about 200 files and `ghc-lib` 300. Today after landing `7915afc6bb9539a4534db99aeb6616a6d145918a`, "Encode shape information in `PmOracle`", `ghc-lib-parser` now needs 543 files and `ghc-lib` gets just 25.
That may be just bad luck for `ghc-lib-parser` and the way it has to be but I thought I should at least mention the knock-on effect of this change on the modularity of the GHC API in case this consequence hasn't been considered?
-- *Shayne Fletcher* Language Engineer */* +1 917 699 7663 *Digital Asset* <https://digitalasset.com/>, creators of *DAML <https://daml.com/>*
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html. If you are not the intended recipient, please delete this message. _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
-- *Shayne Fletcher* Language Engineer */* +1 917 699 7663 *Digital Asset* <https://digitalasset.com/>, creators of *DAML <https://daml.com/>* -- This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html <http://www.digitalasset.com/emaildisclaimer.html>. If you are not the intended recipient, please delete this message.
participants (3)
-
Matthew Pickering -
Sebastian Graf -
Shayne Fletcher