PUBLIC
Looking at this with ghc-debug, at least I can see why we have the huge memory usage when recreating ModDetails: there are lots of HscEnvs stored all over the heap (I assume in thunks inside already compiled modules), when I recreate the ModDetails and replace them in the HUG, I am accumulating more and more copies of the same MOdDetails, since other HscEnvs are of course still pointing to the HUG that had the previous ModDetails, and so on.
Given this, I don't really know yet if remaking the ModDetails would help me or not, since trying it is now blocking on figuring out how I could avoid having multiple HscEnvs in memory at the same time...
From: ghc-devs On Behalf Of Gergo Érdi
Sent: Friday, January 17, 2025 4:08 PM
To: GHC Devs
Cc: Montelatici, Raphael Laurent
Subject: [External] GHC memory usage when typechecking from source vs. loading ModIfaces
Hi,
I'm using the GHC API to typecheck 35,000 modules that form a complicated dependency graph (with multiple top-level modules, i.e. there's no single "god module" that would transitively depend on everything else), and I noticed that peak memory usage is wildly different when everything is done from scratch vs. when everything is loaded from files containing ModIfaces: 17G vs. 8G. This ratio replicates for smaller samples as well, e.g. 80M vs 33M for 407 modules.
I'm aware of https://gitlab.haskell.org/ghc/ghc/-/issues/13586https://urldefense.com/v3/__https://gitlab.haskell.org/ghc/ghc/-/issues/1358... and so when I finish typechecking a module, I take the resulting ModIface and create the ModDetails that ends up in the HomeUnitGraph from that. My understanding of Matt's original GHC fix in https://gitlab.haskell.org/ghc/ghc/-/merge_requests/5478https://urldefense.com/v3/__https://gitlab.haskell.org/ghc/ghc/-/merge_reque... is that it does the same, i.e. it only makes a fresh ModDetails only once per module, after the ModIface is ready.
But of course that still means that ModDetails can only keep growing as more and more parts of it are used for typechecking more and more dependants. Could that be the cause? I tried a crude experiment of "putting the toothpaste back in the tube" by replacing all ModDetails with a fresh one in the HUG after each finished typechecking , but that's a complete disaster for memory usage: even for the small 407 module example, the memory usage shoots up to 1.5G. I can imagine it's because imported Ids are probably not shared anymore between different importer modules.
Any ideas on how I could improve memory usage in the from-scratch case, so that it's more similar to the from-ModIface case?
Thanks,
Gergo
----------------------------------------------------------------------
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries at https: //www.sc.com/en/our-locations
Where you have a Financial Markets relationship with Standard Chartered PLC, Standard Chartered Bank and their subsidiaries (the "Group"), information on the regulatory standards we adhere to and how it may affect you can be found in our Regulatory Compliance Statement at https: //www.sc.com/rcs/ and Regulatory Compliance Disclosures at http: //www.sc.com/rcs/fm
Insofar as this communication is not sent by the Global Research team and contains any market commentary, the market commentary has been prepared by the sales and/or trading desk of Standard Chartered Bank or its affiliate. It is not and does not constitute research material, independent research, recommendation or financial advice. Any market commentary is for information purpose only and shall not be relied on for any other purpose and is subject to the relevant disclaimers available at https: //www.sc.com/en/regulatory-disclosures/#market-disclaimer.
Insofar as this communication is sent by the Global Research team and contains any research materials prepared by members of the team, the research material is for information purpose only and shall not be relied on for any other purpose, and is subject to the relevant disclaimers available at https: //research.sc.com/research/api/application/static/terms-and-conditions.
Insofar as this e-mail contains the term sheet for a proposed transaction, by responding affirmatively to this e-mail, you agree that you have understood the terms and conditions in the attached term sheet and evaluated the merits and risks of the transaction. We may at times also request you to sign the term sheet to acknowledge the same.
Please visit https: //www.sc.com/en/regulatory-disclosures/dodd-frank/ for important information with respect to derivative products.
----------------------------------------------------------------------
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our public website.
----------------------------------------------------------------------
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com
