Hi Gergo, I think there a few things to say here. 1. As Zubin points out we have recently been concerned with improving the memory usage of large module sessions (#25511, !13675, !13593) I imagine all these patches will greatly help the memory usage in your use case. 2. You are absolutely right that ModDetails can get forced and is never reset. If you try !13675, it should be much more easily possible to reset the ModDetails by writing into the IORef which stores each home package. 3. If you share your example or perhaps even a trace from ghc-debug then I will be happy to investigate further as it seems like a great test case for the work we have recently been doing. Cheers, Matt On Fri, Jan 17, 2025 at 10:31 AM Zubin Duggal wrote:

See https://gitlab.haskell.org/ghc/ghc/-/issues/25511

...

PUBLIC

Looking at this with ghc-debug, at least I can see why we have the huge memory usage when recreating ModDetails: there are lots of HscEnvs stored all over the heap (I assume in thunks inside already compiled modules), when I recreate the ModDetails and replace them in the HUG, I am accumulating more and more copies of the same MOdDetails, since other HscEnvs are of course still pointing to the HUG that had the previous ModDetails, and so on.

Given this, I don't really know yet if remaking the ModDetails would help me or not, since trying it is now blocking on figuring out how I could avoid having multiple HscEnvs in memory at the same time...

From: ghc-devs On Behalf Of Gergo Érdi Sent: Friday, January 17, 2025 4:08 PM To: GHC Devs Cc: Montelatici, Raphael Laurent Subject: [External] GHC memory usage when typechecking from source vs. loading ModIfaces

Hi,

I'm using the GHC API to typecheck 35,000 modules that form a complicated dependency graph (with multiple top-level modules, i.e. there's no single "god module" that would transitively depend on everything else), and I noticed that peak memory usage is wildly different when everything is done from scratch vs. when everything is loaded from files containing ModIfaces: 17G vs. 8G. This ratio replicates for smaller samples as well, e.g. 80M vs 33M for 407 modules.

I'm aware of https://gitlab.haskell.org/ghc/ghc/-/issues/13586< https://urldefense.com/v3/__https://gitlab.haskell.org/ghc/ghc/-/issues/13586__;!!ASp95G87aa5DoyK5mB3l!5iF7gCTy5JYm4P-C5-UCaCPVmrNbVCvKbNCIy59XuTRWVJOQGbsLyRBEOGuFElS5o9KuRzNhyzs$> and so when I finish typechecking a module, I take the resulting ModIface and create the ModDetails that ends up in the HomeUnitGraph from that. My understanding of Matt's original GHC fix in https://gitlab.haskell.org/ghc/ghc/-/merge_requests/5478< https://urldefense.com/v3/__https://gitlab.haskell.org/ghc/ghc/-/merge_requests/5478__;!!ASp95G87aa5DoyK5mB3l!5iF7gCTy5JYm4P-C5-UCaCPVmrNbVCvKbNCIy59XuTRWVJOQGbsLyRBEOGuFElS5o9KuEf-sjms$> is that it does the same, i.e. it only makes a fresh ModDetails only once

On 25/01/17 08:54, Erdi, Gergo via ghc-devs wrote: per module, after the ModIface is ready.

...

But of course that still means that ModDetails can only keep growing as

more and more parts of it are used for typechecking more and more dependants. Could that be the cause? I tried a crude experiment of "putting the toothpaste back in the tube" by replacing all ModDetails with a fresh one in the HUG after each finished typechecking , but that's a complete disaster for memory usage: even for the small 407 module example, the memory usage shoots up to 1.5G. I can imagine it's because imported Ids are probably not shared anymore between different importer modules.

...

Any ideas on how I could improve memory usage in the from-scratch case,

so that it's more similar to the from-ModIface case?

...

Thanks, Gergo

---------------------------------------------------------------------- This email and any attachments are confidential and may also be

privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries at https: //www.sc.com/en/our-locations

...

Where you have a Financial Markets relationship with Standard Chartered

PLC, Standard Chartered Bank and their subsidiaries (the "Group"), information on the regulatory standards we adhere to and how it may affect you can be found in our Regulatory Compliance Statement at https: // www.sc.com/rcs/ and Regulatory Compliance Disclosures at http: // www.sc.com/rcs/fm

...

Insofar as this communication is not sent by the Global Research team and

contains any market commentary, the market commentary has been prepared by the sales and/or trading desk of Standard Chartered Bank or its affiliate. It is not and does not constitute research material, independent research, recommendation or financial advice. Any market commentary is for information purpose only and shall not be relied on for any other purpose and is subject to the relevant disclaimers available at https: // www.sc.com/en/regulatory-disclosures/#market-disclaimer.

...

Insofar as this communication is sent by the Global Research team and

contains any research materials prepared by members of the team, the research material is for information purpose only and shall not be relied on for any other purpose, and is subject to the relevant disclaimers available at https: // research.sc.com/research/api/application/static/terms-and-conditions.

...

Insofar as this e-mail contains the term sheet for a proposed

transaction, by responding affirmatively to this e-mail, you agree that you have understood the terms and conditions in the attached term sheet and evaluated the merits and risks of the transaction. We may at times also request you to sign the term sheet to acknowledge the same.

...

Please visit https: //www.sc.com/en/regulatory-disclosures/dodd-frank/

for important information with respect to derivative products.

...

---------------------------------------------------------------------- This email and any attachments are confidential and may also be

privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our public website.

...

---------------------------------------------------------------------- This email and any attachments are confidential and may also be

privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

...

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Thanks Gergo, I think that unless we have access to your code base or a realistic example then the before vs after snapshot will not be so helpful. It's known that `ModDetails` will leak space like this. Let us know how it goes for you. Cheers, Matt On Fri, Jan 17, 2025 at 11:30 AM ÉRDI Gergő wrote:

On Fri, 17 Jan 2025, Matthew Pickering wrote:

...

1. As Zubin points out we have recently been concerned with improving the memory usage of large module sessions (#25511, !13675, !13593)

I imagine all these patches will greatly help the memory usage in your use case.

I'll try these out and report back.

...

2. You are absolutely right that ModDetails can get forced and is never reset.

If you try !13675, it should be much more easily possible to reset the ModDetails by writing into the IORef which stores each home package.

Yes, that makes sense.

...

3. If you share your example or perhaps even a trace from ghc-debug then I will be happy to investigate further as it seems like a great test case for the work we have recently been doing.

Untangling just the parts that exercise the GHC API from all the other in-house bits will be quite a lot of work. But if just a ghc-debug snapshot of e.g. a small example from scratch vs. from existing ModIfaces would be helpful (with e.g. the top HscEnv at the time of finishing all typechecking as a saved closure), I can provide that no prob.

Thanks, Gergo

That's good news. I don't think the first idea will do very much as there are other references to the final "HomeModInfo" not stored in the HPT. Have you constructed a time profile to determine why the runtime is higher? With the second approach you are certainly trading space usage for repeating work. If you actually do have a forest, then ideally you would replace the ModDetails after it will never be used again. You are likely also missing other patches important for memory usage. * https://gitlab.haskell.org/ghc/ghc/-/merge_requests/12582 * https://gitlab.haskell.org/ghc/ghc/-/merge_requests/12347 I can't comment about the 17 HPT, what do the retainer stacks look like in ghc-debug? PS. Please use eventlog2html so the profiles are readable! You can use it on .hp profiles. Cheers, Matt On Thu, Jan 23, 2025 at 3:19 AM Erdi, Gergo wrote:

PUBLIC

Hi Matt & Zubin,

Thanks for the help on this so far!

I managed to hack the linked MR onto 9.8.4 (see https://gitlab.haskell.org/cactus/ghc/-/tree/cactus/backport-13675) and basically it seems to do what it says on the tin on a small example (see attached heap profile examples for typechecking 4313 modules), but I am unsure how to actually use it.

So my understanding of the improvement here is that since now there is only one single HPT [*], I should be able to avoid unnecessary ballooning by doing two things:

- Evicting `HomeModInfo`s wholesale from the HPT that are not going to be needed anymore, because I am done with all modules that would transitively depend on them. This of course only makes sense when typechecking a forest. - Replacing remaining `HomeModInfo`s with new ones that contain the same ModInterface but the ModDetails is replaced with a fresh one from initModDetails.

The attached `-after` profile shows typechecking with both of these ideas implemented. The first one doesn’t seem to help much on its own, but it’s tricky to evaluate that because it is very dependent on the shape of the workload (how tree-y it is). But the second one shows some serious promise in curtailing memory usage. However, it is also very slow – even on this small example, you can see its effect. On my full 35k+ module example, it more than doubles the runtime.

What would be a good policy on when to replace ModDetails with thunks to avoid both the space leak and excessive rehydration churn?

Also, perhaps unrelated, perhaps not – what’s with all those lists?!

Thanks,

Gergo

[*] BTW is it normal that I am still seeing several (17 in a small test case involving a couple hundred modules) HPT constructors in the heap? (I hacked it locally to be a datatype instead of a newtype just so I can see it in the heap). I expected to see only one.

*From:* Matthew Pickering *Sent:* Tuesday, January 21, 2025 8:24 PM *To:* ÉRDI Gergő *Cc:* Zubin Duggal ; Erdi, Gergo ; Montelatici, Raphael Laurent ; GHC Devs < ghc-devs@haskell.org> *Subject:* [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

Thanks Gergo, I think that unless we have access to your code base or a realistic example then the before vs after snapshot will not be so helpful. It's known that `ModDetails` will leak space like this.

Let us know how it goes for you.

Cheers,

Matt

On Fri, Jan 17, 2025 at 11:30 AM ÉRDI Gergő wrote:

On Fri, 17 Jan 2025, Matthew Pickering wrote:

...

1. As Zubin points out we have recently been concerned with improving the memory usage of large module sessions (#25511, !13675, !13593)

I imagine all these patches will greatly help the memory usage in your use case.

I'll try these out and report back.

...

2. You are absolutely right that ModDetails can get forced and is never reset.

If you try !13675, it should be much more easily possible to reset the ModDetails by writing into the IORef which stores each home package.

Yes, that makes sense.

...

3. If you share your example or perhaps even a trace from ghc-debug then I will be happy to investigate further as it seems like a great test case for the work we have recently been doing.

Untangling just the parts that exercise the GHC API from all the other in-house bits will be quite a lot of work. But if just a ghc-debug snapshot of e.g. a small example from scratch vs. from existing ModIfaces would be helpful (with e.g. the top HscEnv at the time of finishing all typechecking as a saved closure), I can provide that no prob.

Thanks, Gergo

------------------------------ This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our public website. ------------------------------ This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

Perhaps I am confused as well. I would have to investigate to see what the precise situation was. On Thu, Jan 23, 2025 at 12:40 PM ÉRDI Gergő wrote:

I'll get back to you with details as I can.

But one thing I noticed is that contrary to my expectations, when replacing the ModDetails with fresh ones, the real cost comes not from the fact that we have to re-evaluate the various parts of the ModDetails, but rather the call to `fillModDetails` itself!

In other words, I can replicate most of the slowdown by calling `fillModDetails` on the interfaces in the HPT but throwing its result away and keeping the HPT as is. This sounds like bad news to me.

Your remark that there are HomeModInfos referenced outside the HPT confuses me. I thought the whole reason this could work, and the whole reason I can see memory usage improvement, is because the HomeModInfos are not stored outside the HPT? If they are, home come my replacing them in the HPT with fresh ModDetails helps with memory usage instead of making it worse?

On Thu, 23 Jan 2025, Matthew Pickering wrote:

...

That's good news.

I don't think the first idea will do very much as there are other references to the final "HomeModInfo" not stored in the HPT.

Have you constructed a time profile to determine why the runtime is higher? With the second approach you are certainly trading space usage for repeating work.

If you actually do have a forest, then ideally you would replace the ModDetails after it will never be used again.

You are likely also missing other patches important for memory usage.

* https://gitlab.haskell.org/ghc/ghc/-/merge_requests/12582 * https://gitlab.haskell.org/ghc/ghc/-/merge_requests/12347

I can't comment about the 17 HPT, what do the retainer stacks look like in ghc-debug?

PS. Please use eventlog2html so the profiles are readable! You can use it on .hp profiles.

Cheers,

Matt

Gergo, I wonder if you have got your condition the wrong way around. The only "safe" time to perform rehydration is AFTER the point it can never be used again. If you rehydrate it just before it is used then you will repeat work which has already been done. If you do this, you will always have a trade-off between space used and runtime. PS: The eventlog2html bug with -p rendering is fixed by https://github.com/mpickering/eventlog2html/pull/192 but you should just generate an eventlog anyway with `-l`. Cheers, Matt On Wed, Feb 5, 2025 at 7:03 AM Erdi, Gergo wrote:

PUBLIC

Hi Matt,

Thanks for your help so far!

One vacation later, I am back looking at this. Unfortunately, the latest results I am seeing only confuse me more.

I have this small test load of a 4313 module forest that I am typechecking. The baseline resource usage, i.e. before any tricks about rehydrating the ModDetails in the HPT, is 1 GB maximum residency, 113s MUT time and 87s GC time. My aim is to reduce the maximum residency with as little disruption as possible to the total runtime.

My first test was the completely brute-force approach of rehydrating every single ModDetails in the HPT after typechecking every single module. Of course, this has catastrophic runtime performance, since I end up re-re-re-re-rehydrating every ModDetails for a total of 8,443,380 times (not counting the initial rehydration just after typechecking to put it in the HPT). So I get 290s MUT time, 252s GC time. But, the max residency goes down to 490 MB, showing that the idea, at least in principle, has legs.

So far so good. But then my problem starts -- how do I get this max residency improvement with acceptable runtime? My idea was that when typechecking a module, it should only unfold parts of ModDetails that are its transitive dependencies, so it should be enough to rehydrate only those ModDetails. Since this still results in 3,603,206 rehydrations, I shouldn't be too optimistic about its performance, but it should still cut the overhead in half. When I try this out, I get MUT time of 257s, GC time of 186s. However, the max residency is 883 MB! But how is it possible that max residency is not the same 490 MB?!?! Does that mean typechecking a module can unfold parts of ModDetails that are not transitive dependencies of it? How would I track this down?

For reference, here is how I do the rehydration of the HPT, let me know if it seems fishy:

``` recreateModDetailsInHpt :: HscEnv -> [ModuleName] -> IO () recreateModDetailsInHpt hsc_env mods = do hpt <- readIORef hptr fixIO \hpt' -> do writeIORef hptr hpt' traverse recreate_hmi hpt pure () where hpt@HPT{ table = hptr } = hsc_HPT hsc_env

recreate_hmi hmi@(HomeModInfo iface _details linkable) | moduleName mod `elem` mods = do !fresh_details <- genModDetails hsc_env iface pure $ HomeModInfo iface fresh_details linkable

| otherwise = pure hmi where mod = mi_module iface ```

In summary, my questions going forward are:

* How come rehydrating transitive dependencies doesn't help as much for max residency as rehydrating all already-loaded modules?

* What exactly does GHC itself do to use this new mutable HPT feature to good effect? I'm sure it doesn't suffer from the above-described quadratic slowdown.

Thanks for the tip on the other two memory usage improvement MRs -- I haven't had time yet to backport them. !12582 in particular seems like it will need quite a bit of work to be applied on 9.8.

Unfortunately, I couldn't get eventlog2html to work -- if I pass an .hp file with the `-p` parameter, I get an HTML file that claims "This eventlog was generated without heap profiling.".

Thanks, Gergo

From: Matthew Pickering Sent: Thursday, January 23, 2025 5:51 PM To: Erdi, Gergo Cc: ÉRDI Gergő ; Zubin Duggal ; Montelatici, Raphael Laurent ; GHC Devs < ghc-devs@haskell.org> Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

That's good news.

I don't think the first idea will do very much as there are other references to the final "HomeModInfo" not stored in the HPT.

Have you constructed a time profile to determine why the runtime is higher? With the second approach you are certainly trading space usage for repeating work.

If you actually do have a forest, then ideally you would replace the ModDetails after it will never be used again.

You are likely also missing other patches important for memory usage.

* https://urldefense.com/v3/__https://gitlab.haskell.org/ghc/ghc/-/merge_reque... * https://urldefense.com/v3/__https://gitlab.haskell.org/ghc/ghc/-/merge_reque...

I can't comment about the 17 HPT, what do the retainer stacks look like in ghc-debug?

PS. Please use eventlog2html so the profiles are readable! You can use it on .hp profiles.

Cheers,

Matt

On Thu, Jan 23, 2025 at 3:19 AM Erdi, Gergo mailto:Gergo.Erdi@sc.com wrote: PUBLIC

Hi Matt & Zubin,

Thanks for the help on this so far!

I managed to hack the linked MR onto 9.8.4 (see https://urldefense.com/v3/__https://gitlab.haskell.org/cactus/ghc/-/tree/cac...) and basically it seems to do what it says on the tin on a small example (see attached heap profile examples for typechecking 4313 modules), but I am unsure how to actually use it.

So my understanding of the improvement here is that since now there is only one single HPT [*], I should be able to avoid unnecessary ballooning by doing two things:

• Evicting `HomeModInfo`s wholesale from the HPT that are not going to be needed anymore, because I am done with all modules that would transitively depend on them. This of course only makes sense when typechecking a forest. • Replacing remaining `HomeModInfo`s with new ones that contain the same ModInterface but the ModDetails is replaced with a fresh one from initModDetails.

The attached `-after` profile shows typechecking with both of these ideas implemented. The first one doesn’t seem to help much on its own, but it’s tricky to evaluate that because it is very dependent on the shape of the workload (how tree-y it is). But the second one shows some serious promise in curtailing memory usage. However, it is also very slow – even on this small example, you can see its effect. On my full 35k+ module example, it more than doubles the runtime.

What would be a good policy on when to replace ModDetails with thunks to avoid both the space leak and excessive rehydration churn?

Also, perhaps unrelated, perhaps not – what’s with all those lists?!

Thanks, Gergo

[*] BTW is it normal that I am still seeing several (17 in a small test case involving a couple hundred modules) HPT constructors in the heap? (I hacked it locally to be a datatype instead of a newtype just so I can see it in the heap). I expected to see only one.

From: Matthew Pickering mailto:matthewtpickering@gmail.com Sent: Tuesday, January 21, 2025 8:24 PM To: ÉRDI Gergő mailto:gergo@erdi.hu Cc: Zubin Duggal mailto:zubin@well-typed.com; Erdi, Gergo ; Montelatici, Raphael Laurent ; GHC Devs mailto:ghc-devs@haskell.org Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

Thanks Gergo, I think that unless we have access to your code base or a realistic example then the before vs after snapshot will not be so helpful. It's known that `ModDetails` will leak space like this.

Let us know how it goes for you.

Cheers,

Matt

On Fri, Jan 17, 2025 at 11:30 AM ÉRDI Gergő mailto:gergo@erdi.hu wrote: On Fri, 17 Jan 2025, Matthew Pickering wrote:

...

1. As Zubin points out we have recently been concerned with improving the memory usage of large module sessions (#25511, !13675, !13593)

I imagine all these patches will greatly help the memory usage in your use case.

I'll try these out and report back.

...

2. You are absolutely right that ModDetails can get forced and is never reset.

If you try !13675, it should be much more easily possible to reset the ModDetails by writing into the IORef which stores each home package.

Yes, that makes sense.

...

3. If you share your example or perhaps even a trace from ghc-debug then I will be happy to investigate further as it seems like a great test case for the work we have recently been doing.

Untangling just the parts that exercise the GHC API from all the other in-house bits will be quite a lot of work. But if just a ghc-debug snapshot of e.g. a small example from scratch vs. from existing ModIfaces would be helpful (with e.g. the top HscEnv at the time of finishing all typechecking as a saved closure), I can provide that no prob.

Thanks, Gergo ________________________________________ This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our public website. ________________________________________ This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

Sure, you can remove them once you are sure they are not used anymore. For clients like `GHCi` that doesn't work obviously as they can be used at any point in the future but for a batch compiler it would be fine. On Mon, Feb 10, 2025 at 11:56 AM ÉRDI Gergő wrote:

On Mon, 10 Feb 2025, Matthew Pickering wrote:

...

I wonder if you have got your condition the wrong way around.

The only "safe" time to perform rehydration is AFTER the point it can never be used again.

If you rehydrate it just before it is used then you will repeat work which has already been done. If you do this, you will always have a trade-off between space used and runtime.

Oops. Yes, I have misunderstood the idea. I thought the idea was that after loading a given module into the HPT, its ModDetails would start out small (because of laziness) and then keep growing in size as more and more of it are traversed, and thus forced, during the typechecking of its dependees, so at some point we would want to reset that into the small initial representation as created by initModDetails.

But if the idea is that I should rehydrate modules when they can't be used anymore, then that brings up the question why even do that, instead of straight removing the HomeModInfos from the HPT?

PUBLIC Hi, Unfortunately, I am forced to return to this problem. Everything below is now in the context of GHC 9.12 plus the mutable HPT patch backported. My test case is typechecking a tree of 2294 modules that form the transitive closure of a single module's dependencies, all in a single process. I have done this typechecking three times, here's what `+RTS -s -RTS` reports for max residency: * "cold": With no on-disk `ModIface` files, i.e. from scratch: 537 MB * "cold-top": With all `ModIface`s already on disk, except for the single top-level module: 302 MB * "warm": With all `ModIface`s already on disk: 211 MB So my stupidly naive question is, why is the "cold" case also not 302 MB? In earlier discussion, `ModDetails` unfolding has come up. Dehydrating `ModDetails` in the HPT all the time is disastrous for runtime, but based on this model I would expect to see improvements from dehydrating "every now and then". So I tried a stupid simple example where after every 100th typechecked module, I run this function on the topologically sorted list of modules processed so far: ``` dehydrateHpt :: HscEnv -> [ModuleName] -> IO () dehydrateHpt hsc_env mods = do let HPT{ table = hptr } = hsc_HPT hsc_env hpt <- readIORef hptr for_ mods \mod -> for_ (lookupUDFM hpt mod) \(HomeModInfo iface _details _linkable) -> do !details <- initModDetails hsc_env iface pure () ``` Buuut the max residency is still 534 MB (see "cold-dehydrate"); in fact, the profile looks exactly the same. Speaking of the profile, in the "cold" case I see a lot of steadily increasing heap usage from the `ModuleGraph`. I could see this happening if typechecking from scratch involves more `modulesUnder` calls which in turn force more and more of the `ModuleGraph`. If so, then maybe this could be worked around by repeatedly remaking the `ModuleGraph` just like I remake the `ModDetails` above. I tried getting rid of this effect by `deepseq`'ing the `ModuleGraph` at the start, with the idea being that this should "equalize" the three scenarios if this really is a substantial source of extra memory usage. This pushes up the warm case's memory usage to 381 MB, which is promising, but I still see a `Word64Map` that is steadily increasing in the "cold-force-modulegraph" case and contributes a lot to the memory usage. Unfortunately, I don't know where that `Word64Map` is (it could be any `Unique`-keyed environment...). So I am now stuck at this point. To spell out my goal explicitly, I would like to typecheck one module after another and not keep anything more in memory around than if I loaded them from `ModIface` files. Thanks, Gergo p.s.: I couldn't find a way in the EventLog output HTML to turn event markers on/off or filter them, so to avoid covering the whole graph with gray lines, I mark only every 100th module. From: Matthew Pickering Sent: Wednesday, February 12, 2025 7:08 PM To: ÉRDI Gergő Cc: Erdi, Gergo ; Zubin Duggal ; Montelatici, Raphael Laurent ; GHC Devs Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces You do also raise a good point about rehydration costs. In oneshot mode, you are basically rehydrating the entire transitive closure of each module when you compile it, which obviously results in a large amount of repeated work. This is why people are investigating ideas of a persistent worker to at least avoid rehydrating all external dependencies as well. On Mon, Feb 10, 2025 at 12:13 PM Matthew Pickering mailto:matthewtpickering@gmail.com wrote: Sure, you can remove them once you are sure they are not used anymore. For clients like `GHCi` that doesn't work obviously as they can be used at any point in the future but for a batch compiler it would be fine. On Mon, Feb 10, 2025 at 11:56 AM ÉRDI Gergő mailto:gergo@erdi.hu wrote: On Mon, 10 Feb 2025, Matthew Pickering wrote:

I wonder if you have got your condition the wrong way around.

The only "safe" time to perform rehydration is AFTER the point it can never be used again.

If you rehydrate it just before it is used then you will repeat work which has already been done. If you do this, you will always have a trade-off between space used and runtime.

Oops. Yes, I have misunderstood the idea. I thought the idea was that after loading a given module into the HPT, its ModDetails would start out small (because of laziness) and then keep growing in size as more and more of it are traversed, and thus forced, during the typechecking of its dependees, so at some point we would want to reset that into the small initial representation as created by initModDetails. But if the idea is that I should rehydrate modules when they can't be used anymore, then that brings up the question why even do that, instead of straight removing the HomeModInfos from the HPT? ---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our public website. ---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

HI Gergo, Do you have a (synthetic?) reproducer? You have probably identified some memory leak. However, without any means to reproduce it becomes very difficult to investigate. I feel like we are getting into very precise details now, where speculating is not going to be so useful. It seems like this is an important thing for you and your company. Is there any budget to pay for some investigation? If that was the case then some effort could be made to create a synthetic producer and make the situation more robust going into the future if your requirements were precisely understood. Cheers, Matt On Fri, Mar 28, 2025 at 10:12 AM Erdi, Gergo wrote:

PUBLIC

Just to add that I get the same "equalizing" behaviour (but in a more "natural" way) if instead of deepseq-ing the ModuleGraph upfront, I just call `hugInstancesBelow` before processing each module. So that's definitely one source of extra memory usage. I wonder if it would be possible to rebuild the ModuleGraph periodically (similar to the ModDetails dehydration), or if there are references to it stored all over the place from `HscEnv`s scattered around in closures etc. (basically the same problem the HPT had before it was made into a mutable reference).

-----Original Message----- From: ghc-devs On Behalf Of Erdi, Gergo via ghc-devs Sent: Friday, March 28, 2025 4:49 PM To: Matthew Pickering ; GHC Devs < ghc-devs@haskell.org> Cc: ÉRDI Gergő ; Montelatici, Raphael Laurent < Raphael.Montelatici@sc.com>; Dijkstra, Atze Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

Hi,

Unfortunately, I am forced to return to this problem. Everything below is now in the context of GHC 9.12 plus the mutable HPT patch backported.

My test case is typechecking a tree of 2294 modules that form the transitive closure of a single module's dependencies, all in a single process. I have done this typechecking three times, here's what `+RTS -s -RTS` reports for max residency:

* "cold": With no on-disk `ModIface` files, i.e. from scratch: 537 MB

* "cold-top": With all `ModIface`s already on disk, except for the single top-level module: 302 MB

* "warm": With all `ModIface`s already on disk: 211 MB

So my stupidly naive question is, why is the "cold" case also not 302 MB?

In earlier discussion, `ModDetails` unfolding has come up. Dehydrating `ModDetails` in the HPT all the time is disastrous for runtime, but based on this model I would expect to see improvements from dehydrating "every now and then". So I tried a stupid simple example where after every 100th typechecked module, I run this function on the topologically sorted list of modules processed so far:

``` dehydrateHpt :: HscEnv -> [ModuleName] -> IO () dehydrateHpt hsc_env mods = do let HPT{ table = hptr } = hsc_HPT hsc_env hpt <- readIORef hptr for_ mods \mod -> for_ (lookupUDFM hpt mod) \(HomeModInfo iface _details _linkable) -> do !details <- initModDetails hsc_env iface pure () ```

Buuut the max residency is still 534 MB (see "cold-dehydrate"); in fact, the profile looks exactly the same.

Speaking of the profile, in the "cold" case I see a lot of steadily increasing heap usage from the `ModuleGraph`. I could see this happening if typechecking from scratch involves more `modulesUnder` calls which in turn force more and more of the `ModuleGraph`. If so, then maybe this could be worked around by repeatedly remaking the `ModuleGraph` just like I remake the `ModDetails` above. I tried getting rid of this effect by `deepseq`'ing the `ModuleGraph` at the start, with the idea being that this should "equalize" the three scenarios if this really is a substantial source of extra memory usage. This pushes up the warm case's memory usage to 381 MB, which is promising, but I still see a `Word64Map` that is steadily increasing in the "cold-force-modulegraph" case and contributes a lot to the memory usage. Unfortunately, I don't know where that `Word64Map` is (it could be any `Unique`-keyed environment...).

So I am now stuck at this point. To spell out my goal explicitly, I would like to typecheck one module after another and not keep anything more in memory around than if I loaded them from `ModIface` files.

Thanks, Gergo

p.s.: I couldn't find a way in the EventLog output HTML to turn event markers on/off or filter them, so to avoid covering the whole graph with gray lines, I mark only every 100th module.

From: Matthew Pickering Sent: Wednesday, February 12, 2025 7:08 PM To: ÉRDI Gergő Cc: Erdi, Gergo ; Zubin Duggal ; Montelatici, Raphael Laurent ; GHC Devs < ghc-devs@haskell.org> Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

You do also raise a good point about rehydration costs.

In oneshot mode, you are basically rehydrating the entire transitive closure of each module when you compile it, which obviously results in a large amount of repeated work. This is why people are investigating ideas of a persistent worker to at least avoid rehydrating all external dependencies as well.

On Mon, Feb 10, 2025 at 12:13 PM Matthew Pickering wrote: Sure, you can remove them once you are sure they are not used anymore.

For clients like `GHCi` that doesn't work obviously as they can be used at any point in the future but for a batch compiler it would be fine.

On Mon, Feb 10, 2025 at 11:56 AM ÉRDI Gergő mailto:gergo@erdi.hu wrote: On Mon, 10 Feb 2025, Matthew Pickering wrote:

...

I wonder if you have got your condition the wrong way around.

The only "safe" time to perform rehydration is AFTER the point it can never be used again.

If you rehydrate it just before it is used then you will repeat work which has already been done. If you do this, you will always have a trade-off between space used and runtime.

Oops. Yes, I have misunderstood the idea. I thought the idea was that after loading a given module into the HPT, its ModDetails would start out small (because of laziness) and then keep growing in size as more and more of it are traversed, and thus forced, during the typechecking of its dependees, so at some point we would want to reset that into the small initial representation as created by initModDetails.

But if the idea is that I should rehydrate modules when they can't be used anymore, then that brings up the question why even do that, instead of straight removing the HomeModInfos from the HPT?

---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our public website.

---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

PUBLIC This sounds extremely interesting, but I don’t understand where you are getting this number from! How do you see in the eventlog HTMLs that I’ve included that there are ~2000 ModuleGraphs? I’ve now tried using ghc-debug to find all ModuleGraph constructors at two points in the run: just before typechecking the first module (after all the extendMG calls) and just after typechecking the last module, and even in the cold case I only see 1 ModuleGraph before and 13 ModuleGraphs after. Also, what do you mean by “precisely one loaded per interface loaded into the EPS”? Since my repro has 2294 modules, wouldn’t that mean 2294 ModuleGraphs by that metric? From: Matthew Pickering Sent: Saturday, March 29, 2025 1:53 AM To: Erdi, Gergo Cc: GHC Devs ; ÉRDI Gergő ; Montelatici, Raphael Laurent ; Dijkstra, Atze Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces Hi Gergo, I quickly tried building `Cabal` with the master branch. There is precisely 1 ModuleGraph allocated for the home session, and precisely one loaded per interface loaded into the EPS. No leaky behaviour like you can see in your eventlogs. It seems there are about 2000 live module graphs in your program, are you doing something with the API to create this many? Cheers, Matt On Fri, Mar 28, 2025 at 12:40 PM Matthew Pickering mailto:matthewtpickering@gmail.com> wrote: HI Gergo, Do you have a (synthetic?) reproducer? You have probably identified some memory leak. However, without any means to reproduce it becomes very difficult to investigate. I feel like we are getting into very precise details now, where speculating is not going to be so useful. It seems like this is an important thing for you and your company. Is there any budget to pay for some investigation? If that was the case then some effort could be made to create a synthetic producer and make the situation more robust going into the future if your requirements were precisely understood. Cheers, Matt On Fri, Mar 28, 2025 at 10:12 AM Erdi, Gergo mailto:Gergo.Erdi@sc.com> wrote: PUBLIC Just to add that I get the same "equalizing" behaviour (but in a more "natural" way) if instead of deepseq-ing the ModuleGraph upfront, I just call `hugInstancesBelow` before processing each module. So that's definitely one source of extra memory usage. I wonder if it would be possible to rebuild the ModuleGraph periodically (similar to the ModDetails dehydration), or if there are references to it stored all over the place from `HscEnv`s scattered around in closures etc. (basically the same problem the HPT had before it was made into a mutable reference). -----Original Message----- From: ghc-devs mailto:ghc-devs-bounces@haskell.org> On Behalf Of Erdi, Gergo via ghc-devs Sent: Friday, March 28, 2025 4:49 PM To: Matthew Pickering mailto:matthewtpickering@gmail.com>; GHC Devs mailto:ghc-devs@haskell.org> Cc: ÉRDI Gergő mailto:gergo@erdi.hu>; Montelatici, Raphael Laurent mailto:Raphael.Montelatici@sc.com>; Dijkstra, Atze mailto:Atze.Dijkstra@sc.com> Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces Hi, Unfortunately, I am forced to return to this problem. Everything below is now in the context of GHC 9.12 plus the mutable HPT patch backported. My test case is typechecking a tree of 2294 modules that form the transitive closure of a single module's dependencies, all in a single process. I have done this typechecking three times, here's what `+RTS -s -RTS` reports for max residency: * "cold": With no on-disk `ModIface` files, i.e. from scratch: 537 MB * "cold-top": With all `ModIface`s already on disk, except for the single top-level module: 302 MB * "warm": With all `ModIface`s already on disk: 211 MB So my stupidly naive question is, why is the "cold" case also not 302 MB? In earlier discussion, `ModDetails` unfolding has come up. Dehydrating `ModDetails` in the HPT all the time is disastrous for runtime, but based on this model I would expect to see improvements from dehydrating "every now and then". So I tried a stupid simple example where after every 100th typechecked module, I run this function on the topologically sorted list of modules processed so far: ``` dehydrateHpt :: HscEnv -> [ModuleName] -> IO () dehydrateHpt hsc_env mods = do let HPT{ table = hptr } = hsc_HPT hsc_env hpt <- readIORef hptr for_ mods \mod -> for_ (lookupUDFM hpt mod) \(HomeModInfo iface _details _linkable) -> do !details <- initModDetails hsc_env iface pure () ``` Buuut the max residency is still 534 MB (see "cold-dehydrate"); in fact, the profile looks exactly the same. Speaking of the profile, in the "cold" case I see a lot of steadily increasing heap usage from the `ModuleGraph`. I could see this happening if typechecking from scratch involves more `modulesUnder` calls which in turn force more and more of the `ModuleGraph`. If so, then maybe this could be worked around by repeatedly remaking the `ModuleGraph` just like I remake the `ModDetails` above. I tried getting rid of this effect by `deepseq`'ing the `ModuleGraph` at the start, with the idea being that this should "equalize" the three scenarios if this really is a substantial source of extra memory usage. This pushes up the warm case's memory usage to 381 MB, which is promising, but I still see a `Word64Map` that is steadily increasing in the "cold-force-modulegraph" case and contributes a lot to the memory usage. Unfortunately, I don't know where that `Word64Map` is (it could be any `Unique`-keyed environment...). So I am now stuck at this point. To spell out my goal explicitly, I would like to typecheck one module after another and not keep anything more in memory around than if I loaded them from `ModIface` files. Thanks, Gergo p.s.: I couldn't find a way in the EventLog output HTML to turn event markers on/off or filter them, so to avoid covering the whole graph with gray lines, I mark only every 100th module. From: Matthew Pickering mailto:matthewtpickering@gmail.com> Sent: Wednesday, February 12, 2025 7:08 PM To: ÉRDI Gergő mailto:gergo@erdi.hu> Cc: Erdi, Gergo mailto:Gergo.Erdi@sc.com>; Zubin Duggal mailto:zubin@well-typed.com>; Montelatici, Raphael Laurent mailto:Raphael.Montelatici@sc.com>; GHC Devs mailto:ghc-devs@haskell.org> Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces You do also raise a good point about rehydration costs. In oneshot mode, you are basically rehydrating the entire transitive closure of each module when you compile it, which obviously results in a large amount of repeated work. This is why people are investigating ideas of a persistent worker to at least avoid rehydrating all external dependencies as well. On Mon, Feb 10, 2025 at 12:13 PM Matthew Pickering mailto:matthewtpickering@gmail.com> wrote: Sure, you can remove them once you are sure they are not used anymore. For clients like `GHCi` that doesn't work obviously as they can be used at any point in the future but for a batch compiler it would be fine. On Mon, Feb 10, 2025 at 11:56 AM ÉRDI Gergő mailto:gergo@erdi.hu> wrote: On Mon, 10 Feb 2025, Matthew Pickering wrote:

I wonder if you have got your condition the wrong way around.

The only "safe" time to perform rehydration is AFTER the point it can never be used again.

If you rehydrate it just before it is used then you will repeat work which has already been done. If you do this, you will always have a trade-off between space used and runtime.

Oops. Yes, I have misunderstood the idea. I thought the idea was that after loading a given module into the HPT, its ModDetails would start out small (because of laziness) and then keep growing in size as more and more of it are traversed, and thus forced, during the typechecking of its dependees, so at some point we would want to reset that into the small initial representation as created by initModDetails. But if the idea is that I should rehydrate modules when they can't be used anymore, then that brings up the question why even do that, instead of straight removing the HomeModInfos from the HPT? ---------------------------------------------------------------------- ---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our public website. ---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

Hi Gergo, I looked in the detailed pane, searched for ModuleGraph, hovered my mouse over the "ModuleGraph` constructor, recorded the number of live bytes, divided that by 32. Matt On Tue, Apr 1, 2025 at 7:04 AM Erdi, Gergo wrote:

PUBLIC

OK scratch that, I was looking at wrong ghc-debug output. Indeed there are 2301 ModuleGraphs in the heap at the end of typechecking :O

*From:* Erdi, Gergo *Sent:* Tuesday, April 1, 2025 12:51 PM *To:* Matthew Pickering *Cc:* GHC Devs ; ÉRDI Gergő ; Montelatici, Raphael Laurent ; Dijkstra, Atze *Subject:* Re: GHC memory usage when typechecking from source vs. loading ModIfaces

This sounds extremely interesting, but I don’t understand where you are getting this number from! How do you see in the eventlog HTMLs that I’ve included that there are ~2000 ModuleGraphs? I’ve now tried using ghc-debug to find all ModuleGraph constructors at two points in the run: just before typechecking the first module (after all the extendMG calls) and just after typechecking the last module, and even in the cold case I only see 1 ModuleGraph before and 13 ModuleGraphs after.

Also, what do you mean by “precisely one loaded per interface loaded into the EPS”? Since my repro has 2294 modules, wouldn’t that mean 2294 ModuleGraphs by that metric?

*From:* Matthew Pickering *Sent:* Saturday, March 29, 2025 1:53 AM *To:* Erdi, Gergo *Cc:* GHC Devs ; ÉRDI Gergő ; Montelatici, Raphael Laurent ; Dijkstra, Atze *Subject:* [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

Hi Gergo,

I quickly tried building `Cabal` with the master branch. There is precisely 1 ModuleGraph allocated for the home session, and precisely one loaded per interface loaded into the EPS. No leaky behaviour like you can see in your eventlogs.

It seems there are about 2000 live module graphs in your program, are you doing something with the API to create this many?

Cheers,

Matt

On Fri, Mar 28, 2025 at 12:40 PM Matthew Pickering < matthewtpickering@gmail.com> wrote:

HI Gergo,

Do you have a (synthetic?) reproducer? You have probably identified some memory leak. However, without any means to reproduce it becomes very difficult to investigate. I feel like we are getting into very precise details now, where speculating is not going to be so useful.

It seems like this is an important thing for you and your company. Is there any budget to pay for some investigation? If that was the case then some effort could be made to create a synthetic producer and make the situation more robust going into the future if your requirements were precisely understood.

Cheers,

Matt

On Fri, Mar 28, 2025 at 10:12 AM Erdi, Gergo wrote:

PUBLIC

Just to add that I get the same "equalizing" behaviour (but in a more "natural" way) if instead of deepseq-ing the ModuleGraph upfront, I just call `hugInstancesBelow` before processing each module. So that's definitely one source of extra memory usage. I wonder if it would be possible to rebuild the ModuleGraph periodically (similar to the ModDetails dehydration), or if there are references to it stored all over the place from `HscEnv`s scattered around in closures etc. (basically the same problem the HPT had before it was made into a mutable reference).

-----Original Message----- From: ghc-devs On Behalf Of Erdi, Gergo via ghc-devs Sent: Friday, March 28, 2025 4:49 PM To: Matthew Pickering ; GHC Devs < ghc-devs@haskell.org> Cc: ÉRDI Gergő ; Montelatici, Raphael Laurent < Raphael.Montelatici@sc.com>; Dijkstra, Atze Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

Hi,

Unfortunately, I am forced to return to this problem. Everything below is now in the context of GHC 9.12 plus the mutable HPT patch backported.

My test case is typechecking a tree of 2294 modules that form the transitive closure of a single module's dependencies, all in a single process. I have done this typechecking three times, here's what `+RTS -s -RTS` reports for max residency:

* "cold": With no on-disk `ModIface` files, i.e. from scratch: 537 MB

* "cold-top": With all `ModIface`s already on disk, except for the single top-level module: 302 MB

* "warm": With all `ModIface`s already on disk: 211 MB

So my stupidly naive question is, why is the "cold" case also not 302 MB?

In earlier discussion, `ModDetails` unfolding has come up. Dehydrating `ModDetails` in the HPT all the time is disastrous for runtime, but based on this model I would expect to see improvements from dehydrating "every now and then". So I tried a stupid simple example where after every 100th typechecked module, I run this function on the topologically sorted list of modules processed so far:

``` dehydrateHpt :: HscEnv -> [ModuleName] -> IO () dehydrateHpt hsc_env mods = do let HPT{ table = hptr } = hsc_HPT hsc_env hpt <- readIORef hptr for_ mods \mod -> for_ (lookupUDFM hpt mod) \(HomeModInfo iface _details _linkable) -> do !details <- initModDetails hsc_env iface pure () ```

Buuut the max residency is still 534 MB (see "cold-dehydrate"); in fact, the profile looks exactly the same.

Speaking of the profile, in the "cold" case I see a lot of steadily increasing heap usage from the `ModuleGraph`. I could see this happening if typechecking from scratch involves more `modulesUnder` calls which in turn force more and more of the `ModuleGraph`. If so, then maybe this could be worked around by repeatedly remaking the `ModuleGraph` just like I remake the `ModDetails` above. I tried getting rid of this effect by `deepseq`'ing the `ModuleGraph` at the start, with the idea being that this should "equalize" the three scenarios if this really is a substantial source of extra memory usage. This pushes up the warm case's memory usage to 381 MB, which is promising, but I still see a `Word64Map` that is steadily increasing in the "cold-force-modulegraph" case and contributes a lot to the memory usage. Unfortunately, I don't know where that `Word64Map` is (it could be any `Unique`-keyed environment...).

So I am now stuck at this point. To spell out my goal explicitly, I would like to typecheck one module after another and not keep anything more in memory around than if I loaded them from `ModIface` files.

Thanks, Gergo

p.s.: I couldn't find a way in the EventLog output HTML to turn event markers on/off or filter them, so to avoid covering the whole graph with gray lines, I mark only every 100th module.

From: Matthew Pickering Sent: Wednesday, February 12, 2025 7:08 PM To: ÉRDI Gergő Cc: Erdi, Gergo ; Zubin Duggal ; Montelatici, Raphael Laurent ; GHC Devs < ghc-devs@haskell.org> Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

You do also raise a good point about rehydration costs.

In oneshot mode, you are basically rehydrating the entire transitive closure of each module when you compile it, which obviously results in a large amount of repeated work. This is why people are investigating ideas of a persistent worker to at least avoid rehydrating all external dependencies as well.

On Mon, Feb 10, 2025 at 12:13 PM Matthew Pickering wrote: Sure, you can remove them once you are sure they are not used anymore.

For clients like `GHCi` that doesn't work obviously as they can be used at any point in the future but for a batch compiler it would be fine.

On Mon, Feb 10, 2025 at 11:56 AM ÉRDI Gergő mailto:gergo@erdi.hu wrote: On Mon, 10 Feb 2025, Matthew Pickering wrote:

...

I wonder if you have got your condition the wrong way around.

The only "safe" time to perform rehydration is AFTER the point it can never be used again.

If you rehydrate it just before it is used then you will repeat work which has already been done. If you do this, you will always have a trade-off between space used and runtime.

Oops. Yes, I have misunderstood the idea. I thought the idea was that after loading a given module into the HPT, its ModDetails would start out small (because of laziness) and then keep growing in size as more and more of it are traversed, and thus forced, during the typechecking of its dependees, so at some point we would want to reset that into the small initial representation as created by initModDetails.

But if the idea is that I should rehydrate modules when they can't be used anymore, then that brings up the question why even do that, instead of straight removing the HomeModInfos from the HPT?

----------------------------------------------------------------------

------------------------------ This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our public website. ------------------------------ This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

Hi Gergo, Since this is a non-reproducible issue with your custom code, I think that we've reached the limit of free debugging via email. There are several consultancies which can be employed to investigate complicated issues like this and perform the necessary changes to GHC to make your use-case easier. If you have a reproducer on a distributed GHC release then it could be investigated. Cheers, Matt On Tue, Apr 1, 2025 at 1:24 PM Erdi, Gergo wrote:

PUBLIC

OK so this `ModuleGraph` leak has the potential to be a huge explainer because of the large number of ModuleNodes and NodeKey_Modules in the heap profile. It turns out the reason we had one ModuleGraph per module was because I was populating the ModuleGraph by repeated calls to `extendMG`, where the new edges were created via `mgModSummaries` on the old `ModuleGraph`, and so the new edge list was a thunk pointing to the old `ModuleGraph`.

I’ve now changed it to a single `mkModuleGraph` call, which gets rid of the extra ModuleGraphs (see attached profile), but unfortunately, the ModuleNodes/ edge NodeKey_Modules are still not shared. Here’s how I build the graph, trying hard with a Data.Map.Map to ensure only use one NodeKey_Module per each target module:

```

registerModulesToGraph :: GhcMonad m => [(UnitId, ModuleName, [ModuleName])] -> m ()

registerModulesToGraph entries = modifySessionM \hsc_env -> do

mod_deps <- for entries \(unit, mod_name, deps) -> do

mod <- liftIO $ registerToModuleFinder hsc_env unit mod_name

let ms = dummyModSummary (hsc_dflags hsc_env) mod deps fingerprint0

pure (ms, deps)

let nodes = [ModuleNode (edges deps) ms | (ms, deps) <- mod_deps]

pure hsc_env{ hsc_mod_graph = mkModuleGraph nodes }

where

edge_map :: Map.Map ModuleName NodeKey

edge_map = Map.fromList [(mod_name, NodeKey_Module $ ModNodeKeyWithUid (notBoot mod_name) uid) | (uid, mod_name, _) <- entries]

edges deps = map (edge_map Map.!) deps

registerToModuleFinder :: HscEnv -> UnitId -> ModuleName -> IO Module

registerToModuleFinder hsc_env unitId modName = do

addHomeModuleToFinder (hsc_FC hsc_env) homeUnit (notBoot modName) location

where

homeUnit = DefiniteHomeUnit unitId Nothing

location = dummyModLocation modName

```

But, as you can see in the attached profile, I still get way more than 2k ModuleNodes and NodeKey_Modules….

*From:* Matthew Pickering *Sent:* Tuesday, April 1, 2025 5:44 PM *To:* Erdi, Gergo *Cc:* GHC Devs ; ÉRDI Gergő ; Montelatici, Raphael Laurent ; Dijkstra, Atze *Subject:* [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

Hi Gergo,

I looked in the detailed pane, searched for ModuleGraph, hovered my mouse over the "ModuleGraph` constructor, recorded the number of live bytes, divided that by 32.

Matt

On Tue, Apr 1, 2025 at 7:04 AM Erdi, Gergo wrote:

PUBLIC

OK scratch that, I was looking at wrong ghc-debug output. Indeed there are 2301 ModuleGraphs in the heap at the end of typechecking :O

*From:* Erdi, Gergo *Sent:* Tuesday, April 1, 2025 12:51 PM *To:* Matthew Pickering *Cc:* GHC Devs ; ÉRDI Gergő ; Montelatici, Raphael Laurent ; Dijkstra, Atze *Subject:* Re: GHC memory usage when typechecking from source vs. loading ModIfaces

This sounds extremely interesting, but I don’t understand where you are getting this number from! How do you see in the eventlog HTMLs that I’ve included that there are ~2000 ModuleGraphs? I’ve now tried using ghc-debug to find all ModuleGraph constructors at two points in the run: just before typechecking the first module (after all the extendMG calls) and just after typechecking the last module, and even in the cold case I only see 1 ModuleGraph before and 13 ModuleGraphs after.

Also, what do you mean by “precisely one loaded per interface loaded into the EPS”? Since my repro has 2294 modules, wouldn’t that mean 2294 ModuleGraphs by that metric?

*From:* Matthew Pickering *Sent:* Saturday, March 29, 2025 1:53 AM *To:* Erdi, Gergo *Cc:* GHC Devs ; ÉRDI Gergő ; Montelatici, Raphael Laurent ; Dijkstra, Atze *Subject:* [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

Hi Gergo,

I quickly tried building `Cabal` with the master branch. There is precisely 1 ModuleGraph allocated for the home session, and precisely one loaded per interface loaded into the EPS. No leaky behaviour like you can see in your eventlogs.

It seems there are about 2000 live module graphs in your program, are you doing something with the API to create this many?

Cheers,

Matt

On Fri, Mar 28, 2025 at 12:40 PM Matthew Pickering < matthewtpickering@gmail.com> wrote:

HI Gergo,

Do you have a (synthetic?) reproducer? You have probably identified some memory leak. However, without any means to reproduce it becomes very difficult to investigate. I feel like we are getting into very precise details now, where speculating is not going to be so useful.

It seems like this is an important thing for you and your company. Is there any budget to pay for some investigation? If that was the case then some effort could be made to create a synthetic producer and make the situation more robust going into the future if your requirements were precisely understood.

Cheers,

Matt

On Fri, Mar 28, 2025 at 10:12 AM Erdi, Gergo wrote:

PUBLIC

Just to add that I get the same "equalizing" behaviour (but in a more "natural" way) if instead of deepseq-ing the ModuleGraph upfront, I just call `hugInstancesBelow` before processing each module. So that's definitely one source of extra memory usage. I wonder if it would be possible to rebuild the ModuleGraph periodically (similar to the ModDetails dehydration), or if there are references to it stored all over the place from `HscEnv`s scattered around in closures etc. (basically the same problem the HPT had before it was made into a mutable reference).

-----Original Message----- From: ghc-devs On Behalf Of Erdi, Gergo via ghc-devs Sent: Friday, March 28, 2025 4:49 PM To: Matthew Pickering ; GHC Devs < ghc-devs@haskell.org> Cc: ÉRDI Gergő ; Montelatici, Raphael Laurent < Raphael.Montelatici@sc.com>; Dijkstra, Atze Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

Hi,

Unfortunately, I am forced to return to this problem. Everything below is now in the context of GHC 9.12 plus the mutable HPT patch backported.

My test case is typechecking a tree of 2294 modules that form the transitive closure of a single module's dependencies, all in a single process. I have done this typechecking three times, here's what `+RTS -s -RTS` reports for max residency:

* "cold": With no on-disk `ModIface` files, i.e. from scratch: 537 MB

* "cold-top": With all `ModIface`s already on disk, except for the single top-level module: 302 MB

* "warm": With all `ModIface`s already on disk: 211 MB

So my stupidly naive question is, why is the "cold" case also not 302 MB?

In earlier discussion, `ModDetails` unfolding has come up. Dehydrating `ModDetails` in the HPT all the time is disastrous for runtime, but based on this model I would expect to see improvements from dehydrating "every now and then". So I tried a stupid simple example where after every 100th typechecked module, I run this function on the topologically sorted list of modules processed so far:

``` dehydrateHpt :: HscEnv -> [ModuleName] -> IO () dehydrateHpt hsc_env mods = do let HPT{ table = hptr } = hsc_HPT hsc_env hpt <- readIORef hptr for_ mods \mod -> for_ (lookupUDFM hpt mod) \(HomeModInfo iface _details _linkable) -> do !details <- initModDetails hsc_env iface pure () ```

Buuut the max residency is still 534 MB (see "cold-dehydrate"); in fact, the profile looks exactly the same.

Speaking of the profile, in the "cold" case I see a lot of steadily increasing heap usage from the `ModuleGraph`. I could see this happening if typechecking from scratch involves more `modulesUnder` calls which in turn force more and more of the `ModuleGraph`. If so, then maybe this could be worked around by repeatedly remaking the `ModuleGraph` just like I remake the `ModDetails` above. I tried getting rid of this effect by `deepseq`'ing the `ModuleGraph` at the start, with the idea being that this should "equalize" the three scenarios if this really is a substantial source of extra memory usage. This pushes up the warm case's memory usage to 381 MB, which is promising, but I still see a `Word64Map` that is steadily increasing in the "cold-force-modulegraph" case and contributes a lot to the memory usage. Unfortunately, I don't know where that `Word64Map` is (it could be any `Unique`-keyed environment...).

So I am now stuck at this point. To spell out my goal explicitly, I would like to typecheck one module after another and not keep anything more in memory around than if I loaded them from `ModIface` files.

Thanks, Gergo

p.s.: I couldn't find a way in the EventLog output HTML to turn event markers on/off or filter them, so to avoid covering the whole graph with gray lines, I mark only every 100th module.

From: Matthew Pickering Sent: Wednesday, February 12, 2025 7:08 PM To: ÉRDI Gergő Cc: Erdi, Gergo ; Zubin Duggal ; Montelatici, Raphael Laurent ; GHC Devs < ghc-devs@haskell.org> Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

You do also raise a good point about rehydration costs.

In oneshot mode, you are basically rehydrating the entire transitive closure of each module when you compile it, which obviously results in a large amount of repeated work. This is why people are investigating ideas of a persistent worker to at least avoid rehydrating all external dependencies as well.

On Mon, Feb 10, 2025 at 12:13 PM Matthew Pickering wrote: Sure, you can remove them once you are sure they are not used anymore.

For clients like `GHCi` that doesn't work obviously as they can be used at any point in the future but for a batch compiler it would be fine.

On Mon, Feb 10, 2025 at 11:56 AM ÉRDI Gergő mailto:gergo@erdi.hu wrote: On Mon, 10 Feb 2025, Matthew Pickering wrote:

...

I wonder if you have got your condition the wrong way around.

The only "safe" time to perform rehydration is AFTER the point it can never be used again.

If you rehydrate it just before it is used then you will repeat work which has already been done. If you do this, you will always have a trade-off between space used and runtime.

Oops. Yes, I have misunderstood the idea. I thought the idea was that after loading a given module into the HPT, its ModDetails would start out small (because of laziness) and then keep growing in size as more and more of it are traversed, and thus forced, during the typechecking of its dependees, so at some point we would want to reset that into the small initial representation as created by initModDetails.

But if the idea is that I should rehydrate modules when they can't be used anymore, then that brings up the question why even do that, instead of straight removing the HomeModInfos from the HPT?

----------------------------------------------------------------------

------------------------------ This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our public website. ------------------------------ This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

What command do I run to generate the files from this patch file? Perhaps a link to a git repo would be a suitable way to share the reproducer? On Wed, Apr 2, 2025 at 10:26 AM Erdi, Gergo wrote:

PUBLIC

Hi Matt,

I think I have something that might demonstrate that GHC (at least GHC 9.12.1) might have a similar problem!

With the attached vacuous module hierarchy, I tried compiling M2294 from scratch, and then with `.hi` files for everything except the toplevel module. I did the same with our GHC-API-using compiler as well. As you can see from the attached event logs, while the details differ, the overall shape of the memory used by ModuleGraph edges (750k of GWIB and NodeKey_Module constructors for the 2321 ModuleNodes and ~60k direct dependency edges) is pretty much the same between our compiler and GHC 9.12, suggesting to me that GHC is duplicating ModuleGraph node information in the dependency edges when building the transitive closure.

Based on these measurements, do you agree that this is a GHC-side problem of memory usage scaling quadratically with the number of dependency edges?

Thanks,

Gergo

p.s.: Sorry for including the reproducer module tree in this weird format as a patch file, but I am behind a mail server that won’t let me send mails with too many individual files in attached archives…

*From:* Matthew Pickering *Sent:* Friday, March 28, 2025 8:40 PM *To:* Erdi, Gergo *Cc:* GHC Devs ; ÉRDI Gergő ; Montelatici, Raphael Laurent ; Dijkstra, Atze *Subject:* [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

HI Gergo,

Do you have a (synthetic?) reproducer? You have probably identified some memory leak. However, without any means to reproduce it becomes very difficult to investigate. I feel like we are getting into very precise details now, where speculating is not going to be so useful.

It seems like this is an important thing for you and your company. Is there any budget to pay for some investigation? If that was the case then some effort could be made to create a synthetic producer and make the situation more robust going into the future if your requirements were precisely understood.

Cheers,

Matt

On Fri, Mar 28, 2025 at 10:12 AM Erdi, Gergo wrote:

PUBLIC

Just to add that I get the same "equalizing" behaviour (but in a more "natural" way) if instead of deepseq-ing the ModuleGraph upfront, I just call `hugInstancesBelow` before processing each module. So that's definitely one source of extra memory usage. I wonder if it would be possible to rebuild the ModuleGraph periodically (similar to the ModDetails dehydration), or if there are references to it stored all over the place from `HscEnv`s scattered around in closures etc. (basically the same problem the HPT had before it was made into a mutable reference).

-----Original Message----- From: ghc-devs On Behalf Of Erdi, Gergo via ghc-devs Sent: Friday, March 28, 2025 4:49 PM To: Matthew Pickering ; GHC Devs < ghc-devs@haskell.org> Cc: ÉRDI Gergő ; Montelatici, Raphael Laurent < Raphael.Montelatici@sc.com>; Dijkstra, Atze Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

Hi,

Unfortunately, I am forced to return to this problem. Everything below is now in the context of GHC 9.12 plus the mutable HPT patch backported.

My test case is typechecking a tree of 2294 modules that form the transitive closure of a single module's dependencies, all in a single process. I have done this typechecking three times, here's what `+RTS -s -RTS` reports for max residency:

* "cold": With no on-disk `ModIface` files, i.e. from scratch: 537 MB

* "cold-top": With all `ModIface`s already on disk, except for the single top-level module: 302 MB

* "warm": With all `ModIface`s already on disk: 211 MB

So my stupidly naive question is, why is the "cold" case also not 302 MB?

In earlier discussion, `ModDetails` unfolding has come up. Dehydrating `ModDetails` in the HPT all the time is disastrous for runtime, but based on this model I would expect to see improvements from dehydrating "every now and then". So I tried a stupid simple example where after every 100th typechecked module, I run this function on the topologically sorted list of modules processed so far:

``` dehydrateHpt :: HscEnv -> [ModuleName] -> IO () dehydrateHpt hsc_env mods = do let HPT{ table = hptr } = hsc_HPT hsc_env hpt <- readIORef hptr for_ mods \mod -> for_ (lookupUDFM hpt mod) \(HomeModInfo iface _details _linkable) -> do !details <- initModDetails hsc_env iface pure () ```

Buuut the max residency is still 534 MB (see "cold-dehydrate"); in fact, the profile looks exactly the same.

Speaking of the profile, in the "cold" case I see a lot of steadily increasing heap usage from the `ModuleGraph`. I could see this happening if typechecking from scratch involves more `modulesUnder` calls which in turn force more and more of the `ModuleGraph`. If so, then maybe this could be worked around by repeatedly remaking the `ModuleGraph` just like I remake the `ModDetails` above. I tried getting rid of this effect by `deepseq`'ing the `ModuleGraph` at the start, with the idea being that this should "equalize" the three scenarios if this really is a substantial source of extra memory usage. This pushes up the warm case's memory usage to 381 MB, which is promising, but I still see a `Word64Map` that is steadily increasing in the "cold-force-modulegraph" case and contributes a lot to the memory usage. Unfortunately, I don't know where that `Word64Map` is (it could be any `Unique`-keyed environment...).

So I am now stuck at this point. To spell out my goal explicitly, I would like to typecheck one module after another and not keep anything more in memory around than if I loaded them from `ModIface` files.

Thanks, Gergo

p.s.: I couldn't find a way in the EventLog output HTML to turn event markers on/off or filter them, so to avoid covering the whole graph with gray lines, I mark only every 100th module.

From: Matthew Pickering Sent: Wednesday, February 12, 2025 7:08 PM To: ÉRDI Gergő Cc: Erdi, Gergo ; Zubin Duggal ; Montelatici, Raphael Laurent ; GHC Devs < ghc-devs@haskell.org> Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

You do also raise a good point about rehydration costs.

In oneshot mode, you are basically rehydrating the entire transitive closure of each module when you compile it, which obviously results in a large amount of repeated work. This is why people are investigating ideas of a persistent worker to at least avoid rehydrating all external dependencies as well.

On Mon, Feb 10, 2025 at 12:13 PM Matthew Pickering wrote: Sure, you can remove them once you are sure they are not used anymore.

For clients like `GHCi` that doesn't work obviously as they can be used at any point in the future but for a batch compiler it would be fine.

On Mon, Feb 10, 2025 at 11:56 AM ÉRDI Gergő mailto:gergo@erdi.hu wrote: On Mon, 10 Feb 2025, Matthew Pickering wrote:

...

I wonder if you have got your condition the wrong way around.

The only "safe" time to perform rehydration is AFTER the point it can never be used again.

If you rehydrate it just before it is used then you will repeat work which has already been done. If you do this, you will always have a trade-off between space used and runtime.

Oops. Yes, I have misunderstood the idea. I thought the idea was that after loading a given module into the HPT, its ModDetails would start out small (because of laziness) and then keep growing in size as more and more of it are traversed, and thus forced, during the typechecking of its dependees, so at some point we would want to reset that into the small initial representation as created by initModDetails.

But if the idea is that I should rehydrate modules when they can't be used anymore, then that brings up the question why even do that, instead of straight removing the HomeModInfos from the HPT?

---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our public website.

---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

------------------------------ This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

I think you are missing https://gitlab.haskell.org/ghc/ghc/-/merge_requests/13593 On HEAD I get maximum residency of about 200M, NodeKey usage is constant. On 9.10.1, I get maximum residency of 400M, NodeKey usage looks quadratic. On Wed, Apr 2, 2025 at 10:47 AM Erdi, Gergo wrote:

PUBLIC

zcat ../repro-hs.patch.gz |patch -p0

*From:* Matthew Pickering *Sent:* Wednesday, April 2, 2025 5:39 PM *To:* Erdi, Gergo *Cc:* GHC Devs ; ÉRDI Gergő ; Montelatici, Raphael Laurent ; Dijkstra, Atze *Subject:* [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

What command do I run to generate the files from this patch file? Perhaps a link to a git repo would be a suitable way to share the reproducer?

On Wed, Apr 2, 2025 at 10:26 AM Erdi, Gergo wrote:

PUBLIC

Hi Matt,

I think I have something that might demonstrate that GHC (at least GHC 9.12.1) might have a similar problem!

With the attached vacuous module hierarchy, I tried compiling M2294 from scratch, and then with `.hi` files for everything except the toplevel module. I did the same with our GHC-API-using compiler as well. As you can see from the attached event logs, while the details differ, the overall shape of the memory used by ModuleGraph edges (750k of GWIB and NodeKey_Module constructors for the 2321 ModuleNodes and ~60k direct dependency edges) is pretty much the same between our compiler and GHC 9.12, suggesting to me that GHC is duplicating ModuleGraph node information in the dependency edges when building the transitive closure.

Based on these measurements, do you agree that this is a GHC-side problem of memory usage scaling quadratically with the number of dependency edges?

Thanks,

Gergo

p.s.: Sorry for including the reproducer module tree in this weird format as a patch file, but I am behind a mail server that won’t let me send mails with too many individual files in attached archives…

*From:* Matthew Pickering *Sent:* Friday, March 28, 2025 8:40 PM *To:* Erdi, Gergo *Cc:* GHC Devs ; ÉRDI Gergő ; Montelatici, Raphael Laurent ; Dijkstra, Atze *Subject:* [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

HI Gergo,

Do you have a (synthetic?) reproducer? You have probably identified some memory leak. However, without any means to reproduce it becomes very difficult to investigate. I feel like we are getting into very precise details now, where speculating is not going to be so useful.

It seems like this is an important thing for you and your company. Is there any budget to pay for some investigation? If that was the case then some effort could be made to create a synthetic producer and make the situation more robust going into the future if your requirements were precisely understood.

Cheers,

Matt

On Fri, Mar 28, 2025 at 10:12 AM Erdi, Gergo wrote:

PUBLIC

Just to add that I get the same "equalizing" behaviour (but in a more "natural" way) if instead of deepseq-ing the ModuleGraph upfront, I just call `hugInstancesBelow` before processing each module. So that's definitely one source of extra memory usage. I wonder if it would be possible to rebuild the ModuleGraph periodically (similar to the ModDetails dehydration), or if there are references to it stored all over the place from `HscEnv`s scattered around in closures etc. (basically the same problem the HPT had before it was made into a mutable reference).

-----Original Message----- From: ghc-devs On Behalf Of Erdi, Gergo via ghc-devs Sent: Friday, March 28, 2025 4:49 PM To: Matthew Pickering ; GHC Devs < ghc-devs@haskell.org> Cc: ÉRDI Gergő ; Montelatici, Raphael Laurent < Raphael.Montelatici@sc.com>; Dijkstra, Atze Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

Hi,

Unfortunately, I am forced to return to this problem. Everything below is now in the context of GHC 9.12 plus the mutable HPT patch backported.

My test case is typechecking a tree of 2294 modules that form the transitive closure of a single module's dependencies, all in a single process. I have done this typechecking three times, here's what `+RTS -s -RTS` reports for max residency:

* "cold": With no on-disk `ModIface` files, i.e. from scratch: 537 MB

* "cold-top": With all `ModIface`s already on disk, except for the single top-level module: 302 MB

* "warm": With all `ModIface`s already on disk: 211 MB

So my stupidly naive question is, why is the "cold" case also not 302 MB?

In earlier discussion, `ModDetails` unfolding has come up. Dehydrating `ModDetails` in the HPT all the time is disastrous for runtime, but based on this model I would expect to see improvements from dehydrating "every now and then". So I tried a stupid simple example where after every 100th typechecked module, I run this function on the topologically sorted list of modules processed so far:

``` dehydrateHpt :: HscEnv -> [ModuleName] -> IO () dehydrateHpt hsc_env mods = do let HPT{ table = hptr } = hsc_HPT hsc_env hpt <- readIORef hptr for_ mods \mod -> for_ (lookupUDFM hpt mod) \(HomeModInfo iface _details _linkable) -> do !details <- initModDetails hsc_env iface pure () ```

Buuut the max residency is still 534 MB (see "cold-dehydrate"); in fact, the profile looks exactly the same.

Speaking of the profile, in the "cold" case I see a lot of steadily increasing heap usage from the `ModuleGraph`. I could see this happening if typechecking from scratch involves more `modulesUnder` calls which in turn force more and more of the `ModuleGraph`. If so, then maybe this could be worked around by repeatedly remaking the `ModuleGraph` just like I remake the `ModDetails` above. I tried getting rid of this effect by `deepseq`'ing the `ModuleGraph` at the start, with the idea being that this should "equalize" the three scenarios if this really is a substantial source of extra memory usage. This pushes up the warm case's memory usage to 381 MB, which is promising, but I still see a `Word64Map` that is steadily increasing in the "cold-force-modulegraph" case and contributes a lot to the memory usage. Unfortunately, I don't know where that `Word64Map` is (it could be any `Unique`-keyed environment...).

So I am now stuck at this point. To spell out my goal explicitly, I would like to typecheck one module after another and not keep anything more in memory around than if I loaded them from `ModIface` files.

Thanks, Gergo

p.s.: I couldn't find a way in the EventLog output HTML to turn event markers on/off or filter them, so to avoid covering the whole graph with gray lines, I mark only every 100th module.

From: Matthew Pickering Sent: Wednesday, February 12, 2025 7:08 PM To: ÉRDI Gergő Cc: Erdi, Gergo ; Zubin Duggal ; Montelatici, Raphael Laurent ; GHC Devs < ghc-devs@haskell.org> Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

You do also raise a good point about rehydration costs.

In oneshot mode, you are basically rehydrating the entire transitive closure of each module when you compile it, which obviously results in a large amount of repeated work. This is why people are investigating ideas of a persistent worker to at least avoid rehydrating all external dependencies as well.

On Mon, Feb 10, 2025 at 12:13 PM Matthew Pickering wrote: Sure, you can remove them once you are sure they are not used anymore.

For clients like `GHCi` that doesn't work obviously as they can be used at any point in the future but for a batch compiler it would be fine.

On Mon, Feb 10, 2025 at 11:56 AM ÉRDI Gergő mailto:gergo@erdi.hu wrote: On Mon, 10 Feb 2025, Matthew Pickering wrote:

...

I wonder if you have got your condition the wrong way around.

The only "safe" time to perform rehydration is AFTER the point it can never be used again.

If you rehydrate it just before it is used then you will repeat work which has already been done. If you do this, you will always have a trade-off between space used and runtime.

Oops. Yes, I have misunderstood the idea. I thought the idea was that after loading a given module into the HPT, its ModDetails would start out small (because of laziness) and then keep growing in size as more and more of it are traversed, and thus forced, during the typechecking of its dependees, so at some point we would want to reset that into the small initial representation as created by initModDetails.

But if the idea is that I should rehydrate modules when they can't be used anymore, then that brings up the question why even do that, instead of straight removing the HomeModInfos from the HPT?

---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our public website.

---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

------------------------------

This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

------------------------------ This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

Yes, but * With all the caveats around forcing ModDetails and using extra memory. * When you typecheck from scratch you may load additional interfaces for external dependencies into the EPS. After a module is typechecked, everything apart from the interface is discarded and everything is regenerated from that interface. It's possible there are some leaks in forcing the interface, since the deep forcing was incomplete (see https://gitlab.haskell.org/ghc/ghc/-/merge_requests/14078) Anyway, best thing is to profile with these invariants in mind, and fix situations where the invariants are violated. Cheers, Matt On Thu, Apr 3, 2025 at 11:54 AM Erdi, Gergo wrote:

PUBLIC

Yes I was, since I am using 9.12.1.

This is great stuff -- with !13593 backported, I don’t see any ModuleGraph-related stuff showing up in top spots of the heap profile! Thank you for pointing me towards this!

I’ll have to do some more digging to see if there is anything else concretely blameable now. But in theory, morally, abstractly, from the 10,000 feet view, would it be correct to say that there should be no memory usage difference between typechecking a large number of modules from scratch vs. loading the `.hi` files for those same modules? (Modulo, of course, the memory usage during any single module’s typechecking).

*From:* Matthew Pickering *Sent:* Wednesday, April 2, 2025 6:03 PM *To:* Erdi, Gergo *Cc:* GHC Devs ; ÉRDI Gergő ; Montelatici, Raphael Laurent ; Dijkstra, Atze *Subject:* [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

I think you are missing https://gitlab.haskell.org/ghc/ghc/-/merge_requests/13593 https://urldefense.com/v3/__https://gitlab.haskell.org/ghc/ghc/-/merge_reque...

On HEAD I get maximum residency of about 200M, NodeKey usage is constant.

On 9.10.1, I get maximum residency of 400M, NodeKey usage looks quadratic.

On Wed, Apr 2, 2025 at 10:47 AM Erdi, Gergo wrote:

PUBLIC

zcat ../repro-hs.patch.gz |patch -p0

*From:* Matthew Pickering *Sent:* Wednesday, April 2, 2025 5:39 PM *To:* Erdi, Gergo *Cc:* GHC Devs ; ÉRDI Gergő ; Montelatici, Raphael Laurent ; Dijkstra, Atze *Subject:* [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

What command do I run to generate the files from this patch file? Perhaps a link to a git repo would be a suitable way to share the reproducer?

On Wed, Apr 2, 2025 at 10:26 AM Erdi, Gergo wrote:

PUBLIC

Hi Matt,

I think I have something that might demonstrate that GHC (at least GHC 9.12.1) might have a similar problem!

With the attached vacuous module hierarchy, I tried compiling M2294 from scratch, and then with `.hi` files for everything except the toplevel module. I did the same with our GHC-API-using compiler as well. As you can see from the attached event logs, while the details differ, the overall shape of the memory used by ModuleGraph edges (750k of GWIB and NodeKey_Module constructors for the 2321 ModuleNodes and ~60k direct dependency edges) is pretty much the same between our compiler and GHC 9.12, suggesting to me that GHC is duplicating ModuleGraph node information in the dependency edges when building the transitive closure.

Based on these measurements, do you agree that this is a GHC-side problem of memory usage scaling quadratically with the number of dependency edges?

Thanks,

Gergo

p.s.: Sorry for including the reproducer module tree in this weird format as a patch file, but I am behind a mail server that won’t let me send mails with too many individual files in attached archives…

*From:* Matthew Pickering *Sent:* Friday, March 28, 2025 8:40 PM *To:* Erdi, Gergo *Cc:* GHC Devs ; ÉRDI Gergő ; Montelatici, Raphael Laurent ; Dijkstra, Atze *Subject:* [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

HI Gergo,

Do you have a (synthetic?) reproducer? You have probably identified some memory leak. However, without any means to reproduce it becomes very difficult to investigate. I feel like we are getting into very precise details now, where speculating is not going to be so useful.

It seems like this is an important thing for you and your company. Is there any budget to pay for some investigation? If that was the case then some effort could be made to create a synthetic producer and make the situation more robust going into the future if your requirements were precisely understood.

Cheers,

Matt

On Fri, Mar 28, 2025 at 10:12 AM Erdi, Gergo wrote:

PUBLIC

Just to add that I get the same "equalizing" behaviour (but in a more "natural" way) if instead of deepseq-ing the ModuleGraph upfront, I just call `hugInstancesBelow` before processing each module. So that's definitely one source of extra memory usage. I wonder if it would be possible to rebuild the ModuleGraph periodically (similar to the ModDetails dehydration), or if there are references to it stored all over the place from `HscEnv`s scattered around in closures etc. (basically the same problem the HPT had before it was made into a mutable reference).

-----Original Message----- From: ghc-devs On Behalf Of Erdi, Gergo via ghc-devs Sent: Friday, March 28, 2025 4:49 PM To: Matthew Pickering ; GHC Devs < ghc-devs@haskell.org> Cc: ÉRDI Gergő ; Montelatici, Raphael Laurent < Raphael.Montelatici@sc.com>; Dijkstra, Atze Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

Hi,

Unfortunately, I am forced to return to this problem. Everything below is now in the context of GHC 9.12 plus the mutable HPT patch backported.

My test case is typechecking a tree of 2294 modules that form the transitive closure of a single module's dependencies, all in a single process. I have done this typechecking three times, here's what `+RTS -s -RTS` reports for max residency:

* "cold": With no on-disk `ModIface` files, i.e. from scratch: 537 MB

* "cold-top": With all `ModIface`s already on disk, except for the single top-level module: 302 MB

* "warm": With all `ModIface`s already on disk: 211 MB

So my stupidly naive question is, why is the "cold" case also not 302 MB?

In earlier discussion, `ModDetails` unfolding has come up. Dehydrating `ModDetails` in the HPT all the time is disastrous for runtime, but based on this model I would expect to see improvements from dehydrating "every now and then". So I tried a stupid simple example where after every 100th typechecked module, I run this function on the topologically sorted list of modules processed so far:

``` dehydrateHpt :: HscEnv -> [ModuleName] -> IO () dehydrateHpt hsc_env mods = do let HPT{ table = hptr } = hsc_HPT hsc_env hpt <- readIORef hptr for_ mods \mod -> for_ (lookupUDFM hpt mod) \(HomeModInfo iface _details _linkable) -> do !details <- initModDetails hsc_env iface pure () ```

Buuut the max residency is still 534 MB (see "cold-dehydrate"); in fact, the profile looks exactly the same.

Speaking of the profile, in the "cold" case I see a lot of steadily increasing heap usage from the `ModuleGraph`. I could see this happening if typechecking from scratch involves more `modulesUnder` calls which in turn force more and more of the `ModuleGraph`. If so, then maybe this could be worked around by repeatedly remaking the `ModuleGraph` just like I remake the `ModDetails` above. I tried getting rid of this effect by `deepseq`'ing the `ModuleGraph` at the start, with the idea being that this should "equalize" the three scenarios if this really is a substantial source of extra memory usage. This pushes up the warm case's memory usage to 381 MB, which is promising, but I still see a `Word64Map` that is steadily increasing in the "cold-force-modulegraph" case and contributes a lot to the memory usage. Unfortunately, I don't know where that `Word64Map` is (it could be any `Unique`-keyed environment...).

So I am now stuck at this point. To spell out my goal explicitly, I would like to typecheck one module after another and not keep anything more in memory around than if I loaded them from `ModIface` files.

Thanks, Gergo

p.s.: I couldn't find a way in the EventLog output HTML to turn event markers on/off or filter them, so to avoid covering the whole graph with gray lines, I mark only every 100th module.

From: Matthew Pickering Sent: Wednesday, February 12, 2025 7:08 PM To: ÉRDI Gergő Cc: Erdi, Gergo ; Zubin Duggal ; Montelatici, Raphael Laurent ; GHC Devs < ghc-devs@haskell.org> Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

You do also raise a good point about rehydration costs.

In oneshot mode, you are basically rehydrating the entire transitive closure of each module when you compile it, which obviously results in a large amount of repeated work. This is why people are investigating ideas of a persistent worker to at least avoid rehydrating all external dependencies as well.

On Mon, Feb 10, 2025 at 12:13 PM Matthew Pickering wrote: Sure, you can remove them once you are sure they are not used anymore.

For clients like `GHCi` that doesn't work obviously as they can be used at any point in the future but for a batch compiler it would be fine.

On Mon, Feb 10, 2025 at 11:56 AM ÉRDI Gergő mailto:gergo@erdi.hu wrote: On Mon, 10 Feb 2025, Matthew Pickering wrote:

...

I wonder if you have got your condition the wrong way around.

The only "safe" time to perform rehydration is AFTER the point it can never be used again.

If you rehydrate it just before it is used then you will repeat work which has already been done. If you do this, you will always have a trade-off between space used and runtime.

Oops. Yes, I have misunderstood the idea. I thought the idea was that after loading a given module into the HPT, its ModDetails would start out small (because of laziness) and then keep growing in size as more and more of it are traversed, and thus forced, during the typechecking of its dependees, so at some point we would want to reset that into the small initial representation as created by initModDetails.

But if the idea is that I should rehydrate modules when they can't be used anymore, then that brings up the question why even do that, instead of straight removing the HomeModInfos from the HPT?

---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our public website.

---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

------------------------------

This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

------------------------------

This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

------------------------------ This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com