Re: clearing GHCi (and, by extension, hint) loaded module dependencies

newer
GHC static binaries with...

older
ghc-6.12.3 package base - build...

Alex Suraci

21 Sep 2010 21 Sep '10

8:19 p.m.

On Tue, Sep 21, 2010 at 10:30 AM, Daniel Gorín wrote:

...

Hi

What I make of this is that you should run a new interpreter (i.e. use runInterpreter(T)) instead of calling reset.

Could you try this this approach and see if it works?

Daniel

Every "load:" call already runs its own interpreter. I guess the session is sticking around somewhere in-between. I've been playing around with hint and the GHC API trying to figure this one out (with Simon's tips in mind), but haven't found a way to discard the session, or even on what level that should be done. In runInterpreter? Or in separate runGhc calls? The best guess I have so far is using getSession/setSession inside the interpreter call to restore it to the way it was before, but that seems to have no effect. Here's what I'm using to test: http://hpaste.org/40015/hintghc_memory Even with setSession, the cache seems to be sticking around - subsequent loads are faster, and the memory usage doesn't go down. -- Alex

Show replies by date

Simon Marlow

23 Sep 23 Sep

7:11 a.m.

New subject: clearing GHCi (and, by extension, hint) loaded module dependencies

On 22/09/2010 01:19, Alex Suraci wrote:

...

On Tue, Sep 21, 2010 at 10:30 AM, Daniel Gorín wrote:

...
Hi

What I make of this is that you should run a new interpreter (i.e. use runInterpreter(T)) instead of calling reset.

Could you try this this approach and see if it works?

Daniel

Every "load:" call already runs its own interpreter. I guess the session is sticking around somewhere in-between.

I've been playing around with hint and the GHC API trying to figure this one out (with Simon's tips in mind), but haven't found a way to discard the session, or even on what level that should be done. In runInterpreter? Or in separate runGhc calls? The best guess I have so far is using getSession/setSession inside the interpreter call to restore it to the way it was before, but that seems to have no effect.

Separate runGhc calls should be enough. Session is not a value, it is a reference. To reset it, you need to actually discard the old one and make a new one, which is what runGhc does. If you are making separate runGhc calls and still don't see the memory being reclaimed, then there might be a leak - but it's hard to see where, since everything is reachable from the Session only. I suppose we have some global linker state which might be holding onto stuff, but as you say that loads are faster in subsequent runs, that implies that you're using the same Session. Cheers, Simon

Alex Suraci

26 Sep 26 Sep

1:14 p.m.

New subject: clearing GHCi (and, by extension, hint) loaded module dependencies

On Sep 23, 2010, at 7:11 AM, Simon Marlow wrote:

...

If you are making separate runGhc calls and still don't see the memory being reclaimed, then there might be a leak - but it's hard to see where, since everything is reachable from the Session only. I suppose we have some global linker state which might be holding onto stuff, but as you say that loads are faster in subsequent runs, that implies that you're using the same Session.

Hm, I'm not doing any explicit Session handling/reusing. A new hint interpreter is run with every "load", and nothing is saved or continued between "load"s; by the time the interpreted function is executed, the interpreter has completed and its session is no longer needed. If the Session is sticking around anywhere, I'm not sure where it would be. Slightly verbose experimentation results: As an experiment I just tried adding hint's InterpreterT onto my language's VM stack, so that it's always running one continuous interpreter, and calling "reset" after interpreting the "load" function. An empty script bumped the usage by ~70MB, and subsequent loads of the same script had no discernible increase and completed instantly. (It seems the "reset" hint function has no effect on this.) Loading another script after the empty one (one that actually does something - an interface for the Snap webserver) only bumped up the RAM usage modestly (30MB), implying that some of its dependencies were already loaded. Doing the reverse (snap script -> empty script) jumped the RAM usage up by ~100MB initially, and loading the empty script repeatedly had no increase and completed instantly. Repeating these tests with separate runInterpreter calls (how it was before) has the same characteristics, except that repeated loads of already-loaded scripts bump up the RAM usage slightly (~8-10MB for empty, ~20-40MB for snap). Hope this helps, - Alex

Simon Marlow

5 Oct 5 Oct

9:25 a.m.

New subject: clearing GHCi (and, by extension, hint) loaded module dependencies

On 26/09/2010 18:14, Alex Suraci wrote:

...

On Sep 23, 2010, at 7:11 AM, Simon Marlow wrote:

...
If you are making separate runGhc calls and still don't see the memory being reclaimed, then there might be a leak - but it's hard to see where, since everything is reachable from the Session only. I suppose we have some global linker state which might be holding onto stuff, but as you say that loads are faster in subsequent runs, that implies that you're using the same Session.

Hm, I'm not doing any explicit Session handling/reusing. A new hint interpreter is run with every "load", and nothing is saved or continued between "load"s; by the time the interpreted function is executed, the interpreter has completed and its session is no longer needed. If the Session is sticking around anywhere, I'm not sure where it would be.

Slightly verbose experimentation results:

As an experiment I just tried adding hint's InterpreterT onto my language's VM stack, so that it's always running one continuous interpreter, and calling "reset" after interpreting the "load" function. An empty script bumped the usage by ~70MB, and subsequent loads of the same script had no discernible increase and completed instantly. (It seems the "reset" hint function has no effect on this.)

Loading another script after the empty one (one that actually does something - an interface for the Snap webserver) only bumped up the RAM usage modestly (30MB), implying that some of its dependencies were already loaded. Doing the reverse (snap script -> empty script) jumped the RAM usage up by ~100MB initially, and loading the empty script repeatedly had no increase and completed instantly.

Repeating these tests with separate runInterpreter calls (how it was before) has the same characteristics, except that repeated loads of already-loaded scripts bump up the RAM usage slightly (~8-10MB for empty, ~20-40MB for snap).

How are you measuring the memory usage, btw? Cheers, Simon

Alex Suraci

9:32 a.m.

New subject: clearing GHCi (and, by extension, hint) loaded module dependencies

On Oct 5, 2010, at 9:25 AM, Simon Marlow wrote:

...

On 26/09/2010 18:14, Alex Suraci wrote:

...
On Sep 23, 2010, at 7:11 AM, Simon Marlow wrote:

...
If you are making separate runGhc calls and still don't see the memory being reclaimed, then there might be a leak - but it's hard to see where, since everything is reachable from the Session only. I suppose we have some global linker state which might be holding onto stuff, but as you say that loads are faster in subsequent runs, that implies that you're using the same Session.

Hm, I'm not doing any explicit Session handling/reusing. A new hint interpreter is run with every "load", and nothing is saved or continued between "load"s; by the time the interpreted function is executed, the interpreter has completed and its session is no longer needed. If the Session is sticking around anywhere, I'm not sure where it would be.

Slightly verbose experimentation results:

As an experiment I just tried adding hint's InterpreterT onto my language's VM stack, so that it's always running one continuous interpreter, and calling "reset" after interpreting the "load" function. An empty script bumped the usage by ~70MB, and subsequent loads of the same script had no discernible increase and completed instantly. (It seems the "reset" hint function has no effect on this.)

Loading another script after the empty one (one that actually does something - an interface for the Snap webserver) only bumped up the RAM usage modestly (30MB), implying that some of its dependencies were already loaded. Doing the reverse (snap script -> empty script) jumped the RAM usage up by ~100MB initially, and loading the empty script repeatedly had no increase and completed instantly.

Repeating these tests with separate runInterpreter calls (how it was before) has the same characteristics, except that repeated loads of already-loaded scripts bump up the RAM usage slightly (~8-10MB for empty, ~20-40MB for snap).

How are you measuring the memory usage, btw?

Cheers, Simon

I'm just looking in Activity Monitor and filtering my app's name, repeating the tests a few times to be sure. I can try again with some other method if you'd like. Also: I'll be putting together a smaller test-case soon. - Alex

Simon Marlow

10:50 a.m.

New subject: clearing GHCi (and, by extension, hint) loaded module dependencies

On 05/10/2010 14:32, Alex Suraci wrote:

...

On Oct 5, 2010, at 9:25 AM, Simon Marlow wrote:

...
On 26/09/2010 18:14, Alex Suraci wrote:

...
On Sep 23, 2010, at 7:11 AM, Simon Marlow wrote:

...
If you are making separate runGhc calls and still don't see the memory being reclaimed, then there might be a leak - but it's hard to see where, since everything is reachable from the Session only. I suppose we have some global linker state which might be holding onto stuff, but as you say that loads are faster in subsequent runs, that implies that you're using the same Session.

Hm, I'm not doing any explicit Session handling/reusing. A new hint interpreter is run with every "load", and nothing is saved or continued between "load"s; by the time the interpreted function is executed, the interpreter has completed and its session is no longer needed. If the Session is sticking around anywhere, I'm not sure where it would be.

Slightly verbose experimentation results:

As an experiment I just tried adding hint's InterpreterT onto my language's VM stack, so that it's always running one continuous interpreter, and calling "reset" after interpreting the "load" function. An empty script bumped the usage by ~70MB, and subsequent loads of the same script had no discernible increase and completed instantly. (It seems the "reset" hint function has no effect on this.)

Loading another script after the empty one (one that actually does something - an interface for the Snap webserver) only bumped up the RAM usage modestly (30MB), implying that some of its dependencies were already loaded. Doing the reverse (snap script -> empty script) jumped the RAM usage up by ~100MB initially, and loading the empty script repeatedly had no increase and completed instantly.

Repeating these tests with separate runInterpreter calls (how it was before) has the same characteristics, except that repeated loads of already-loaded scripts bump up the RAM usage slightly (~8-10MB for empty, ~20-40MB for snap).

How are you measuring the memory usage, btw?

Cheers, Simon

I'm just looking in Activity Monitor and filtering my app's name, repeating the tests a few times to be sure. I can try again with some other method if you'd like.

Also: I'll be putting together a smaller test-case soon.

Oh, in that case the results are to be expected. GHC prior to version 7.0.1 didn't release memory back to the OS even when the memory requirements of the program dropped: see http://hackage.haskell.org/trac/ghc/ticket/698 In 7.0.1 we do release memory back to the OS, although I think on Windows we're still not releasing address space. Cheers, Simon

Alex Suraci

6 Oct 6 Oct

3:26 p.m.

New subject: clearing GHCi (and, by extension, hint) loaded module dependencies

On Tue, Oct 5, 2010 at 10:50 AM, Simon Marlow wrote:

...

Oh, in that case the results are to be expected. GHC prior to version 7.0.1 didn't release memory back to the OS even when the memory requirements of the program dropped: see

http://hackage.haskell.org/trac/ghc/ticket/698

In 7.0.1 we do release memory back to the OS, although I think on Windows we're still not releasing address space.

Cheers, Simon

Oh. Well then, good call. :) I've got Atomo up and running on GHC 7 now. Very nice performance boosts! Plus memory appears to be freed, though I can't test loading as it panics: atomo $ atomo > load: "src/empty.hs" atomo: atomo: panic! (the 'impossible' happened) (GHC version 7.0.0.20100924 for i386-apple-darwin): Testing for extension flag Opt_Generics before flattening Please report this as a GHC bug: http://www.haskell.org/ghc/reportabug Daniel: I've attached a patch containing the changes I made to Hint to get it compiling (though not working apparently) on GHC 7. Hope it helps! -- Alex

Ian Lynagh

23 Oct 23 Oct

6:52 p.m.

New subject: clearing GHCi (and, by extension, hint) loaded module dependencies

On Wed, Oct 06, 2010 at 03:26:21PM -0400, Alex Suraci wrote:

...

I've got Atomo up and running on GHC 7 now. Very nice performance boosts! Plus memory appears to be freed, though I can't test loading as it panics:

atomo $ atomo > load: "src/empty.hs" atomo: atomo: panic! (the 'impossible' happened) (GHC version 7.0.0.20100924 for i386-apple-darwin): Testing for extension flag Opt_Generics before flattening

This will be fixed in the next RC. Thanks Ian

5323

Age (days ago)

5354

Last active (days ago)

List overview

Download

7 comments

3 participants

participants (3)

Alex Suraci
Ian Lynagh
Simon Marlow