Re: heap profiling

On Sat, Jun 19, 2010 at 8:46 AM, Claus Reinke wrote:

...

...

I put the simple version at http://hpaste.org/fastcgi/hpaste.fcgi/view?id=26329#a26329

This one displays much better performance with DList + Writer.Strict than List + StrictWriter so I guess it's not too surprising. However, *something* is still generating a lot of lag

Some lag is to be expected: logging is about creating and recording data for later use. You'd need to annotate

Indeed, though I don't think this is the case, because I get lots of lag even when no logs are written.

In the part you deleted I mentioned one source of lag that does not disappear when no logs are written, and a way of using profiling cost centers to track down other sources (the ones I mentioned accounted for 2/3 of lag in the "profile_logging" profile, according to "-hblag -hc -L40" on SCC-annotated source).

My conclusion is that my contrived tests are not useful because the whole point is to get a controlled situation by simplifying it, but even the simple version displays mysterious behaviour.

It is sometimes difficult to produce a simplified version that has strictly fewer problems than the full version, especially if the simple version is not derived from the full version but from simplifying assumptions. Nevertheless, the simpler version has less code, and it is therefore helpful to try and understand its behaviour before returning to full complexity. In particular, it is important to figure out which problematic aspects of the simple version also apply to the full version and which problems can be ignored because they cannot occur in the full version.

I should go back to poking at the production version. Even if I don't understand the changes I'm making, at least when I get a speedup I'm more sure it translates to a real speed up in the application.

That way lies madness. Not recommended unless you are producing code that noone is ever going to look at again, and even then you'll be writing similar code in the future, so it is better to put in the energy once to puzzle out those mysterious details!-) Claus