Where did these threads come from?
Hi all, I'm working on a server process that has (by design) fairly high memory usage but serves relatively few requests and spends most of its time idle. By 'fairly high' I mean 100s of MB, nothing ridiculous. I'm trying to improve its quiescent CPU usage as this is currently the limiting factor for how cheaply we can run this service - we're aiming for the cheapest t-class EC2 machine we can get. The quiescent CPU usage is currently about 50%. This seems to be largely due to the idle GC, because running with -I0 cuts it down to about 0.3%. Pretty much every time any thread wakes up it triggers the idle GC a short while later and because of the high memory usage these GCs are somewhat expensive. Using threadscope I found the biggest problem was the reaper thread in resource-pool which wakes up every second whether there's anything to reap or not. Fortunately this was easy to spot as the reaper thread labels itself as such. I've replaced this with an event-driven reaper that only wakes up on demand and cut the baseline from 50% down to just under 10%. There are two more threads I can now see waking up periodically in threadscope, with respective periods approximately 10 sec and 30 sec. I have a hunch what the 30-sec one is but no idea about the 10-sec one. If I can kill off the 10 sec one then that should take CPU usage down to <5% which would be just fine. My question: is there any reliable way to determine where these threads came from, in terms of the location of the forkIO call that made them? I've just been looking for likely candidates and labelling threads and this is rather tedious. Cheers, David
Ah yes, and the more obvious question: should I just run it with -I0? I'm
kind of assuming that the idle GC is a Good Thing since it's on by default.
Cheers,
On 13 November 2015 at 11:11, David Turner
Hi all,
I'm working on a server process that has (by design) fairly high memory usage but serves relatively few requests and spends most of its time idle. By 'fairly high' I mean 100s of MB, nothing ridiculous. I'm trying to improve its quiescent CPU usage as this is currently the limiting factor for how cheaply we can run this service - we're aiming for the cheapest t-class EC2 machine we can get.
The quiescent CPU usage is currently about 50%. This seems to be largely due to the idle GC, because running with -I0 cuts it down to about 0.3%. Pretty much every time any thread wakes up it triggers the idle GC a short while later and because of the high memory usage these GCs are somewhat expensive.
Using threadscope I found the biggest problem was the reaper thread in resource-pool which wakes up every second whether there's anything to reap or not. Fortunately this was easy to spot as the reaper thread labels itself as such. I've replaced this with an event-driven reaper that only wakes up on demand and cut the baseline from 50% down to just under 10%.
There are two more threads I can now see waking up periodically in threadscope, with respective periods approximately 10 sec and 30 sec. I have a hunch what the 30-sec one is but no idea about the 10-sec one. If I can kill off the 10 sec one then that should take CPU usage down to <5% which would be just fine.
My question: is there any reliable way to determine where these threads came from, in terms of the location of the forkIO call that made them? I've just been looking for likely candidates and labelling threads and this is rather tedious.
Cheers,
David
What idle GC does for you is tries to make you take the latency hit for
garbage collection when your program is not CPU-bound, rather than doing it
when you hit an arbitrary allocation threshold (probably during request
processing when you're actually doing work). So in that case it's a good
thing, yes --- but for yours I'd probably turn it off too. Other tweaks you
can try include tweaking the RTS gc settings if you haven't already,
especially increasing -A.
On Fri, Nov 13, 2015 at 3:50 AM, David Turner wrote: Ah yes, and the more obvious question: should I just run it with -I0? I'm
kind of assuming that the idle GC is a Good Thing since it's on by default. Cheers, On 13 November 2015 at 11:11, David Turner Hi all, I'm working on a server process that has (by design) fairly high memory
usage but serves relatively few requests and spends most of its time idle.
By 'fairly high' I mean 100s of MB, nothing ridiculous. I'm trying to
improve its quiescent CPU usage as this is currently the limiting factor
for how cheaply we can run this service - we're aiming for the cheapest
t-class EC2 machine we can get. The quiescent CPU usage is currently about 50%. This seems to be largely
due to the idle GC, because running with -I0 cuts it down to about 0.3%.
Pretty much every time any thread wakes up it triggers the idle GC a short
while later and because of the high memory usage these GCs are somewhat
expensive. Using threadscope I found the biggest problem was the reaper thread in
resource-pool which wakes up every second whether there's anything to reap
or not. Fortunately this was easy to spot as the reaper thread labels
itself as such. I've replaced this with an event-driven reaper that only
wakes up on demand and cut the baseline from 50% down to just under 10%. There are two more threads I can now see waking up periodically in
threadscope, with respective periods approximately 10 sec and 30 sec. I
have a hunch what the 30-sec one is but no idea about the 10-sec one. If I
can kill off the 10 sec one then that should take CPU usage down to <5%
which would be just fine. My question: is there any reliable way to determine where these threads
came from, in terms of the location of the forkIO call that made them? I've
just been looking for likely candidates and labelling threads and this is
rather tedious. Cheers, David _______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe --
Gregory Collins
participants (2)
-
David Turner -
Gregory Collins