Duncan Coutts pushed to branch wip/io-manager-deadlock-detection at Glasgow Haskell Compiler / GHC Commits: ba7a6066 by Duncan Coutts at 2026-09-28T09:52:17+01:00 Add tests for deadlock detection, issue #26408 We have three tests: T26408a: make sure that we can detect a /partial deadlock/ that is not also a /system deadlock/ T26408b: make sure that we can detect a /system deadlock/ promptly without relying on idle GC T26408c: make sure that we do not falsely detect a deadlock when there is nothing to do except wait on signals - - - - - 80e2ceee by Duncan Coutts at 2026-09-28T09:52:17+01:00 Update the user guide with the revised idle GC behaviour It's now not just for the threaded RTS, but general. Add a new section in the RTS chapter: Idle GC and deadlock detection. Provide general documentation on both topics. In particular, document the behaviour of deadlock detection and how it depends on idle GC. Move the existing entries for -I and -Iw to this section. Add a changelog entry. - - - - - 11 changed files: - + changelog.d/idle-gc-and-deadlock-detection - docs/users_guide/profiling.rst - docs/users_guide/runtime_control.rst - testsuite/tests/rts/Makefile - + testsuite/tests/rts/T26408a.hs - + testsuite/tests/rts/T26408a.stderr - + testsuite/tests/rts/T26408b.hs - + testsuite/tests/rts/T26408b.stderr - + testsuite/tests/rts/T26408c.hs - + testsuite/tests/rts/T26408c.stdout - testsuite/tests/rts/all.T Changes: ===================================== changelog.d/idle-gc-and-deadlock-detection ===================================== @@ -0,0 +1,22 @@ +section: rts +issues: #27040 #26408 +mrs: !14820 +synopsis: Idle GC and deadlock detection in the non-threaded RTS now behave the same as in the threaded RTS +description: { +Idle GC now works in the non-threaded RTS and is enabled by default. + +Deadlock detection in the non-threaded RTS now follows the same design and +behaviour as in the threaded RTS. In particular, it can detect partial +deadlocks where some but not all threads are deadlocked. The detection is no +longer prompt however, as the detection runs as part of idle GC, which (by +default) is performed 0.3 seconds after all capabilities become idle. + +The user guide now documents that deadlock detection relies on idle GC, +and thus turning off idle GC disables deadlock detection. This is true +of the threaded RTS in previous GHC releases but it was not previously +documented behaviour. + +Internally, the idle GC and deadlock detection now follow the same design and +use 99% the same code between the non-threaded and threaded RTS. Having a +single design improves the comprehension and maintainability of the RTS code. +} ===================================== docs/users_guide/profiling.rst ===================================== @@ -691,11 +691,13 @@ enclosed between ``+RTS ... -RTS`` as usual): order to increase the resolution of the time profiler. Using a value of zero disables the RTS clock completely, and has the - effect of disabling timers that depend on it: the context switch - timer and the heap profiling timer. Context switches will still - happen, but deterministically and at a rate much faster than normal. - Disabling the interval timer is useful for debugging, because it - eliminates a source of non-determinism at runtime. + effect of disabling the timers and features that depend on it: the context + switch timer, the heap profiling timer, the eventlog flush timer, and the + idle GC timer. Context switches will still happen, but deterministically + and at a rate much faster than normal. Disabling the interval timer is + useful for debugging, because it eliminates a source of non-determinism at + runtime. Note however that disabling the idle GC timer not only disables + idle GC but also deadlock detection (see rts-flag:`-I`). .. rts-flag:: -xc ===================================== docs/users_guide/runtime_control.rst ===================================== @@ -731,61 +731,6 @@ performance. the default small ``-A`` value is suboptimal, as it can be in programs that create large amounts of long-lived data. -.. rts-flag:: -I ⟨seconds⟩ - - :default: 0.3 seconds in the threaded runtime, 0 in the non-threaded runtime - - .. index:: - single: idle GC - - Set the amount of idle time which must pass before a idle GC is - performed. Setting ``-I0`` disables the idle GC. - - In the threaded and SMP versions of the RTS (see :ghc-flag:`-threaded`, - :ref:`options-linker`), a major GC is automatically performed if the - runtime has been idle (no Haskell computation has been running) for a - period of time. - - For an interactive application, it is probably a good idea to use - the idle GC, because this will allow finalizers to run and - deadlocked threads to be detected in the idle time when no Haskell - computation is happening. Also, it will mean that a GC is less - likely to happen when the application is busy, and so responsiveness - may be improved. However, if the amount of live data in the heap is - particularly large, then the idle GC can cause a significant delay, - and too small an interval could adversely affect interactive - responsiveness. - - The idle period timer only resets after some activity - by a Haskell thread. If your program is doing literally nothing then - after the first idle collection is triggered then no more future collections - will be scheduled until more work is performed. - - This is an experimental feature, please let us know if it causes - problems and/or could benefit from further tuning. - -.. rts-flag:: -Iw ⟨seconds⟩ - - :default: 0 seconds - - .. index:: - single: idle GC - - Set the minimum wait time between runs of the idle GC. - - By default, if idle GC is enabled in the threaded runtime, a major - GC will be performed every time the process goes idle for a - sufficiently long duration (see :rts-flag:`-I ⟨seconds⟩`). For - large server processes accepting regular but infrequent requests - (e.g., once per second), an expensive, major GC may run after - every request. As an alternative to shutting off idle GC entirely - (with ``-I0``), a minimum wait time between idle GCs can be - specified with this flag. For example, ``-Iw60`` will ensure that - an idle GC runs at most once per minute. - - This is an experimental feature, please let us know if it causes - problems and/or could benefit from further tuning. - .. rts-flag:: -ki ⟨size⟩ :default: 1k @@ -1011,6 +956,116 @@ performance. ``GHC.Stats.getRTSStats`` from Haskell. +Idle GC and deadlock detection +------------------------------ + +The RTS performs idle GC and deadlock detection. These features are more +closely related and dependent than one might expect. In particular, deadlock +detection relies on idle GC. + +The primary purpose of idle GC is to reclaim memory and return it to the system +when the application has been idle for a short time. It also gives an +opportunity for finalisers to run more promptly. See :rts-flag:`-I ⟨seconds⟩`. + +Deadlock detection is a debugging feature to try to discover when some or all +Haskell threads are deadlocked. For the purpose of this explanation we define: + + * a "partial deadlock" to be a set of threads that are deadlocked; and + * a "system deadlock" is when all threads are deadlocked. + +The deadlock detection is implemented by the GC, and it is capable of detecting +partial deadlocks. If a deadlock is discovered, the RTS will throw an +asynchronous exception to (some or all of) the blocked threads to enable +progress and enable better reporting of the problem. In particular it throws +the async exceptions ``BlockedIndefinitelyOnMVar`` or +``BlockedIndefinitelyOnSTM``, as appropriate. + +The deadlock detection can be run during a major GC, but it is more expensive +than a normal major GC so it is not run during normal major GCs. The deadlock +detection mode of GC is run by the idle GC. This is on the basis that when the +application is idle is when we are most likely to find a deadlock, and also +that the extra time cost is unlikely to be a problem because the application is +already idle. + +There are a few consequence of this: + + * Deadlock detection relies on idle GC and will not run if idle GC is disabled + with ``-I0``. + * Deadlock detection will not run if the application remains busy, and thus + idle GC is not invoked. + * Deadlock detection is not prompt: it will only occur after the idle GC + delay, depending on both :rts-flag:`-I ⟨seconds⟩` and + :rts-flag:`-Iw ⟨seconds⟩`. + +In the non-threaded RTS there is additional behaviour: system deadlocks will be +detected promptly, and this works even if idle GC is otherwise disabled. + +One cannot rely on deadlock detection as part of the normal program behaviour. +It should only be used for debugging. Firstly, deadlock detection can be +disabled or indefinately deferred due to activity. Secondly, the set of threads +that deadlock detection throws exceptions to is not guaranteed to be minimal: +a thread waiting on a deadlocked thread may also have an exception thrown to it +even if it could make progress after the deadlocked thread is terminated. + +Deadlock detection can be invoked manually using ``performDeadlockDetection`` +from ``System.Mem.Experimental``. + + +.. rts-flag:: -I ⟨seconds⟩ + + :default: 0.3 seconds + + .. index:: + single: idle GC + + A major GC is automatically performed if the runtime has been idle (no + Haskell computation has been running) for a period of time. Set the amount + of idle time which must pass before a idle GC is performed. + + Setting ``-I0`` disables the idle GC. This also has the unfortunate side + effect of disabling thread deadlock detection (the implementation of which + uses the idle GC). + + For an interactive application, it is probably a good idea to use + the idle GC, because this will allow finalizers to run and + deadlocked threads to be detected in the idle time when no Haskell + computation is happening. Also, it will mean that a GC is less + likely to happen when the application is busy, and so responsiveness + may be improved. However, if the amount of live data in the heap is + particularly large, then the idle GC can cause a significant delay, + and too small an interval could adversely affect interactive + responsiveness. + + The idle period timer only resets after some activity + by a Haskell thread. If your program is doing literally nothing then + after the first idle collection is triggered then no more future collections + will be scheduled until more work is performed. + + Please let us know if it causes problems and/or could benefit from further + tuning. + +.. rts-flag:: -Iw ⟨seconds⟩ + + :default: 0 seconds + + .. index:: + single: idle GC + + Set the minimum wait time between runs of the idle GC. + + By default (and if idle GC is not disabled) a major + GC will be performed every time the process goes idle for a + sufficiently long duration (see :rts-flag:`-I ⟨seconds⟩`). For + large server processes accepting regular but infrequent requests + (e.g., once per second), an expensive, major GC may run after + every request. As an alternative to shutting off idle GC entirely + (with ``-I0``), a minimum wait time between idle GCs can be + specified with this flag. For example, ``-Iw60`` will ensure that + an idle GC runs at most once per minute. + + This is an experimental feature, please let us know if it causes + problems and/or could benefit from further tuning. + .. _rts-options-statistics: ===================================== testsuite/tests/rts/Makefile ===================================== @@ -157,3 +157,11 @@ HSC2HS_OPTS = --cc="$(TEST_CC)" $(addprefix --cflag=,$(TEST_CC_OPTS)) --ld=$(TES IOManager.hs: IOManager.hsc '$(HSC2HS)' $(HSC2HS_OPTS) $< + +.PHONY: T26408c +T26408c: + "$(TEST_HC)" $(TEST_HC_OPTS) -rtsopts T26408c.hs; \ + ./T26408c +RTS -I0 -T & \ + sleep 1; \ + kill -s USR1 $$!; \ + wait $$! ===================================== testsuite/tests/rts/T26408a.hs ===================================== @@ -0,0 +1,54 @@ +import Control.Concurrent +import Control.Concurrent.STM +import Control.Exception +import Control.Monad + +-- | Test to make sure that we can detect a /partial deadlock/ that is /not/ +-- also a /system deadlock/: that is we can detect a set of deadlocked threads +-- even when there are other unrelated threads that are waiting on I\/O or +-- timeouts. +-- +-- Historically this did not work in the non-threaded RTS which would only do +-- deadlock detection if there was a system deadlock: no runnable threads /and/ +-- no pending I\/O or timers. +-- +-- See Note [Deadlock detection] +-- +main :: IO () +main = do + + -- Set up two threads that are deadlocked on each other + aDone <- newTVarIO False + bDone <- newTVarIO False + let blockingThread theirDone ourDone = + atomically $ do + done <- readTVar theirDone + guard done + writeTVar ourDone True + _ <- forkIO (blockingThread bDone aDone) + _ <- forkIO (blockingThread aDone bDone) + + -- Set up another thread that is blocked on a long timeout. + -- + -- We use a timeout rather than I/O as it's more portable, whereas I/O waits + -- are different between posix and windows I/O managers. + -- + -- One gotcha is that when the timeout completes then the deadlock will be + -- detected again (since the bug is about I/O or timeouts masking deadlock + -- detection). So for a reliable test the timeout used here must be longer + -- than the test framework's own timeout. So we use maxBound, and we adjust + -- the test framework's timeout to be short (see run_timeout_multiplier). + _ <- forkIO (threadDelay maxBound) + + -- Wait on the two deadlocked threads to terminate. This is now a partial + -- deadlock but not a system deadlock (the main thread and the first two + -- threads are deadlocked, but the one waiting on the timeout is ok). + -- + -- We now expect that the threads that are deadlocked are detected as such + -- and an exception is thrown to some or all of them. + -- + -- Note that if this fails, the test itself will effectively deadlock and + -- will rely on the test framework's timeout. + atomically $ do + status <- mapM readTVar [aDone, bDone] + guard (or status) ===================================== testsuite/tests/rts/T26408a.stderr ===================================== @@ -0,0 +1,3 @@ +T26408a: Uncaught exception ghc-internal:GHC.Internal.IO.Exception.BlockedIndefinitelyOnSTM: + +thread blocked indefinitely in an STM transaction ===================================== testsuite/tests/rts/T26408b.hs ===================================== @@ -0,0 +1,41 @@ +import Control.Concurrent +import Control.Concurrent.STM +import Control.Exception +import Control.Monad + +-- | Test to make sure that we can detect a /system deadlock/ promptly which +-- means without relying on idle GC. To check this we have to turn off idle GC, +-- (+RTS -I0) which is what normally triggers deadlock detection. +-- +-- This is only expected to work in the non-threaded RTS. In principle this +-- test could also work with the threaded RTS when using only a single +-- capability, if the I\/O managers can support it. Currently that is not the +-- case. +-- +-- See Note [Deadlock detection] +-- +main :: IO () +main = do + + -- Set up two threads that are deadlocked on each other + aDone <- newTVarIO False + bDone <- newTVarIO False + let blockingThread theirDone ourDone = + atomically $ do + done <- readTVar theirDone + guard done + writeTVar ourDone True + _ <- forkIO (blockingThread bDone aDone) + _ <- forkIO (blockingThread aDone bDone) + + -- Wait on the two deadlocked threads to terminate. This is now a system + -- deadlock, involving all three threads (the main thread and the other two). + -- + -- We now expect that the threads that are deadlocked are detected as such + -- and an exception is thrown to some or all of them. + -- + -- Note that if this fails, the test itself will effectively deadlock and + -- will rely on the test framework's timeout. + atomically $ do + status <- mapM readTVar [aDone, bDone] + guard (or status) ===================================== testsuite/tests/rts/T26408b.stderr ===================================== @@ -0,0 +1,3 @@ +T26408b: Uncaught exception ghc-internal:GHC.Internal.IO.Exception.BlockedIndefinitelyOnSTM: + +thread blocked indefinitely in an STM transaction ===================================== testsuite/tests/rts/T26408c.hs ===================================== @@ -0,0 +1,61 @@ +import Control.Concurrent +import Control.Concurrent.STM +import Control.Exception +import Control.Monad + +import System.Posix.Signals +import GHC.Stats.Experimental + +-- | Test to make sure that we do not falsely detect a deadlock when there is +-- nothing to do except wait on signals. This is to make sure that our prompt +-- system deadlock detection is not too eager. +-- +-- We arrange a partial deadlock that can be broken if a signal arrives and +-- the signal handler can unblock things. +-- +-- This should /not/ result in any deadlock being detected, and no other odd +-- behaviour, like getting caught in a loop and doing too many idle GCs. So we +-- count and report the number of major GCs. +-- +main :: IO () +main = do + + -- Set up two threads that are deadlocked on each other + aDone <- newTVarIO False + bDone <- newTVarIO False + let blockingThread theirDone ourDone = + atomically $ do + done <- readTVar theirDone + guard done + writeTVar ourDone True + + installHandler userDefinedSignal1 + (Catch $ do + atomically $ writeTVar aDone True + >> writeTVar bDone True + putStrLn "caught SIGUSR1") + Nothing + + _ <- forkIO (blockingThread bDone aDone) + _ <- forkIO (blockingThread aDone bDone) + yield + + -- Wait on the deadlocked threads to terminate. We now expect that the threads + -- that are deadlocked are detected as such and an exception is raised. + -- Note that if this fails, the test itself will effectively deadlock and + -- will rely on the test framework's timeout. + putStrLn "waiting on deadlocked threads..." + atomically $ do + status <- mapM readTVar [aDone, bDone] + guard (or status) + + -- We expect exactly 1 major GC: the one triggered for deadlock detection + -- where we find no deadlock. We should then sleep without doing more GC. + -- An easy mistake would be looping: detecting probable deadlock, do idle GC. + -- See notifyIdleGcDeadlock(). + stats <- getRTSStats + putStrLn ("terminated normally, major GCs = " ++ show (major_gcs stats)) + -- This test _could_ be sensitive to changes in default heap sizes etc. If + -- this number ends up as 2 for example, then adjust the RTS opts (in the + -- makefile) so that the heap is big enough that normally no major GC would + -- be needed. ===================================== testsuite/tests/rts/T26408c.stdout ===================================== @@ -0,0 +1,5 @@ +[1 of 2] Compiling Main ( T26408c.hs, T26408c.o ) +[2 of 2] Linking T26408c +waiting on deadlocked threads... +caught SIGUSR1 +terminated normally, major GCs = 1 ===================================== testsuite/tests/rts/all.T ===================================== @@ -682,6 +682,24 @@ test('T22859', omit_ways(llvm_ways + ['nonmoving_thr', 'nonmoving_thr_sanity'])], compile_and_run, ['-with-rtsopts -A8K']) +# Deadlock detection testsL +# A: check we can detect a partial deadlock that is not a system deadlock +test('T26408a', [exit_code(1), run_timeout_multiplier(0.1)], compile_and_run, ['']) +# B: check we can detect system deadlock with idle GC turned off (+RTS -I0) +test('T26408b', [omit_ways(threaded_ways), extra_run_opts('+RTS -I0 -RTS'), + exit_code(1), run_timeout_multiplier(0.1)], + compile_and_run, ['-rtsopts']) +# C: check we do not falsely detect deadlock when blocked only on signals +# Skip for win32, wasm and js since this test requires posix signals +test('T26408c', [when(opsys('mingw32'), skip), + when(arch('wasm32'), skip), js_skip, + run_timeout_multiplier(0.1) + ], makefile_test, ['T26408c']) +# TODO: change omit_ways(threaded_ways) above +# to all_io_managers_except['mio', 'winio'] +# and test +RTS -N1 with threaded ways, but will be broken for now. + + # These tests need access to the internal RTS headers. # TODO: there is probably some cleaner way to do this, and it should probably # be guarded for in-tree tests, since it cannot work against an arbitrary View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/e5743b87c87d2a58b1e3c0c25519524... -- View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/e5743b87c87d2a58b1e3c0c25519524... You're receiving this email because of your account on gitlab.haskell.org. Manage all notifications: https://gitlab.haskell.org/-/profile/notifications | Help: https://gitlab.haskell.org/help