Duncan Coutts pushed to branch wip/io-manager-deadlock-detection at Glasgow Haskell Compiler / GHC Commits: 53eead15 by Wolfgang Jeltsch at 2026-09-24T20:12:03-04:00 Correct the documentation of `explicitUnits` Resolves #27787. - - - - - 35bf6f4b by Simon Jakobi at 2026-09-24T20:12:49-04:00 Remove stale specHeader bullet from Note [Don't make fillers for constraint types] The Note said that Specialise.specHeader also calls mkAbsentFiller. abfc224a27c removed that call: the specialiser no longer drops dead arguments at all, see Note [Do not drop dead args from specialisations] in GHC.Core.Opt.Specialise. Assisted-by: Claude Fable 5.1 - - - - - 1abef1ef by Wolfgang Jeltsch at 2026-09-25T17:02:07-04:00 Improve the `mhu-closure` makefile - - - - - a79a68ea by Wolfgang Jeltsch at 2026-09-25T17:02:07-04:00 Make `ghci-multi` hide all unneeded packages - - - - - d6e72450 by Wolfgang Jeltsch at 2026-09-25T17:02:07-04:00 Improve the definition of the home unit closure property - - - - - c9a2766d by Wolfgang Jeltsch at 2026-09-25T17:02:07-04:00 Re-implement the home unit closure check Compared to the previous implementation, the new one has the following advantages: * It is correct. - It distinguishes between units that have the same unit ID but different ABI hashes. - When `-hide-all-packages` is not used, it considers as home unit dependencies also units that are made implicitly available because they are in the package database. * It seems to be faster in usual settings. - In particular, it does not have a preparation phase in which it merges the dependency information from all home units, so that its running time does not grow linearly with the size of the package database in normal Cabal scenarios. * It is (hopefully) clearer. * It is better documented. - - - - - fa96acf8 by Wolfgang Jeltsch at 2026-09-25T17:02:07-04:00 Remove erroneous GHC flag in `hadrian-multi` The flag was `-Werror=-Wno-error=incomplete-record-selectors`. Its purpose was to permit incomplete record selectors, but because of the illegal combination of `-Werror=` and `-Wno-error` it was ignored, without any negative consequences. - - - - - 9f48a5b9 by Alan Zimmerman at 2026-09-25T17:02:50-04:00 EPA: Add specific layout field to AnnList Up to now AnnList has effectively served to represent lists of things having layout, regardless of whether they do or not. In Haskell, layout is optional in many places, as there is also an alternative syntax using braces and semi-colons, which is free-form. This commit introduces AnnListLayout, a data structure capturing this distinction. It is deliberately set up similarly to EpLayout, so that these two can be merged in time. The transition is done mechanically, where AnnListBraces is not used yet, the original Nothing case maps to AnnListNoLayout, and the Just case to AnnListLayout. Later commits will fine tune these against the actual parser layout. - - - - - 388f9c66 by Duncan Coutts at 2026-09-27T18:43:02+01:00 Move idle GC tracking to its own file, step 1 of 2 Add a new IdleGC.{c,h} module. Move the RecentActivity type, recent_activity variable and {get,set}RecentActivity wrappers. - - - - - fac92964 by Duncan Coutts at 2026-09-27T18:43:02+01:00 Move idle GC functionality to its own file, step 2 of 2 We can now fully encapsulate the recent_activity state, exposing just three hooks, and one query used in the scheduler and timer tick. This is a prelude to making changes to the implementation of tracking when to perform an idle gc. - - - - - 39167176 by Duncan Coutts at 2026-09-27T18:43:02+01:00 Correct comments in idle GC code The comments still referred to stopTimer/startTimer. These were replaced recently by pauseTimer/unpauseTimer, but the comment was not updated. - - - - - 6ceae159 by Duncan Coutts at 2026-09-27T18:43:02+01:00 Make signal handling be a responsibility of the I/O manager(s) Previously it was scattered between I/O managers and the scheduler, and especially the scheduler's deadlock detection. Previously the scheduler would poll for pending signals each iteration of the scheduler loop. The scheduler also had some hairy signal functionality in the deadlock detection: in the non-threaded RTS (only) if there were still no threads running after deadlock detection then it would block waiting for signals. But signals can and (in my opinion) should be thought of as just a funny kind of I/O, and thus should be a responsibility of the I/O manager. So now we have the I/O managers poll for signals when they are polling for I/O completion (and removing the separate poll in the scheduler). And when I/O managers block waiting for I/O then they now also start signal handlers if they get interrupted by a signal. Crucially, if there is no pending I/O or timers, the awaitCompletedTimeoutsOrIO will still block waiting for signals. This patch puts us into an intermediate state: it temporarily breaks deadlock detection in the non-threaded RTS. The waiting on I/O currently happens before deadlock detection. This means we'll now wait forever on signals before doing deadlock detection. We need to move waiting after deadlock detection. We'll do that in a later patch. - - - - - 2d010a57 by Duncan Coutts at 2026-09-27T18:43:02+01:00 Clean up the RTS internal signal handling API Now that the I/O manager is responsible for signals, we can simplify the API we present for signal handling. We now just need startPendingSignalHandlers, which is called from the I/O managers. We can get rid of awaitUserSignals. We also don't need RtsSignals.h to re-export the platform-specific posix/Signals.h or win32/ConsoleHandler.h We can also hide more of the implementation of signals. Less has to be exposed in posix/Signals.h or win32/ConsoleHandler.h. Indeed, posix/Signals.h becomes empty and we remove it. Partly this is because we don't need inline functions (or macros) in the interface. Also remove signal_handlers from RTS ABI exported symbols list. It does not appear to have any users in the core libs, and its really an internal implementation detail. It should not be exposed unless it's really necessary. - - - - - 345837c6 by Duncan Coutts at 2026-09-27T18:43:02+01:00 In the scheduler, move I/O blocking after deadlock detection To make deadlock detection effective in the non-threaded RTS when there are deadlocked threads and other unrelated threads waiting on I/O, we need to arrange to do deadlock detection before we block in scheduler to wait on I/O. The solution is to: 1. adjust scheduleFindWork, which runs before deadlock detection, to only poll for I/O and not block; and 2. add a step after deadlock detection to wait on I/O if there are still no threads to run (and there's any I/O or timeouts outstanding) The scheduleCheckBlockedThreads is now so simple that it made more sense to inline it into scheduleFindWork. - - - - - 2456bc80 by Duncan Coutts at 2026-09-27T18:43:02+01:00 Remove bogus anyPendingTimeoutsOrIO guard from scheduleDetectDeadlock The deadlock detection was only invoked if both of these conditions hold: 1. the run queue is empty 2. there is no pending I/O or timeouts The second condition is unnecessary. The deadlock detection mechanism can find deadlocks even if there are other threads waiting on I/O or timers. Having this extra condition means that we fail to detect blocked threads if there are any threads waiting on I/O or timers. Part of fixing issue #26408 - - - - - 84a9ada4 by Duncan Coutts at 2026-09-27T18:43:02+01:00 Don't consider pending I/O for early context switch optimisation Context switches are normally initiated by the timer signal. If however the user specifies "context switch as often as possible", with +RTS -C0 then the scheduler arranges for an early context switch (when it's just about to run a Haskell thread). Context switching very often is expensive, so as an optimisation there cases where we do not arrange an early context switch: 1. if there's no other threads to run 2. if there is no pending I/O or timers This patch eliminates case 2, leaving only case 1. The rationale is as follows. The use of this was inconsistent across platforms and threaded/non-threaded RTS ways. It only worked on the non-threaded RTS and on Windows only worked for the win32-legacy I/O manager. On all other combinations anyPendingTimeoutsOrIO would always return false. The fact that nobody noticed and complained about this inconsistency suggests that the feature is not relied upon. If however it turns out that applications do rely on this, then the proper thing to do is not to restore this check, but to add a new I/O manager hint function that returns if there is any pending events that are likely to happen *soon*: for example timeouts expiring within one timeslice, or I/O waits on things likely to complete soon like disk I/O, but not for example socket/pipe I/O. The motivation to avoid this use of anyPendingTimeoutsOrIO is to allow us to eliminate anyPendingTimeoutsOrIO entirely. All other uses of this are just guards on {await,poll}CompletedTimeoutsOrIO and the guards can safely be folded into those functions. This will better cope with some I/O managers having no proper implementation of anyPendingTimeoutsOrIO. Ultimately this will let us simplify the scheduler which currently has to have special #ifdef mingw32_HOST_OS cases to cope with the lack of a working anyPendingTimeoutsOrIO for some Windows I/O managers - - - - - 0bfa75e3 by Duncan Coutts at 2026-09-27T18:43:02+01:00 Remove anyPendingTimeoutsOrIO guarding {poll,await}CompletedTimeoutsOrIO Previously the API of the I/O manager used a two step process: check anyPendingTimeoutsOrIO and then call {poll,await}CompletedTimeoutsOrIO. This was primarily there as a performance thing, to cheaply check if we need to do anything. And then because anyPendingTimeoutsOrIO existed, it was used for other things too. We have now eliminated the other uses, and are just left with the performance pattern. But this was problematic because not all I/O managers correctly implement anyPendingTimeoutsOrIO (specifically the win32 ones), and now that we also make I/O managers responsible for signals then we need to poll/await even if there is no pending I/O or timeouts. If there is no pending I/O or timeouts then await needs to degenerate to just waiting forever for any signals. - - - - - 1af20d4f by Duncan Coutts at 2026-09-27T18:43:02+01:00 Remove anyPendingTimeoutsOrIO, it is no longer used And this avoids the problems arising from the win32 I/O managers having had a bogus implementation. - - - - - b89c0adc by Duncan Coutts at 2026-09-27T18:43:02+01:00 Remove second scheduler call to awaitCompletedTimeoutsOrIO Previously awaitCompletedTimeoutsOrIO was called both before and after deadlock detection in the scheduler. The reason for that was that the win32 I/O managers had a bogus implementation of anyPendingTimeoutsOrIO and this was used to guard the call of awaitCompletedTimeoutsOrIO prior to deadlock detection. This meant the first call site was never actually called when using the win32 I/O managers. This was the reason for the second call: the first one was never used. What a mess. So now we have a simple design in the scheduler: 1. poll for completed I/O, timers or signals 2. if no runnable threads: do deadlock detection 3. if still no runnable threads: block waiting for I/O, timers or signals. - - - - - e9d80b12 by Duncan Coutts at 2026-09-27T18:43:03+01:00 Lift emptyRunQueue guard out of scheduleDetectDeadlock this improved the clarity of the logic when reading the scheduler code. And improve the comment about scheduleDoGC that was rather unclear. - - - - - f446219c by Duncan Coutts at 2026-09-27T18:43:03+01:00 Make interruptIOManager work for WinIO (non-threaded) Fixes #27403 The idle GC scheme now fundamentally relies on interruptIOManager working, so it's no longer acceptable for it to be missing for WinIO. Not having it would cause lock ups. - - - - - 901ef7bb by Duncan Coutts at 2026-09-27T18:43:03+01:00 Make non-threaded deadlock detection also rely on idle GC Only do deadlock detection GC when idle GC kicks in. This also relies on using wakeUpRts, so now do this unconditionally. Previously wakeUpRts was for the threaded rts only. - - - - - 6c43daa1 by Duncan Coutts at 2026-09-27T18:43:03+01:00 Enable idle GC by default on non-threaded RTS The behaviour is now uniform between the threaded and non-threaded RTS ways. The deadlock detection now relies on idle GC for both threaded and non-threaded ways. Previously deadlock detection did not rely on idle GC for the non-threaded way. Also tweak test T7275 to account for idle GC. This test's output is sensitive to the number of major GCs run. Since this commit enables idle GC for the non-threaded RTS, for this test that increases the number of major GCs, since the test program is frequently idle for more than 300ms. - - - - - 32e5b061 by Duncan Coutts at 2026-09-27T18:43:03+01:00 Fix state of idle GC control vars with +RTS -V0 Currently when the user uses +RTS -I0, then doIdleGC is set to false. But if the master tick interval -V is set to 0 then the idleGCDelayTime was being set to 0 but doIdleGC was not being set to false, which is inconsistent, and almost certainly buggy. - - - - - e2cdb1a0 by Duncan Coutts at 2026-09-27T18:43:03+01:00 Add a long Note [Deadlock detection] It describes the historical and modern designs and their trade-offs. The point is we've now unified the code for deadlock detection between the threaded and non-threaded ways, by changing the non-threaded to follow the same design as the threaded. - - - - - 9f15516c by Duncan Coutts at 2026-09-27T18:43:03+01:00 Refactor the utils providing the I/O timeout for different system APIs We have to provide a timeout in three different time formats (int milliseconds, struct timeval *, and struct timespec *). These are for platform APIs including poll, select, ppoll, pselect, epoll_wait, epoll_wait2 and in future kevent and io_uring. Previously we had one util for each format that both worked out what the timeout should be and converted it into the particular format. This duplicates some logic and it also makes it awkward to modify the timeout value (since there's no common format). Modifying the timeout value is exactly what we want to do, in a subsequent commit, where we want to adjust the timeout to take account of idle GC. - - - - - bbf4dbb7 by Duncan Coutts at 2026-09-27T18:43:03+01:00 Update the old select I/O manager to use the new timeout utils This will make it much easier to adjust the timeout to take account of idle GC. - - - - - 97a922ba by Duncan Coutts at 2026-09-27T18:43:03+01:00 Add opportunistic deadlock detection without idle GC Our general design is now that comprehensive deadlock detection relies on idle GC. This allows us to detect partial deadlocks that are not system deadlocks. But it's not prompt. It's a nice to have feature to be able to promptly detect the cases that are system deadlocks. There's no hope to do this in the threaded RTS with -N > 1, but it can be done for the non-threaded RTS and _could_ be done for the threaded RTS with -N1. We can integrate this feature into the general design without too many contortions. We just need the I/O managers to check before sleeping if they would be about to sleep indefinately with no I/O and no registered signal handlers. If so, they can notify the idle GC that we're deadlocked and we schedule and idle GC. - - - - - aada868f by Duncan Coutts at 2026-09-27T18:43:03+01:00 Add idle GC mode for !HAVE_PREEMPTION We have to do idle GC (and thus deadlock detection) differently when we do not have the ticker and the ability to interrupt the I/O manager when it's blocking. - - - - - e81a7fd7 by Duncan Coutts at 2026-09-27T18:43:03+01:00 Add and use USED_IF_PREEMPTION to avoid warnings - - - - - aade1797 by Duncan Coutts at 2026-09-27T18:43:03+01:00 Add performDeadlockDetection and export via ghc-experimental When documenting the idle GC and deadlock detection behaviour in the user guide, it becomes obiovus that in some cases one would want manual control over deadlock detection. In particular, for applications that disable idle GC, for debugging or robustness, they may want to invoke deadlock detecting GC manually. Bind it in ghc-internal GHC.Internal.System.Mem and re-export via ghc-experimental in System.Mem.Experimental. It is clearly implementation specific behaviour, especially since in GHC the deadlock detection is tied to doing a major GC. - - - - - d0af4f3d by Duncan Coutts at 2026-09-27T18:43:03+01:00 Add tests for deadlock detection, issue #26408 We have three tests: T26408a: make sure that we can detect a /partial deadlock/ that is not also a /system deadlock/ T26408b: make sure that we can detect a /system deadlock/ promptly without relying on idle GC T26408c: make sure that we do not falsely detect a deadlock when there is nothing to do except wait on signals - - - - - e5743b87 by Duncan Coutts at 2026-09-27T18:43:03+01:00 Update the user guide with the revised idle GC behaviour It's now not just for the threaded RTS, but general. Add a new section in the RTS chapter: Idle GC and deadlock detection. Provide general documentation on both topics. In particular, document the behaviour of deadlock detection and how it depends on idle GC. Move the existing entries for -I and -Iw to this section. Add a changelog entry. - - - - - 78 changed files: - .gitlab-ci.yml - + changelog.d/idle-gc-and-deadlock-detection - compiler/GHC/Core/Opt/WorkWrap/Utils.hs - compiler/GHC/Driver/Downsweep.hs - compiler/GHC/Driver/Errors/Ppr.hs - compiler/GHC/Driver/Errors/Types.hs - compiler/GHC/Parser.y - compiler/GHC/Parser/Annotation.hs - compiler/GHC/Parser/PostProcess.hs - compiler/GHC/Unit/Env.hs - compiler/GHC/Unit/State.hs - docs/users_guide/profiling.rst - docs/users_guide/runtime_control.rst - hadrian/ghci-multi-cabal.in - libraries/ghc-experimental/src/System/Mem/Experimental.hs - libraries/ghc-internal/src/GHC/Internal/System/Mem.hs - rts/Capability.c - rts/IOManager.c - rts/IOManager.h - + rts/IdleGC.c - + rts/IdleGC.h - rts/Linker.c - rts/RtsFlags.c - rts/RtsSignals.h - rts/RtsStartup.c - rts/RtsSymbols.c - rts/Schedule.c - rts/Schedule.h - rts/Timer.c - rts/include/rts/OSThreads.h - rts/include/rts/storage/GC.h - rts/posix/Poll.c - rts/posix/Poll.h - rts/posix/Select.c - rts/posix/SelectBis.c - rts/posix/SelectBis.h - rts/posix/Signals.c - − rts/posix/Signals.h - rts/posix/Timeout.c - rts/posix/Timeout.h - rts/rts.cabal - rts/win32/AsyncMIO.c - rts/win32/AsyncMIO.h - rts/win32/AsyncWinIO.c - rts/win32/AsyncWinIO.h - rts/win32/AwaitEvent.c - rts/win32/ConsoleHandler.c - rts/win32/ConsoleHandler.h - testsuite/tests/driver/multipleHomeUnits/mhu-closure/Makefile - testsuite/tests/driver/multipleHomeUnits/mhu-closure/mhu-closure.stderr - testsuite/tests/driver/multipleHomeUnits/mhu-closure/mhu-closure.stdout - testsuite/tests/ghc-api/exactprint/T22919.stderr - testsuite/tests/ghc-api/exactprint/ZeroWidthSemi.stderr - testsuite/tests/interface-stability/ghc-experimental-exports.stdout - testsuite/tests/interface-stability/ghc-experimental-exports.stdout-mingw32 - testsuite/tests/module/mod185.stderr - testsuite/tests/parser/should_compile/DumpParsedAst.stderr - testsuite/tests/parser/should_compile/DumpParsedAstComments.stderr - testsuite/tests/parser/should_compile/DumpRenamedAst.stderr - testsuite/tests/parser/should_compile/DumpSemis.stderr - testsuite/tests/parser/should_compile/KindSigs.stderr - testsuite/tests/parser/should_compile/T15279.stderr - testsuite/tests/parser/should_compile/T20718.stderr - testsuite/tests/parser/should_compile/T20846.stderr - testsuite/tests/printer/Test20297.stdout - testsuite/tests/printer/Test24533.stdout - testsuite/tests/profiling/should_run/Makefile - testsuite/tests/rts/Makefile - + testsuite/tests/rts/T26408a.hs - + testsuite/tests/rts/T26408a.stderr - + testsuite/tests/rts/T26408b.hs - + testsuite/tests/rts/T26408b.stderr - + testsuite/tests/rts/T26408c.hs - + testsuite/tests/rts/T26408c.stdout - testsuite/tests/rts/all.T - utils/check-exact/ExactPrint.hs - utils/check-exact/Main.hs - utils/check-exact/Transform.hs The diff was not included because it is too large. View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/f3468063892902795d7fd9fc26b2b72... -- View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/f3468063892902795d7fd9fc26b2b72... You're receiving this email because of your account on gitlab.haskell.org. Manage all notifications: https://gitlab.haskell.org/-/profile/notifications | Help: https://gitlab.haskell.org/help