#12038: Shutdown interacts badly with requestSync() -------------------------------------+------------------------------------- Reporter: simonmar | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.2.1 Component: Runtime System | Version: 7.10.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by Ben Gamari ): In [changeset:"2f319609ef0ebffc46aad7f4ad14d5c26750e3ba/ghc" 2f31960/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="2f319609ef0ebffc46aad7f4ad14d5c26750e3ba" Fix ASSERT failure and re-enable setnumcapabilities001 The assertion failure was fairly benign, I think, but this fixes it. I've been running the test repeatedly for the last 30 mins and it hasn't triggered. There are other problems exposed by this test (see #12038), but I've worked around those in the test itself for now. I also copied the relevant bits of the parallel library here so that we don't need parallel for the test to run. (cherry picked from commit cfc5df43a7789832a2789e517d8270650cc31b7f) }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12038#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12038: Shutdown interacts badly with requestSync() -------------------------------------+------------------------------------- Reporter: simonmar | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.2.1 Component: Runtime System | Version: 7.10.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonmar): Oh dear. Which ways? What errors? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12038#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12038: Shutdown interacts badly with requestSync() -------------------------------------+------------------------------------- Reporter: simonmar | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.4.1 Component: Runtime System | Version: 7.10.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): I scattered a few `HasCallStacks` about and let it run again and saw the following failure, {{{ setnumcapabilities001: sendWakeup CallStack (from HasCallStack): sendWakeup, called at libraries/base/GHC/Event/TimerManager.hs:205:19 in base:GHC.Event.TimerManager wakeManager, called at libraries/base/GHC/Event/TimerManager.hs:223:7 in base:GHC.Event.TimerManager registerTimeout, called at libraries/base/GHC/Event/Thread.hs:59:10 in base:GHC.Event.Thread: invalid argument (Bad file descriptor) }}} The last frame of the callstack corresponds to the `registerTimeout` in `threadDelay`. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12038#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12038: Shutdown interacts badly with requestSync() -------------------------------------+------------------------------------- Reporter: simonmar | Owner: Type: bug | Status: patch Priority: normal | Milestone: 8.4.1 Component: Runtime System | Version: 7.10.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D2926 Wiki Page: | -------------------------------------+------------------------------------- Comment (by Ben Gamari ): In [changeset:"6de7613604216f65fae92d8066a078bf9cd3c088/ghc" 6de76136/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="6de7613604216f65fae92d8066a078bf9cd3c088" event manager: Don't worry if attempt to wake dead manager fails This fixes #12038, where the TimerManager would attempt to wake up a manager that was already dead, resulting in setnumcapabilities001 occassionally failing during shutdown with unexpected output on stderr. I'm frankly still not entirely confident in this solution but perhaps it will help to get a few more eyes on this. My hypothesis is that the TimerManager is racing: thread TimerManager worker ------- -------------------- requests that thread manager shuts down begins to clean up, closing eventfd calls wakeManager, which tries to write to closed eventfd To prevent this `wakeManager` will need to synchronize with the TimerManger worker to ensure that the worker doesn't clean up the `Control` while another thread is trying to send a wakeup. However, this would add a bit of overhead on every timer interaction, which feels rather costly for what is really a problem only at shutdown. Moreover, it seems that the event manager (e.g. `GHC.Event.Manager`) is also afflicted by a similar race. This patch instead simply tries to catch the write failure after it has happened and silence it in the case that the fd has vanished. It feels rather hacky but it seems to work. Test Plan: Run `setnumcapabilities001` repeatedly Reviewers: austin, hvr, simonmar Reviewed By: simonmar Subscribers: thomie Differential Revision: https://phabricator.haskell.org/D2926 GHC Trac Issues: #12038 }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12038#comment:13 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12038: Shutdown interacts badly with requestSync() -------------------------------------+------------------------------------- Reporter: simonmar | Owner: Type: bug | Status: closed Priority: normal | Milestone: 8.2.1 Component: Runtime System | Version: 7.10.3 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D2926 Wiki Page: | -------------------------------------+------------------------------------- Comment (by Ben Gamari ): In [changeset:"d5cd505bc484edee3dbd5d41fb7a27c2e18d528d/ghc" d5cd505b/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="d5cd505bc484edee3dbd5d41fb7a27c2e18d528d" event manager: Don't worry if attempt to wake dead manager fails This fixes #12038, where the TimerManager would attempt to wake up a manager that was already dead, resulting in setnumcapabilities001 occassionally failing during shutdown with unexpected output on stderr. I'm frankly still not entirely confident in this solution but perhaps it will help to get a few more eyes on this. My hypothesis is that the TimerManager is racing: thread TimerManager worker ------- -------------------- requests that thread manager shuts down begins to clean up, closing eventfd calls wakeManager, which tries to write to closed eventfd To prevent this `wakeManager` will need to synchronize with the TimerManger worker to ensure that the worker doesn't clean up the `Control` while another thread is trying to send a wakeup. However, this would add a bit of overhead on every timer interaction, which feels rather costly for what is really a problem only at shutdown. Moreover, it seems that the event manager (e.g. `GHC.Event.Manager`) is also afflicted by a similar race. This patch instead simply tries to catch the write failure after it has happened and silence it in the case that the fd has vanished. It feels rather hacky but it seems to work. Test Plan: Run `setnumcapabilities001` repeatedly Reviewers: hvr, austin, simonmar Subscribers: thomie Differential Revision: https://phabricator.haskell.org/D2957 GHC Trac Issues: #12038 }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12038#comment:15 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12038: Shutdown interacts badly with requestSync() -------------------------------------+------------------------------------- Reporter: simonmar | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.4.1 Component: Runtime System | Version: 7.10.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #5553 | Differential Rev(s): Phab:D2926 Wiki Page: | -------------------------------------+------------------------------------- Changes (by dfeuer): * related: => #5553 Comment: The fix for this ''appears'' to have fixed #5553 as well. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12038#comment:18 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12038: Shutdown interacts badly with requestSync() -------------------------------------+------------------------------------- Reporter: simonmar | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 7.10.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #5553 | Differential Rev(s): Phab:D2926 Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * milestone: 8.6.1 => Comment: Demilestoning due to lack of progress. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12038#comment:21 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler