[GHC] #14310: Assertion triggered by STM invariant.
#14310: Assertion triggered by STM invariant. --------------------------------------+---------------------------------- Reporter: mbw | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.2.1 Keywords: | Operating System: Linux Architecture: x86_64 (amd64) | Type of failure: Runtime crash Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: --------------------------------------+---------------------------------- The program attached depends on the async and stm packages. It will push some numbers through a pipeline, while keeping track of some state. If invoked without arguments, the program will output the state and result of each iteration. If invoked via "./repro check", an STM invariant will be installed when the state is initialized. This invariant will cause the program to a) hangup, presumably due to a livelock, if the program has been compiled without "-debug", b) crash with the following error: "internal error: ASSERTION FAILED: file rts/RaiseAsync.c, line 1001", if the "-debug" flag has been used for compilation. The latter case also happens with GHC 8.0.2: "internal error: ASSERTION FAILED: file rts/RaiseAsync.c, line 997". Furthermore, this problem only occurs if the program is compiled using the additional flags "-rtsopts -threaded -with-rtsopts=-N". -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14310 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14310: Assertion triggered by STM invariant. ----------------------------------+-------------------------------------- Reporter: mbw | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.2.1 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 (amd64) Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ----------------------------------+-------------------------------------- Changes (by mbw): * Attachment "Main.hs" added. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14310 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14310: Assertion triggered by STM invariant. ----------------------------------+-------------------------------------- Reporter: mbw | Owner: (none) Type: bug | Status: new Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.2.1 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 (amd64) Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ----------------------------------+-------------------------------------- Changes (by bgamari): * priority: normal => high * milestone: => 8.4.1 Old description:
The program attached depends on the async and stm packages. It will push some numbers through a pipeline, while keeping track of some state.
If invoked without arguments, the program will output the state and result of each iteration. If invoked via "./repro check", an STM invariant will be installed when the state is initialized. This invariant will cause the program to
a) hangup, presumably due to a livelock, if the program has been compiled without "-debug",
b) crash with the following error: "internal error: ASSERTION FAILED: file rts/RaiseAsync.c, line 1001", if the "-debug" flag has been used for compilation.
The latter case also happens with GHC 8.0.2: "internal error: ASSERTION FAILED: file rts/RaiseAsync.c, line 997".
Furthermore, this problem only occurs if the program is compiled using the additional flags "-rtsopts -threaded -with-rtsopts=-N".
New description: The program attached depends on the async and stm packages. It will push some numbers through a pipeline, while keeping track of some state. If invoked without arguments, the program will output the state and result of each iteration. If invoked via `./repro check`, an STM invariant will be installed when the state is initialized. This invariant will cause the program to 1. hangup, presumably due to a livelock, if the program has been compiled without `-debug`, or 2. crash with the following error: {{{ internal error: ASSERTION FAILED: file rts/RaiseAsync.c, line 1001", }}} if the "-debug" flag has been used for compilation. The latter case also happens with GHC 8.0.2: {{{ internal error: ASSERTION FAILED: file rts/RaiseAsync.c, line 997 }}} Furthermore, this problem only occurs if the program is compiled using the additional flags `-rtsopts -threaded -with-rtsopts=-N`. -- -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14310#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14310: Assertion triggered by STM invariant. ----------------------------------+-------------------------------------- Reporter: mbw | Owner: (none) Type: bug | Status: new Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.2.1 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 (amd64) Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ----------------------------------+-------------------------------------- Comment (by bgamari): Well, the good news is that this is reproducible with `+RTS -Dm` (STM debugging enabled). The bad news is that it produces 170MB of output before crashing. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14310#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14310: Assertion triggered by STM invariant. ----------------------------------+-------------------------------------- Reporter: mbw | Owner: (none) Type: bug | Status: new Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.2.1 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 (amd64) Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ----------------------------------+-------------------------------------- Changes (by bgamari): * Attachment "Main2.hs" added. Variant that doesn't rely on async -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14310 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14310: Assertion triggered by STM invariant. ----------------------------------+-------------------------------------- Reporter: mbw | Owner: (none) Type: bug | Status: new Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.2.1 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 (amd64) Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ----------------------------------+-------------------------------------- Comment (by bgamari): It turns out that `async` isn't strictly necessary to reproduce this, thankfully. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14310#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14310: Assertion triggered by STM invariant. ----------------------------------+-------------------------------------- Reporter: mbw | Owner: (none) Type: bug | Status: new Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.2.1 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 (amd64) Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ----------------------------------+-------------------------------------- Comment (by bgamari): For the record, the assertion in question was originally introduced in 96757f6a4ec72dc609468d3da442db38a73df23e, essentially the beginning of time. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14310#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14310: Assertion triggered by STM invariant. ----------------------------------+-------------------------------------- Reporter: mbw | Owner: (none) Type: bug | Status: new Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.2.1 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 (amd64) Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ----------------------------------+-------------------------------------- Comment (by bgamari): I mention this because I'm a bit skeptical of the assertion itself; it looks like it appeared a fair bit before the invariant work. If you look at `stg_atomically_frame` it seems pretty clear that we will end up with nested transactions in the normal course of checking invariants. Obviously there is still something wrong since the problem deadlocks without the debug RTS, but I'm wondering whether the assert is really pointing in the right direction. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14310#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14310: Assertion triggered by STM invariant. ----------------------------------+-------------------------------------- Reporter: mbw | Owner: (none) Type: bug | Status: new Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.2.1 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 (amd64) Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ----------------------------------+-------------------------------------- Comment (by bgamari): I'll also say that the comment right below the assertion doesn't instill confidence, {{{ // The ATOMICALLY_FRAME expects to be returned a // result from the transaction, which it stores in the // stack frame. Hence we arrange to return a dummy // result, so that the GC doesn't get upset (#3578). // Perhaps a better way would be to have a different // ATOMICALLY_FRAME instance for condemned // transactions, but I don't fully understand the // interaction with STM invariants. }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14310#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14310: Assertion triggered by STM invariant.
----------------------------------+--------------------------------------
Reporter: mbw | Owner: (none)
Type: bug | Status: new
Priority: high | Milestone: 8.4.1
Component: Compiler | Version: 8.2.1
Resolution: | Keywords:
Operating System: Linux | Architecture: x86_64 (amd64)
Type of failure: Runtime crash | Test Case:
Blocked By: | Blocking:
Related Tickets: | Differential Rev(s):
Wiki Page: |
----------------------------------+--------------------------------------
Comment (by bgamari):
If I compile without the debug RTS and run the test program in the same
way, I find that the program isn't actually dead-locked. Rather it is
live-locked with two threads apparently spinning against one another:
{{{
pthread_cond_wait@@GLIBC_2.3.2 () at
../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 in ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S
>>> info threads
Id Target Id Frame
* 1 Thread 0x7f88ade01740 (LWP 4341) "Main"
pthread_cond_wait@@GLIBC_2.3.2 () at
../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
2 Thread 0x7f88abec5700 (LWP 4342) "ghc_worker" 0x00007f88ac9650f3 in
epoll_wait () at ../sysdeps/unix/syscall-template.S:84
3 Thread 0x7f88ab6c4700 (LWP 4343) "ghc_ticker" 0x00007f88ad08b20d in
read () at ../sysdeps/unix/syscall-template.S:84
4 Thread 0x7f88aaec3700 (LWP 4344) "ghc_worker" 0x00007f88ac9650f3 in
epoll_wait () at ../sysdeps/unix/syscall-template.S:84
5 Thread 0x7f88aa6c2700 (LWP 4345) "ghc_worker" 0x00007f88ac95b6ad in
poll () at ../sysdeps/unix/syscall-template.S:84
6 Thread 0x7f88a9ec1700 (LWP 4346) "ghc_worker" 0x00000000004a3c21 in
stg_atomically_frame_info ()
7 Thread 0x7f88a96c0700 (LWP 4347) "ghc_worker" stmAbortTransaction
(cap=0x4e01b0, trec=0x42000d2238) at rts/STM.c:1020
>>> thread 6
[Switching to thread 6 (Thread 0x7f88a9ec1700 (LWP 4346))]
#0 0x00000000004a3c21 in stg_atomically_frame_info ()
>>> bt
#0 0x00000000004a3c21 in stg_atomically_frame_info ()
#1 0x0000000000000000 in ?? ()
>>> x/8a $rbp
0x42002d7338: 0x4a3bb8 0x42002fd510
0x42002d7348: 0x42001d56d8 0x4daed9
0x42002d7358: 0x413bf0
0x42002fd538
0x42002d7368: 0x49feb0 0x0
>>> thread 7
[Switching to thread 7 (Thread 0x7f88a96c0700 (LWP 4347))]
#0 stmAbortTransaction (cap=0x4e01b0, trec=0x42000d2238) at
rts/STM.c:1020
1020 rts/STM.c: No such file or directory.
>>> bt
#0 stmAbortTransaction (cap=0x4e01b0, trec=0x42000d2238) at
rts/STM.c:1020
#1 0x00000000004a3c14 in stg_atomically_frame_info ()
#2 0x0000000000000000 in ?? ()
>>> up
#1 0x00000000004a3c14 in stg_atomically_frame_info ()
>>> x/8a $rbp
0x42002d54a0: 0x4a3bb8 0x420029d690
0x42002d54b0: 0x42000d2218 0x4daed9
0x42002d54c0: 0x413bf0
0x420029d6b8
0x42002d54d0: 0x49feb0 0x0
>>>
}}}
--
Ticket URL:
GHC
#14310: Assertion triggered by STM invariant. ----------------------------------+-------------------------------------- Reporter: mbw | Owner: (none) Type: bug | Status: new Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.2.1 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 (amd64) Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ----------------------------------+-------------------------------------- Comment (by bgamari): For the record, STM invariant support was added in 9cef40bd4dd2536c7a370a1a9b78461c152805cc, quite a while after the commit which added the assert. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14310#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14310: Assertion triggered by STM invariant. ----------------------------------+-------------------------------------- Reporter: mbw | Owner: (none) Type: bug | Status: new Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.2.1 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 (amd64) Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ----------------------------------+-------------------------------------- Comment (by bgamari): Here is a representative snipped from the `+RTS -Dm` output of a run of the testcase, {{{ ... 7fcb1bfff700: STM: 0x42002fdea0 : stmCommitTransaction()=0 7fcb1bfff700: STM: 0x505e28 : stmStartTransaction with 611 tokens 7fcb1bfff700: STM: 0x505e28 : stmStartTransaction()=0x42002fdea0 7fcb1bfff700: STM: 0x42002fdea0 : stmReadTVar(0x42002ff8f8) 7fcb1bfff700: STM: 0x42002fdea0 : get_entry_for TVar 0x42002ff8f8 7fcb1bfff700: STM: 0x42002fdea0 : FOR_EACH_ENTRY, current_chunk=0x42002ff3f8 limit=0 7fcb1bfff700: STM: 0x42002fdea0 : read_current_value(0x42002ff8f8)=0x42002ff919 7fcb1bfff700: STM: 0x42002fdea0 : stmReadTVar(0x42002ff8f8)=0x42002ff919 7fcb1bfff700: STM: 0x42002fdea0 : stmReadTVar(0x42002ff938) 7fcb1bfff700: STM: 0x42002fdea0 : get_entry_for TVar 0x42002ff938 7fcb1bfff700: STM: 0x42002fdea0 : FOR_EACH_ENTRY, current_chunk=0x42002ff3f8 limit=1 7fcb1bfff700: STM: 0x42002fdea0 : read_current_value(0x42002ff938)=0x42002ff95a 7fcb1bfff700: STM: 0x42002fdea0 : stmReadTVar(0x42002ff938)=0x42002ff95a 7fcb1bfff700: STM: 0x42002fdea0 : stmWriteTVar(0x42002ff8f8, 0x420002eee9) 7fcb1bfff700: STM: 0x42002fdea0 : get_entry_for TVar 0x42002ff8f8 7fcb1bfff700: STM: 0x42002fdea0 : FOR_EACH_ENTRY, current_chunk=0x42002ff3f8 limit=2 7fcb1bfff700: STM: 0x42002fdea0 : stmWriteTVar done 7fcb1bfff700: STM: 0x42002fdea0 : stmReadTVar(0x42002ff970) 7fcb1bfff700: STM: 0x42002fdea0 : get_entry_for TVar 0x42002ff970 7fcb1bfff700: STM: 0x42002fdea0 : FOR_EACH_ENTRY, current_chunk=0x42002ff3f8 limit=2 7fcb1bfff700: STM: 0x42002fdea0 : read_current_value(0x42002ff970)=0x42002ff991 7fcb1bfff700: STM: 0x42002fdea0 : stmReadTVar(0x42002ff970)=0x42002ff991 7fcb1bfff700: STM: 0x42002fdea0 : stmWriteTVar(0x42002ff9b0, 0x420002efd2) 7fcb1bfff700: STM: 0x42002fdea0 : get_entry_for TVar 0x42002ff9b0 7fcb1bfff700: STM: 0x42002fdea0 : FOR_EACH_ENTRY, current_chunk=0x42002ff3f8 limit=3 7fcb1bfff700: STM: 0x42002fdea0 : read_current_value(0x42002ff9b0)=0x4fe191 7fcb1bfff700: STM: 0x42002fdea0 : stmWriteTVar done 7fcb1bfff700: STM: 0x42002fdea0 : stmWriteTVar(0x42002ff970, 0x420002efe9) 7fcb1bfff700: STM: 0x42002fdea0 : get_entry_for TVar 0x42002ff970 7fcb1bfff700: STM: 0x42002fdea0 : FOR_EACH_ENTRY, current_chunk=0x42002ff3f8 limit=4 7fcb1bfff700: STM: 0x42002fdea0 : stmWriteTVar done 7fcb1bfff700: STM: 0x42002fdea0 : stmReadTVar(0x42002ff688) 7fcb1bfff700: STM: 0x42002fdea0 : get_entry_for TVar 0x42002ff688 7fcb1bfff700: STM: 0x42002fdea0 : FOR_EACH_ENTRY, current_chunk=0x42002ff3f8 limit=4 7fcb1bfff700: STM: 0x42002fdea0 : read_current_value(0x42002ff688)=0x42002ff230 7fcb1bfff700: STM: 0x42002fdea0 : stmReadTVar(0x42002ff688)=0x42002ff230 7fcb1bfff700: STM: 0x42002fdea0 : stmWriteTVar(0x42002ff688, 0x420002a088) 7fcb1bfff700: STM: 0x42002fdea0 : get_entry_for TVar 0x42002ff688 7fcb1bfff700: STM: 0x42002fdea0 : FOR_EACH_ENTRY, current_chunk=0x42002ff3f8 limit=5 7fcb1bfff700: STM: 0x42002fdea0 : stmWriteTVar done 7fcb1bfff700: STM: 0x42002fdea0 : stmGetInvariantsToCheck, head was 0x505e18 7fcb1bfff700: STM: 0x42002fdea0 : lock_stm() 7fcb1bfff700: STM: 0x42002fdea0 : lock_tvar(0x42002ff8f8) 7fcb1bfff700: STM: 0x42002fdea0 : checking for invariants on 0x42002ff8f8 7fcb1bfff700: STM: 0x42002fdea0 : unlock_tvar(0x42002ff8f8, 0x42002ff919) 7fcb1bfff700: STM: 0x42002fdea0 : lock_tvar(0x42002ff970) 7fcb1bfff700: STM: 0x42002fdea0 : checking for invariants on 0x42002ff970 7fcb1bfff700: STM: 0x42002fdea0 : unlock_tvar(0x42002ff970, 0x42002ff991) 7fcb1bfff700: STM: 0x42002fdea0 : lock_tvar(0x42002ff9b0) 7fcb1bfff700: STM: 0x42002fdea0 : checking for invariants on 0x42002ff9b0 7fcb1bfff700: STM: 0x42002fdea0 : unlock_tvar(0x42002ff9b0, 0x4fe191) 7fcb1bfff700: STM: 0x42002fdea0 : lock_tvar(0x42002ff688) 7fcb1bfff700: STM: 0x42002fdea0 : checking for invariants on 0x42002ff688 7fcb1bfff700: STM: 0x42002fdea0 : Touching invariant 0x42002ffd50 7fcb1bfff700: STM: 0x42002fdea0 : Not already found 0x42002ffd50 7fcb1bfff700: STM: 0x42002fdea0 : unlock_tvar(0x42002ff688, 0x42002ff230) 7fcb1bfff700: STM: 0x42002fdea0 : unlock_stm() 7fcb1bfff700: STM: 0x42002fdea0 : stmGetInvariantsToCheck, head now 0x420002c6d8 7fcb1bfff700: STM: 0x42002fdea0 : stmStartTransaction with 610 tokens 7fcb1bfff700: STM: 0x42002fdea0 : stmStartTransaction()=0x420002c6f8 7fcb1bfff700: STM: 0x420002c6f8 : stmReadTVar(0x42002ff688) 7fcb1bfff700: STM: 0x420002c6f8 : get_entry_for TVar 0x42002ff688 7fcb1bfff700: STM: 0x420002c6f8 : FOR_EACH_ENTRY, current_chunk=0x420002c720 limit=0 7fcb1bfff700: STM: 0x42002fdea0 : FOR_EACH_ENTRY, current_chunk=0x42002ff3f8 limit=5 7fcb1bfff700: STM: 0x420002c6f8 : stmReadTVar(0x42002ff688)=0x420002a088 7fcb1bfff700: STM: 0x420002c6f8 : stmAbortTransaction 7fcb1bfff700: STM: 0x420002c6f8 : lock_stm() 7fcb1bfff700: STM: 0x420002c6f8 : retaining read-set into parent 0x42002fdea0 7fcb1bfff700: STM: 0x420002c6f8 : FOR_EACH_ENTRY, current_chunk=0x420002c720 limit=1 7fcb1bfff700: STM: 0x42002fdea0 : FOR_EACH_ENTRY, current_chunk=0x42002ff3f8 limit=5 7fcb1bfff700: STM: 0x420002c6f8 : unlock_stm() 7fcb1bfff700: STM: 0x420002c6f8 : stmAbortTransaction done 7fcb1bfff700: STM: 0x42002fdea0 : stmCommitTransaction() 7fcb1bfff700: STM: 0x42002fdea0 : lock_stm() 7fcb1bfff700: STM: 0x42002fdea0 : locking invariants 7fcb1bfff700: STM: 0x42002fdea0 : locking invariant 0x42002ffd50 7fcb1bfff700: STM: 0x42002fdea0 : failed to lock 0x42002ffd50 7fcb1bfff700: STM: 0x42002fdea0 : finished locking invariants 7fcb1bfff700: STM: 0x42002fdea0 : FOR_EACH_ENTRY, current_chunk=0x42002ff3f8 limit=5 7fcb1bfff700: STM: 0x42002fdea0 : unlock_stm() 7fcb1bfff700: STM: 0x42002fdea0 : stmCommitTransaction()=0 }}} AFAICT the log essentially consists of tens of thousands of repeats similar to this. The `failed to lock` line is quite intriguing; -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14310#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14310: Assertion triggered by STM invariant. ----------------------------------+-------------------------------------- Reporter: mbw | Owner: (none) Type: bug | Status: new Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.2.1 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 (amd64) Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ----------------------------------+-------------------------------------- Comment (by bgamari): So I think I see one issue: in the event that `validate_and_acquire_ownership(cap, trec, (!use_read_phase), true);` in `stmCommitTransaction` returns `false` we will never unlock the invariants that we previously locked. I suspect we'll want something like this, {{{#!patch diff --git a/rts/STM.c b/rts/STM.c index 5c8fd4ff40..02ac22519b 100644 --- a/rts/STM.c +++ b/rts/STM.c @@ -1441,6 +1441,17 @@ StgBool stmCommitTransaction(Capability *cap, StgTRecHeader *trec) { } else { revert_ownership(cap, trec, false); } + } else { + // we encountered inconsistency; unlock the invariants that we locked as we + // are giving up on committing. + if (touched_invariants) { + StgInvariantCheckQueue *q = trec -> invariants_to_check; + while (q != END_INVARIANT_CHECK_QUEUE) { + StgAtomicInvariant *inv = q -> invariant; + unlock_inv(inv); + q = q -> next_queue_entry; + } + } } unlock_stm(trec); }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14310#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14310: Assertion triggered by STM invariant. ----------------------------------+-------------------------------------- Reporter: mbw | Owner: (none) Type: bug | Status: new Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.2.1 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 (amd64) Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4065 Wiki Page: | ----------------------------------+-------------------------------------- Changes (by bgamari): * differential: => Phab:D4065 Comment: Unfortunately the patch in comment:10 doesn't fix the assertion failure. Nevertheless, I strongly suspect that the patch is correct. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14310#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14310: Assertion triggered by STM invariant. ----------------------------------+-------------------------------------- Reporter: mbw | Owner: (none) Type: bug | Status: new Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.2.1 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 (amd64) Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4065 Wiki Page: | ----------------------------------+-------------------------------------- Comment (by bgamari): Actually, disregard comment:11; I was testing with the wrong compiler. It (with some tweaks; see the Diff) actually appears to resolve both issues. I would still like to hear the opinion of someone in the know, however. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14310#comment:12 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14310: Assertion triggered by STM invariant. ----------------------------------+-------------------------------------- Reporter: mbw | Owner: (none) Type: bug | Status: new Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.2.1 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 (amd64) Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4065 Wiki Page: | ----------------------------------+-------------------------------------- Comment (by bgamari): Unfortunately I'm still seeing occasional assertion failures with the patch. Frankly, I'm very confused by the treatment of invariants in `stmCommitTransaction`; it seems to me like we will happily proceed with committing a transaction (and touching invariants) even if we aren't able to lock all invariants (see the `break` around line 1350 of `STM.c`). This seems quite suspicious. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14310#comment:13 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14310: Assertion triggered by STM invariant. -------------------------------------+------------------------------------- Reporter: mbw | Owner: (none) Type: bug | Status: patch Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.2.1 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4065, Wiki Page: | Phab:D4067 -------------------------------------+------------------------------------- Changes (by bgamari): * status: new => patch * differential: Phab:D4065 => Phab:D4065, Phab:D4067 Comment: Alright, I believe the combination of Phab:D4065 and Phab:D4067 should nail this. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14310#comment:14 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14310: Assertion triggered by STM invariant. -------------------------------------+------------------------------------- Reporter: mbw | Owner: (none) Type: bug | Status: patch Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.2.1 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4065, Wiki Page: | Phab:D4067 -------------------------------------+------------------------------------- Comment (by fryguybob): I'll take a look at this later today. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14310#comment:15 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14310: Assertion triggered by STM invariant. -------------------------------------+------------------------------------- Reporter: mbw | Owner: (none) Type: bug | Status: patch Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.2.1 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4065, Wiki Page: | Phab:D4067 -------------------------------------+------------------------------------- Comment (by bgamari): For what it's worth, I left the testcase, compiled with these patches, running in a loop overnight. It still had not crashed this morning. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14310#comment:16 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14310: Assertion triggered by STM invariant. -------------------------------------+------------------------------------- Reporter: mbw | Owner: (none) Type: bug | Status: patch Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.2.1 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4065, Wiki Page: | Phab:D4067 -------------------------------------+------------------------------------- Comment (by bgamari): fryguybob had some very useful feedback on these patches. It seems like there is yet another issue: the assertion in `raiseAsync` is itself wrong (as I suspected in comment:5). To summarize: Normal top-level STM transactions (e.g. initiated by an `atomically` call in Haskell) by definition should have `enclosing_trec == NO_TREC`. The `ATOMICALLY_FRAME` handler in `raiseAsync` checks this invariant with an assert. In the case of nested transactions initiated by `orElse` we will hit a `CATCH_RETRY_FRAME` before hitting an `ATOMICALLY_FRAME`, avoiding this assertion. However, in the case of nested transactions started by the STM machinery for the purpose that we are checking invariants we have no such `CATCH_RETRY_FRAME`. `stg_atomically_frame` just pushes an `ATOMICALLY_FRAME`. Since this frame will have `enclosing_trec != NO_TREC`, this trips the assertion. Instead we should be asserting something like, {{{#!c DEBUG_ONLY(StgAtomicallyFrame *atomically = (StgAtomicallyFrame *) frame); ASSERT(tso->trec->enclosing_trec == NO_TREC || atomically->next_invariant_to_check != END_INVARIANT_CHECK_QUEUE); }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14310#comment:17 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14310: Assertion triggered by STM invariant. -------------------------------------+------------------------------------- Reporter: mbw | Owner: (none) Type: bug | Status: patch Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.2.1 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4065, Wiki Page: | Phab:D4067, Phab:D4073 -------------------------------------+------------------------------------- Changes (by bgamari): * differential: Phab:D4065, Phab:D4067 => Phab:D4065, Phab:D4067, Phab:D4073 Comment: Okay, I think the last piece of this is Phab:D4073. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14310#comment:18 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14310: Assertion triggered by STM invariant. -------------------------------------+------------------------------------- Reporter: mbw | Owner: (none) Type: bug | Status: patch Priority: high | Milestone: 8.6.1 Component: Compiler | Version: 8.2.1 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4065, Wiki Page: | Phab:D4067, Phab:D4073 -------------------------------------+------------------------------------- Changes (by bgamari): * milestone: 8.4.1 => 8.6.1 Comment: Unfortunately the above patches are rather subtly wrong. It's unclear whether I will have time to dive back into this before the release and frankly I'm not particularly inclined to given that we will likely be [[https://github.com/ghc-proposals/ghc-proposals/pull/77|removing]] the invariants mechanism in 8.6. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14310#comment:19 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14310: Assertion triggered by STM invariant. -------------------------------------+------------------------------------- Reporter: mbw | Owner: (none) Type: bug | Status: patch Priority: high | Milestone: 8.6.1 Component: Compiler | Version: 8.2.1 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4065, Wiki Page: | Phab:D4067, Phab:D4073, Phab:D4760 -------------------------------------+------------------------------------- Changes (by bgamari): * differential: Phab:D4065, Phab:D4067, Phab:D4073 => Phab:D4065, Phab:D4067, Phab:D4073, Phab:D4760 Comment: I carry out the removal (almost entirely taken from a patch by fryguybob) in Phab:D4760. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14310#comment:20 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14310: Assertion triggered by STM invariant. -------------------------------------+------------------------------------- Reporter: mbw | Owner: (none) Type: bug | Status: closed Priority: high | Milestone: 8.6.1 Component: Compiler | Version: 8.2.1 Resolution: fixed | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: #14324 | Differential Rev(s): Phab:D4065, Wiki Page: | Phab:D4067, Phab:D4073, Phab:D4760 -------------------------------------+------------------------------------- Changes (by bgamari): * status: patch => closed * resolution: => fixed * related: => #14324 Comment: Removal is being track in #14234. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14310#comment:21 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14310: Assertion triggered by STM invariant.
-------------------------------------+-------------------------------------
Reporter: mbw | Owner: (none)
Type: bug | Status: closed
Priority: high | Milestone: 8.6.1
Component: Compiler | Version: 8.2.1
Resolution: fixed | Keywords:
Operating System: Linux | Architecture: x86_64
| (amd64)
Type of failure: Runtime crash | Test Case:
Blocked By: | Blocking:
Related Tickets: #14324 | Differential Rev(s): Phab:D4065,
Wiki Page: | Phab:D4067, Phab:D4073, Phab:D4760
-------------------------------------+-------------------------------------
Comment (by Ben Gamari
participants (1)
-
GHC