#13970: Segmentation fault inside threadPaused -------------------------------------+------------------------------------- Reporter: albertov | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 8.2.1-rc3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): I don't suppose you could provide a reproducer for this? -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13970#comment:1> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#13970: Segmentation fault inside threadPaused -------------------------------------+------------------------------------- Reporter: albertov | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 8.2.1-rc3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by albertov): Replying to [comment:2 bgamari]:

What might also help is if you could compile your program with `-debug` and paste the output of `x/64a tso->stackobj->sp` after the program crashes. It seems like we are getting confused walking the stack, so it would be interesting to know what the stack looks like.

I did but I still had no symbols in gdb. I must be striping them somewhere in my build... I will take a look and report back. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13970#comment:3> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#13970: Segmentation fault inside threadPaused -------------------------------------+------------------------------------- Reporter: albertov | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 8.2.1-rc3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): Note that I believe `Cabal` strips by default`.

It's hard for me to provide a reproducible example since it is in a complex proprietary program and I have of what could be causing this besides that increasing the number of capabilities with -N tends to increase the frequency of the crashes.

Quite understandable. Anything you can offer would be very much appreciated. I tested this patch with the testcase from #13615 with over a day of runtime without seeing this crash, so it seems that you are tickling something odd. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13970#comment:5> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#13970: Segmentation fault inside threadPaused -------------------------------------+------------------------------------- Reporter: albertov | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 8.2.1-rc3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by albertov): Replying to [comment:2 bgamari]:

What might also help is if you could compile your program with `-debug` and paste the output of `x/64a tso->stackobj->sp` after the program crashes. It seems like we are getting confused walking the stack, so it would be interesting to know what the stack looks like.

I finally managed to get debugging symbols. I had to pass {{{dontStrip = true}}} to the Nix derivation that builds ghc from git. Anyway, it crashed again at the same point and here's what gdb says: {{{ (gdb) x/64a tso->stackobj->sp 0x421cdbfea8: 0x7f9b3d2cc5c8 <stg_enter_info> 0x420c746c00 0x421cdbfeb8: 0x100 0x4204442df9 0x421cdbfec8: 0xf4 0xfffffffffffffff4 0x421cdbfed8: 0x7f9b3d2cce88 <stg_upd_frame_info> 0x420c746c00 0x421cdbfee8: 0x7f9b4367b308 <sigym4zmpropagzmenginezm0zi1zi0zi0zmFERo4wFJ8F465LTBnpBC6B_Sigym4ziPropagziEngine_zdwpolyzugo3_info+64> 0x6465c88fdad5932f 0x421cdbfef8: 0xffffffffffffffff 0x6 0x421cdbff08: 0x0 0x7f9b4367be08 0x421cdbff18: 0x42042655c8 0x42042655e0 0x421cdbff28: 0x4202361568 0x7f9b3d2cce88 <stg_upd_frame_info> 0x421cdbff38: 0x4204442dc8 0x7f9b436ac7b0 0x421cdbff48: 0x63b 0xffffffffffffffe4 0x421cdbff58: 0x420447bb61 0x7f9b441837a0 0x421cdbff68: 0x42187a9841 0x420447bb71 0x421cdbff78: 0xfffffffffffffff4 0x4204265518 0x421cdbff88: 0x42183d5c09 0x42183f1523 0x421cdbff98: 0x626 0x42187a0119 0x421cdbffa8: 0x7f9b3d2c8588 <stg_maskAsyncExceptionszh_ret_info> 0x7f9b3dbf2cc0 0x421cdbffb8: 0x7f9b3d2c8ae8 <stg_catch_frame_info> 0xc 0x421cdbffc8: 0x7f9b3e119592 0x7f9b3dbf2d80 0x421cdbffd8: 0x42183f155a 0x7f9b3d2c8ae8 <stg_catch_frame_info> 0x421cdbffe8: 0xc 0x7f9b3e14ee2a 0x421cdbfff8: 0x7f9b3d2c8088 <stg_stop_thread_info> 0x7f9b41aa3340 <linearzm1zi20zi6zm10zzIy9OH0MgIRAfBAdsUHU_LinearziV2_V2_con_info> 0x421cdc0008: 0x421cdc00d1 0x421cdc00c1 0x421cdc0018: 0x7f9b41aa3340 <linearzm1zi20zi6zm10zzIy9OH0MgIRAfBAdsUHU_LinearziV2_V2_con_info> 0x421cdc00f1 0x421cdc0028: 0x421cdc00e1 0x7f9b41aa3340 <linearzm1zi20zi6zm10zzIy9OH0MgIRAfBAdsUHU_LinearziV2_V2_con_info> 0x421cdc0038: 0x421cdc0111 0x421cdc0101 0x421cdc0048: 0x7f9b41aa3340 <linearzm1zi20zi6zm10zzIy9OH0MgIRAfBAdsUHU_LinearziV2_V2_con_info> 0x421cdc0131 0x421cdc0058: 0x421cdc0121 0x7f9b41aa3340 <linearzm1zi20zi6zm10zzIy9OH0MgIRAfBAdsUHU_LinearziV2_V2_con_info> 0x421cdc0068: 0x421cdc0151 0x421cdc0141 0x421cdc0078: 0x7f9b41aa3340 <linearzm1zi20zi6zm10zzIy9OH0MgIRAfBAdsUHU_LinearziV2_V2_con_info> 0x421cdc0171 0x421cdc0088: 0x421cdc0161 0x7f9b41aa3340 <linearzm1zi20zi6zm10zzIy9OH0MgIRAfBAdsUHU_LinearziV2_V2_con_info> 0x421cdc0098: 0x421cdc0191 0x421cdc0181 }}} BTW, I've got an approval to factor out the spread algorithm from our program as an opensource library so I'll begin working on that tomorrow. Hopefully the bug still manifests itself. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13970#comment:7> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#13970: Segmentation fault inside threadPaused -------------------------------------+------------------------------------- Reporter: albertov | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 8.2.1-rc3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by albertov): I've got a SEGFAULT in a new location which seems related to the same issue: {{{ Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f747043a32f in stg_BLACKHOLE_info () from /nix/store/ka5975xi1b7vcw98a1agqhb0y4gxcwbj- ghc-8.2.0.20170704/lib/ghc-8.2.0.20170704/rts/libHSrts_thr_debug- ghc8.2.0.20170704.so [Current thread is 1 (LWP 25315)] warning: File "/nix/store/xfrkm34sk0a13ha9bpki61l2k5g1v8dh- gcc-5.4.0-lib/lib/libstdc++.so.6.0.21-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto- load". (gdb) bt #0 0x00007f747043a32f in stg_BLACKHOLE_info () from /nix/store/ka5975xi1b7vcw98a1agqhb0y4gxcwbj- ghc-8.2.0.20170704/lib/ghc-8.2.0.20170704/rts/libHSrts_thr_debug- ghc8.2.0.20170704.so #1 0x0000000000000000 in ?? () (gdb) info locals No symbol table info available. (gdb) disassemble Dump of assembler code for function stg_BLACKHOLE_info: 0x00007f747043a240 <+0>: mov 0x8(%rbx),%rax 0x00007f747043a244 <+4>: test $0x7,%al 0x00007f747043a246 <+6>: jne 0x7f747043a32c <stg_BLACKHOLE_info+236> 0x00007f747043a24c <+12>: mov (%rax),%rcx 0x00007f747043a24f <+15>: cmp 0x2d7f2(%rip),%rcx # 0x7f7470467a48 0x00007f747043a256 <+22>: je 0x7f747043a240 <stg_BLACKHOLE_info> 0x00007f747043a258 <+24>: cmp 0x2d7e1(%rip),%rcx # 0x7f7470467a40 0x00007f747043a25f <+31>: je 0x7f747043a2a4 <stg_BLACKHOLE_info+100> 0x00007f747043a261 <+33>: cmp 0x2d7b0(%rip),%rcx # 0x7f7470467a18 0x00007f747043a268 <+40>: je 0x7f747043a2a4 <stg_BLACKHOLE_info+100> 0x00007f747043a26a <+42>: cmp 0x2d79f(%rip),%rcx # 0x7f7470467a10 0x00007f747043a271 <+49>: je 0x7f747043a2a4 <stg_BLACKHOLE_info+100> 0x00007f747043a273 <+51>: test $0x7,%al 0x00007f747043a275 <+53>: jne 0x7f747043a345 <stg_BLACKHOLE_info+261> 0x00007f747043a27b <+59>: mov (%rax),%rbx 0x00007f747043a27e <+62>: cmpl $0x1a,-0x8(%rbx) 0x00007f747043a282 <+66>: jb 0x7f747043a34b <stg_BLACKHOLE_info+267> 0x00007f747043a288 <+72>: cmpl $0x1c,-0x8(%rbx) 0x00007f747043a28c <+76>: jb 0x7f747043a332 <stg_BLACKHOLE_info+242> 0x00007f747043a292 <+82>: cmpl $0x1d,-0x8(%rbx) 0x00007f747043a296 <+86>: jb 0x7f747043a33c <stg_BLACKHOLE_info+252> 0x00007f747043a29c <+92>: mov %rbx,%rcx 0x00007f747043a29f <+95>: mov %rax,%rbx 0x00007f747043a2a2 <+98>: jmpq *%rcx 0x00007f747043a2a4 <+100>: lea -0xeebb(%rip),%rax # 0x7f747042b3f0 <allocate> 0x00007f747043a2ab <+107>: lea -0x18(%r13),%rcx 0x00007f747043a2af <+111>: mov $0x4,%edx 0x00007f747043a2b4 <+116>: sub $0x8,%rsp 0x00007f747043a2b8 <+120>: mov %rdx,%rsi 0x00007f747043a2bb <+123>: mov %rcx,%rdi 0x00007f747043a2be <+126>: mov %rax,%rcx 0x00007f747043a2c1 <+129>: xor %eax,%eax 0x00007f747043a2c3 <+131>: callq *%rcx 0x00007f747043a2c5 <+133>: add $0x8,%rsp 0x00007f747043a2c9 <+137>: lea 0x6e0(%rip),%rcx # 0x7f747043a9b0 <stg_MSG_BLACKHOLE_info> 0x00007f747043a2d0 <+144>: mov %rcx,(%rax) 0x00007f747043a2d3 <+147>: mov 0x368(%r13),%rcx 0x00007f747043a2da <+154>: mov %rcx,0x10(%rax) 0x00007f747043a2de <+158>: mov %rbx,0x18(%rax) 0x00007f747043a2e2 <+162>: lea -0x2e299(%rip),%rcx # 0x7f747040c050 <messageBlackHole> 0x00007f747043a2e9 <+169>: lea -0x18(%r13),%rdx 0x00007f747043a2ed <+173>: mov %rax,%rsi 0x00007f747043a2f0 <+176>: sub $0x8,%rsp 0x00007f747043a2f4 <+180>: mov %rdx,%rdi 0x00007f747043a2f7 <+183>: mov %rax,%rdx 0x00007f747043a2fa <+186>: xor %eax,%eax 0x00007f747043a2fc <+188>: mov %rdx,%r14 0x00007f747043a2ff <+191>: callq *%rcx 0x00007f747043a301 <+193>: add $0x8,%rsp 0x00007f747043a305 <+197>: test %rax,%rax 0x00007f747043a308 <+200>: je 0x7f747043a240 <stg_BLACKHOLE_info> 0x00007f747043a30e <+206>: mov 0x368(%r13),%rax 0x00007f747043a315 <+213>: movw $0x2,0x22(%rax) 0x00007f747043a31b <+219>: mov 0x368(%r13),%rax 0x00007f747043a322 <+226>: mov %r14,0x28(%rax) 0x00007f747043a326 <+230>: jmpq 0x7f7470438d58 <stg_block_blackhole> 0x00007f747043a32b <+235>: nop 0x00007f747043a32c <+236>: mov %rax,%rbx => 0x00007f747043a32f <+239>: jmpq *0x0(%rbp) 0x00007f747043a332 <+242>: cmpl $0x1b,-0x8(%rbx) 0x00007f747043a336 <+246>: jb 0x7f747043a29c <stg_BLACKHOLE_info+92> 0x00007f747043a33c <+252>: mov 0x8(%rax),%rax 0x00007f747043a340 <+256>: jmpq 0x7f747043a273 <stg_BLACKHOLE_info+51> 0x00007f747043a345 <+261>: mov %rax,%rbx 0x00007f747043a348 <+264>: jmpq *0x0(%rbp) 0x00007f747043a34b <+267>: cmpl $0xf,-0x8(%rbx) 0x00007f747043a34f <+271>: jb 0x7f747043a36d <stg_BLACKHOLE_info+301> 0x00007f747043a351 <+273>: cmpl $0x19,-0x8(%rbx) 0x00007f747043a355 <+277>: jb 0x7f747043a35d <stg_BLACKHOLE_info+285> 0x00007f747043a357 <+279>: mov %rax,%rbx 0x00007f747043a35a <+282>: jmpq *0x0(%rbp) 0x00007f747043a35d <+285>: movslq -0x8(%rbx),%rcx 0x00007f747043a361 <+289>: cmp $0x17,%rcx 0x00007f747043a365 <+293>: jne 0x7f747043a29c <stg_BLACKHOLE_info+92> 0x00007f747043a36b <+299>: jmp 0x7f747043a357 <stg_BLACKHOLE_info+279> 0x00007f747043a36d <+301>: cmpl $0x8,-0x8(%rbx) 0x00007f747043a371 <+305>: jb 0x7f747043a29c <stg_BLACKHOLE_info+92> 0x00007f747043a377 <+311>: movslq -0x8(%rbx),%rbx 0x00007f747043a37b <+315>: add $0xfffffffffffffff8,%rbx 0x00007f747043a37f <+319>: lea 0x1eca2(%rip),%rcx # 0x7f7470459028 0x00007f747043a386 <+326>: movslq (%rcx,%rbx,8),%rbx 0x00007f747043a38a <+330>: add %rbx,%rcx 0x00007f747043a38d <+333>: jmpq *%rcx End of assembler dump. }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13970#comment:9> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#13970: Segmentation fault inside threadPaused -------------------------------------+------------------------------------- Reporter: albertov | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 8.2.1-rc3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by albertov): My hunch was wrong. Substituting HashMap for Map in the variable stored in a TMVar which threads contend for did not eliminate the segfaults. It made them much more frequent however. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13970#comment:11> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#13970: Segmentation fault inside threadPaused -------------------------------------+------------------------------------- Reporter: albertov | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 8.2.1-rc3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by hsyl20): GHC uses [https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/SymbolNames z-encoding], hence "zd" is "$" and "zu" is "_" . You can remove the `$w` prefix probably introduced by the worker/wrapper transformation, leaving you with `poly_go13` or maybe `poly_go` in the Haskell source. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13970#comment:13> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#13970: Segmentation fault inside threadPaused -------------------------------------+------------------------------------- Reporter: albertov | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 8.2.1-rc3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by mpickering): You can use the `ghc` option `--show-iface` to inspect the `.hi` file. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13970#comment:15> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#13970: Segmentation fault inside threadPaused -------------------------------------+------------------------------------- Reporter: albertov | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 8.2.1-rc3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by albertov): Replying to [comment:14 albertov]:

Replying to [comment:13 hsyl20]:

...

GHC uses [https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/SymbolNames z-encoding], hence "zd" is "$" and "zu" is "_" . You can remove the `$w` prefix probably introduced by the worker/wrapper transformation, leaving you with `poly_go13` or maybe `poly_go` in the Haskell source.

There's no function by those names in that module, however, I could manage to find "poly_go13" it's header (.hi) file. It's in a binary format which gives me no hints regarding where it might come from. Is there any way to inspect it in a more amicable way? Thanks!

I've found out by compiling with {{{-ddump-simpl}}} that {{{poly_go13}}} seems to be a specialization of {{{Data.Map.lookup}}}: {{{ $wpoly_go13 [InlPrag=[0], Occ=LoopBreaker] :: forall a. Int# -> Int# -> Map BlockIndex a -> Maybe a [GblId, Arity=3, Caf=NoCafRefs, Str=<L,U><L,U><S,1*U>] $wpoly_go13 = \ (@ a) (ww :: Int#) (ww1 :: Int#) (w :: Map BlockIndex a) -> case w of { Bin ipv ipv1 ipv2 ipv3 ipv4 -> case ipv1 of { V2 b1 b2 -> case b1 of { I# y# -> case b2 of { I# y#1 -> case tagToEnum# @ Bool (<# ww y#) of { False -> case tagToEnum# @ Bool (==# ww y#) of { False -> $wpoly_go13 @ a ww ww1 ipv4; True -> case tagToEnum# @ Bool (<# ww1 y#1) of { False -> case tagToEnum# @ Bool (==# ww1 y#1) of { False -> $wpoly_go13 @ a ww ww1 ipv4; True -> Just @ a ipv2 }; True -> $wpoly_go13 @ a ww ww1 ipv3 } }; True -> $wpoly_go13 @ a ww ww1 ipv3 } } } }; Tip -> Nothing @ a } end Rec } }}} where {{{ type BlockIndex = Linear.V2.V2 Int }}} I can relate to the part of my program where this comes from and, interestingly, this was originally a {{{Data.HashMap.Strict.HashMap}}} which I changed to a {{{Data.Map.Strict.Map}}} to see if it made a difference (see comment:10). For some reason, segfaults are much more frequent with the Data.Map (so I've left it like this to help debug). This Map is stored in a TMVar which threads regularly {{{M.lookup blockIx <$> readTMVar}}} neighboring Blocks to send them "work". When the lookup fails they take the lock, create a new Block and a new thread to process it, put the Block back in the Map and put the Map back in the TMVar. So, although the BlockIndex is strict, the value isn't so perhaps this is where shared un-evaluated thunks are created which manifests the bug? (if my intuition about the problem is correct). Anyway, now I have some ideas on how to attempt to reproduce this bug outside of my program, which might be quicker than factoring out the whole engine out of the propietary parts. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13970#comment:17> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#13970: Segmentation fault inside threadPaused -------------------------------------+------------------------------------- Reporter: albertov | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 8.2.1-rc3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by albertov): After removing many dependencies and some cleanup we've opensourced the engine of the problematic wildfire simulator. Fortunately the bug still manifests itself (which rules out some kind of nasty interaction with the foreign libraries it linked to). To reproduce clone https://github.com/meteogrid/propag , `cabal new-build` it and run the `propag-demo` executable. Using the rc3 pre-release and running with `+RTS -N12` it crashed 100% of the times I've tried (the refactor has increased the change of crash). Sorry for this monster of a reproducible case. It's the best I could do to minimize the amount of third-party dependencies for the time being. I'm planning to remove the dependency on the wildfire-specific stuff which should remove more dependencies (and make it useful as an arbitrary 2D spread automata). I've tried initially to reproduce it starting from scratch but did not succeed so I pulled out all the bigger dependencies (eg: meteogrid/bindings-gdal) which require bindings to c++ libraries. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13970#comment:19> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#13970: Segmentation fault inside threadPaused -------------------------------------+------------------------------------- Reporter: albertov | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 8.2.1-rc3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by albertov): Replying to [comment:20 bgamari]:

Thanks albertov! I'm building your repro as we speak.

Thanks! Did you manage to reproduce the segfault? By the way, I forgot to warn you that this creates a temporary work directory using `System.IO.Temp.withSystemTempDirectory` named `propag-work-XXXX`. It is mean to be cleared after running but if it crashes it wont be so make sure you clean them as they can add up to a lot of space after many experiments. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13970#comment:21> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#13970: Segmentation fault inside threadPaused -------------------------------------+------------------------------------- Reporter: albertov | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 8.2.1-rc3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by albertov): Replying to [comment:22 bgamari]:

I have managed to reproduce it a few times although it does seem to take a while. Moreover, it (perhaps not surprisingly) is quite dependent upon paralellism. I was unable to reproduce it in a reasonable amount of time on my dual-core, four-thread laptop. However, on my 4-core, eight-thread server I was able to reproduce it within ten minutes or so. How many cores does your test environment have?

I'm testing on a 6-core, 12-thread machine. It is indeed hard to reproduce although it is no much more frequent. I've noticed that giving a large `-N` value to the RTS increases the odds. I'm testing with `-N30`. Have you tried increasing this value way over your number of cores? Another interesting thing is that without this [https://github.com/meteogrid/propag/blob/master/app/Main.hs#L69 pause] I was unable to reproduce the crash and noticed that a breakpoint I had set at `suspendComputation` was never being hit. Adding this pause causes `suspendComputation` to be called multiple times and eventually manifests the problem (when running outside gdb). Maybe playing with that pause helps increasing the odds in your environment? -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13970#comment:23> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#13970: Segmentation fault inside threadPaused -------------------------------------+------------------------------------- Reporter: albertov | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 8.2.1-rc3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): I have confirmed that c1c0985416a6f9766c03d361449f556905bf8e1d really is the first bad commit. I also noticed that it is possible to reproduce the crash using `forkIO` instead of `forkOS`, which makes it a bit easier to debug. Presumably this is safe since there are no native dependencies. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13970#comment:25> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#13970: Segmentation fault inside threadPaused -------------------------------------+------------------------------------- Reporter: albertov | Owner: (none) Type: bug | Status: patch Priority: highest | Milestone: 8.2.1 Component: Runtime System | Version: 8.2.1-rc3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by Ben Gamari <ben@…>): In [changeset:"bade356f79d44c9f6e8918a89d9ffac7f5608dbf/ghc" bade356/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="bade356f79d44c9f6e8918a89d9ffac7f5608dbf" rts: Claim AP_STACK before adjusting Sp In the fix to #13615 we introduced some logic to atomically blackhole AP_STACKs closures upon entry. However, this logic was placed *after* a stack pointer adjustment. This meant that if someone else beat us to blackholing the AP_STACK we would suspend the thread with uninitialized content on the stack. This would then later blow up when threadPaused attempted to walk the stack, hence #13970. Silly bug but still cost lots of head-scratching to find. Thanks to albertov for the great repro. Fixes #13970. Bug originally introduced by the fix to #13615. Reviewers: austin, erikd, simonmar Reviewed By: erikd, simonmar Subscribers: rwbarton, thomie GHC Trac Issues: #13970, #13615 Differential Revision: https://phabricator.haskell.org/D3760 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13970#comment:27> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#13970: Segmentation fault inside threadPaused -------------------------------------+------------------------------------- Reporter: albertov | Owner: (none) Type: bug | Status: closed Priority: highest | Milestone: 8.2.1 Component: Runtime System | Version: 8.2.1-rc3 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by albertov): Amazing. I'm building a new ghc right now to test it. Sorry I couldn't do it before, I didn't expect it to be fixed so quickly! Thanks bgamari! -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13970#comment:29> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#13970: Segmentation fault inside threadPaused -------------------------------------+------------------------------------- Reporter: albertov | Owner: (none) Type: bug | Status: closed Priority: highest | Milestone: 8.2.1 Component: Runtime System | Version: 8.2.1-rc3 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): I'm glad it helped! -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13970#comment:31> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler