[GHC] #12070: SMP primitives broken on power(pc)
#12070: SMP primitives broken on power(pc) -------------------------------------+------------------------------------- Reporter: hvr | Owner: trommler Type: bug | Status: new Priority: highest | Milestone: 8.0.1 Component: Runtime | Version: 8.0.1-rc4 System | Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- I originally noticed this when working on the AIX port (32-bit powerpc), and recently saw this also on Linux/powerpc64, which lead to talking to Peter Trommler who already had a suspicion: Here's for example the CAS definition (in `<stg/SMP.h>`): {{{#!c StgWord cas(StgVolatilePtr p, StgWord o, StgWord n) { StgWord result; __asm__ __volatile__ ( "1: ldarx %0, 0, %3\n" " cmpd %0, %1\n" " bne 2f\n" " stdcx. %2, 0, %3\n" " bne- 1b\n" "2:" :"=&r" (result) :"r" (o), "r" (n), "r" (p) :"cc", "memory" ); return result; }}} The important detail is the lack any barrier instructions, such as `isync` at the end. This results in infrequent heap-corruptions which in turn result in all sorts of infrequent and hard to track down runtime-crashes (including in `ghc -j`). Peter has already a patch in the works which simply replaces the atomic powerpc primitives with `__sync_*` intrinsics which turn out to be more portable than inline-asm. I've been testing the patch already and it seems to have made all issues I experienced so far disappear, as well as fixing the `concprog01` test which was also failing infrequently. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12070> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12070: SMP primitives broken on power(pc) -------------------------------------+--------------------------------- Reporter: hvr | Owner: trommler Type: bug | Status: new Priority: highest | Milestone: 8.0.1 Component: Runtime System | Version: 8.0.1-rc4 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: powerpc Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+--------------------------------- Changes (by hvr): * architecture: Unknown/Multiple => powerpc -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12070#comment:1> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12070: SMP primitives broken on power(pc) -------------------------------------+--------------------------------- Reporter: hvr | Owner: trommler Type: bug | Status: new Priority: highest | Milestone: 8.0.1 Component: Runtime System | Version: 8.0.1-rc4 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: powerpc Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+--------------------------------- Description changed by hvr: @@ -24,0 +24,1 @@ + } @@ -29,1 +30,7 @@ - (including in `ghc -j`). + (including in `ghc --make -j`) such as for instance + + {{{ + internal error: END_TSO_QUEUE object entered! + (GHC version 8.0.0.20160421 for powerpc64_unknown_linux) + }}} + @@ -33,3 +40,30 @@ - portable than inline-asm. I've been testing the patch already and it seems - to have made all issues I experienced so far disappear, as well as fixing - the `concprog01` test which was also failing infrequently. + portable than inline-asm. This would result in e.g. + + + {{{#!c + StgWord + cas(StgVolatilePtr p, StgWord o, StgWord n) + { + return __sync_val_compare_and_swap (p, o, n); + } + }}} + + which then gets compiled as + + {{{#!asm + 000000000000004c <.cas>: + 4c: 7c 00 04 ac sync + 50: 7d 20 18 a8 ldarx r9,0,r3 + 54: 7c 29 20 00 cmpd r9,r4 + 58: 40 c2 00 0c bne- 64 <.cas+0x18> + 5c: 7c a0 19 ad stdcx. r5,0,r3 + 60: 40 c2 ff f0 bne- 50 <.cas+0x4> + 64: 4c 00 01 2c isync + 68: 7d 23 4b 78 mr r3,r9 + 6c: 4e 80 00 20 blr + }}} + + + I've been testing the patch already and it seems to have made all issues I + experienced so far disappear, as well as fixing the `concprog01` test + which was also failing infrequently. New description: I originally noticed this when working on the AIX port (32-bit powerpc), and recently saw this also on Linux/powerpc64, which lead to talking to Peter Trommler who already had a suspicion: Here's for example the CAS definition (in `<stg/SMP.h>`): {{{#!c StgWord cas(StgVolatilePtr p, StgWord o, StgWord n) { StgWord result; __asm__ __volatile__ ( "1: ldarx %0, 0, %3\n" " cmpd %0, %1\n" " bne 2f\n" " stdcx. %2, 0, %3\n" " bne- 1b\n" "2:" :"=&r" (result) :"r" (o), "r" (n), "r" (p) :"cc", "memory" ); return result; } }}} The important detail is the lack any barrier instructions, such as `isync` at the end. This results in infrequent heap-corruptions which in turn result in all sorts of infrequent and hard to track down runtime-crashes (including in `ghc --make -j`) such as for instance {{{ internal error: END_TSO_QUEUE object entered! (GHC version 8.0.0.20160421 for powerpc64_unknown_linux) }}} Peter has already a patch in the works which simply replaces the atomic powerpc primitives with `__sync_*` intrinsics which turn out to be more portable than inline-asm. This would result in e.g. {{{#!c StgWord cas(StgVolatilePtr p, StgWord o, StgWord n) { return __sync_val_compare_and_swap (p, o, n); } }}} which then gets compiled as {{{#!asm 000000000000004c <.cas>: 4c: 7c 00 04 ac sync 50: 7d 20 18 a8 ldarx r9,0,r3 54: 7c 29 20 00 cmpd r9,r4 58: 40 c2 00 0c bne- 64 <.cas+0x18> 5c: 7c a0 19 ad stdcx. r5,0,r3 60: 40 c2 ff f0 bne- 50 <.cas+0x4> 64: 4c 00 01 2c isync 68: 7d 23 4b 78 mr r3,r9 6c: 4e 80 00 20 blr }}} I've been testing the patch already and it seems to have made all issues I experienced so far disappear, as well as fixing the `concprog01` test which was also failing infrequently. -- -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12070#comment:2> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12070: SMP primitives broken on power(pc) -------------------------------------+--------------------------------- Reporter: hvr | Owner: trommler Type: bug | Status: new Priority: highest | Milestone: 8.0.1 Component: Runtime System | Version: 8.0.1-rc4 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: powerpc Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+--------------------------------- Changes (by erikd): * cc: erikd (added) -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12070#comment:3> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12070: SMP primitives broken on power(pc) -------------------------------------+------------------------------------- Reporter: hvr | Owner: trommler Type: bug | Status: new Priority: highest | Milestone: 8.0.1 Component: Runtime System | Version: 8.0.1-rc4 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: powerpc Type of failure: None/Unknown | Test Case: | concurrent/prog001/concprog001 Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by trommler): * testcase: => concurrent/prog001/concprog001 -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12070#comment:4> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12070: SMP primitives broken on power(pc) -------------------------------------+------------------------------------- Reporter: hvr | Owner: trommler Type: bug | Status: patch Priority: highest | Milestone: 8.0.1 Component: Runtime System | Version: 8.0.1-rc4 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: powerpc Type of failure: None/Unknown | Test Case: | concurrent/prog001/concprog001 Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D2225 Wiki Page: | -------------------------------------+------------------------------------- Changes (by trommler): * status: new => patch * differential: => Phab:D2225 Comment: Validated on powerpc64 and AMD64. Please, someone check that it does not break on ARM. The description for the built_in used in `xchg` suggests that the value stored in memory is implementation dependent. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12070#comment:5> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12070: SMP primitives broken on power(pc) -------------------------------------+------------------------------------- Reporter: hvr | Owner: trommler Type: bug | Status: patch Priority: highest | Milestone: 8.0.1 Component: Runtime System | Version: 8.0.1-rc4 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: powerpc Type of failure: None/Unknown | Test Case: | concurrent/prog001/concprog001 Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D2225 Wiki Page: | -------------------------------------+------------------------------------- Comment (by Ben Gamari <ben@…>): In [changeset:"563a4857abcee4a6e43c68323274309c58f42aa0/ghc" 563a485/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="563a4857abcee4a6e43c68323274309c58f42aa0" PPC: Implement SMP primitives using gcc built-ins The SMP primitives were missing appropriate memory barriers (sync, isync instructions) on all PowerPCs. Use the built-ins _sync_* provided by gcc and clang. This reduces code size significantly. Remove broken mark for concprog001 on powerpc64. The referenced ticket number (11259) was wrong. Test Plan: validate on powerpc and ARM Reviewers: erikd, austin, simonmar, bgamari, hvr Reviewed By: bgamari, hvr Subscribers: thomie Differential Revision: https://phabricator.haskell.org/D2225 GHC Trac Issues: #12070 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12070#comment:6> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12070: SMP primitives broken on power(pc) -------------------------------------+------------------------------------- Reporter: hvr | Owner: trommler Type: bug | Status: closed Priority: highest | Milestone: 8.0.1 Component: Runtime System | Version: 8.0.1-rc4 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: powerpc Type of failure: None/Unknown | Test Case: | concurrent/prog001/concprog001 Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D2225 Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * status: patch => closed * resolution: => fixed Comment: Merged to `ghc-8.0` as f4e6b32831bd718a040b382865ac7aea1254bf4e. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12070#comment:7> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
participants (1)
-
GHC