#10375: arm: ghci.debugger tests hit illegal instruction -------------------------------------+--------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 7.12.1 Component: GHCi | Version: 7.11 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+--------------------------------- Old description:

Some GHCi tests on arm/linux ie:

{{{ (cd testsuite ; make TEST="print018 print020 print021 print022 print025") }}}

fail, all with the same error:

{{{ Stderr: Illegal instruction }}}

New description: Some GHCi tests on arm/linux ie: {{{ (cd testsuite ; make TEST="print018 print020 print021 print022 print025") }}} fail, all with the same error: {{{ Stderr: Illegal instruction }}} -- Comment (by erikd): Tests `ghci053 ghcirun001 ghcirun002`also fail in the same way. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+--------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 7.12.1 Component: GHCi | Version: 7.11 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+--------------------------------- -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+--------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 7.12.1 Component: GHCi | Version: 7.11 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+--------------------------------- Comment (by erikd): Yes, arm specific. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+--------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 7.10.2 Component: GHCi | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+--------------------------------- Comment (by erikd): If I compile the RTS with `-debug` and run {{{ inplace/bin/ghc-stage +RTS -DS -Dl -RTS --interactive }}} it fails as follows: {{{ Prelude> data X = A | B deriving Eq lookupSymbol: looking up ghczmprim_GHCziTypes_True_closure lookupSymbol: symbol not found lookupSymbol: looking up ghczmprim_GHCziTypes_False_closure lookupSymbol: symbol not found lookupSymbol: looking up ghczmprim_GHCziTypes_False_closure lookupSymbol: symbol not found lookupSymbol: looking up ghczmprim_GHCziTypes_True_closure lookupSymbol: symbol not found lookupSymbol: looking up ghczmprim_GHCziClasses_DZCEq_con_info lookupSymbol: symbol not found lookupSymbol: looking up ghczmprim_GHCziClasses_zeze_closure lookupSymbol: symbol not found lookupSymbol: looking up ghczmprim_GHCziClasses_not_closure lookupSymbol: symbol not found Prelude> A == A lookupSymbol: looking up ghczmprim_GHCziClasses_zeze_closure lookupSymbol: symbol not found lookupSymbol: looking up ghczmprim_GHCziTypes_ZC_con_info lookupSymbol: symbol not found lookupSymbol: looking up ghczmprim_GHCziTypes_ZMZN_closure lookupSymbol: symbol not found lookupSymbol: looking up base_GHCziBase_returnIO_closure lookupSymbol: symbol not found lookupSymbol: looking up base_GHCziShow_zdfShowBool_closure lookupSymbol: symbol not found lookupSymbol: looking up base_SystemziIO_print_closure lookupSymbol: symbol not found lookupSymbol: looking up base_GHCziBase_thenIO_closure lookupSymbol: symbol not found Segmentation fault }}} This (apart from the segfault) is identical to what happens on x86_64/linux compiled the same way. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+--------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 7.10.2 Component: GHCi | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+--------------------------------- Comment (by erikd): A bit of `printf` debugging shows that the segault is happening in the `schedule` function of `rts/Schedule.c`, specifically on the call to `StgRun` around line number 460: {{{ case ThreadRunGHC: { StgRegTable *r; r = StgRun((StgFunPtr) stg_returnToStackTop, &cap->r); cap = regTableToCapability(r); ret = r->rRet; break; } }}} and on Arm, I have confirmed that `StgRun` is the assembler version. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+--------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 7.10.2 Component: GHCi | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+--------------------------------- Comment (by erikd): Managed to run `ghc-stage2 --interactive` in one terminal and the attach GDB to it in another using `ghc -tui -p <pid>`. I can then manually enter the two lines of test case code and have it segfault. The GDB backtrace is a little odd (even by haskell program standards): {{{ Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb14ff460 (LWP 19914)] Cannot access memory at address 0xb88849d4 (gdb) bt #0 0xb88849d4 in ?? () #1 0x70000000 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) }}} but that address `0xb88849d4` being in both the "Cannot access memory at" message and the stack might point to stack corruption while the `0x70000000` address on the stack is *very* suspicious. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+--------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 7.10.2 Component: GHCi | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+--------------------------------- Comment (by bgamari): Hmm, excellent sleuthing work so far. Have you tried running in Valgrind? It typically struggles to do anything useful on Haskell code but you never know. You may also want to add a `printf` outputting all of the relevant local bindings around line 460 to see where exactly this mysterious address is coming from. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:13 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+--------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 7.10.2 Component: GHCi | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+--------------------------------- Comment (by erikd): Using the gdb macros on the wiki I can retrieve the following info after the segfault: {{{ Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb14ff460 (LWP 7214)] 0xb88d09d4 in ?? () (gdb) bt #0 0xb88d09d4 in ?? () #1 0x70000000 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb) pregs $1 = {rR1 = {w = 0, a = 0x0, c = 0, f = 0, i = 0, p = 0x0} ,rR2 = {w = 0, a = 0x0, c = 0, f = 0, i = 0, p = 0x0} , rR3 = {w = 0, a = 0x0, c = 0, f = 0, i = 0, p = 0x0} , rR4 = {w = 0, a = 0x0, c = 0, f = 0, i = 0, p = 0x0} , rR5 = {w = 0, a = 0x0, c = 0, f = 0, i = 0, p = 0x0} , rR6 = {w = 0, a = 0x0, c = 0, f = 0, i = 0, p = 0x0} , rR7 = {w = 0, a = 0x0, c = 0, f = 0, i = 0, p = 0x0} , rR8 = {w = 0, a = 0x0, c = 0, f = 0, i = 0, p = 0x0} , rR9 = {w = 0, a = 0x0, c = 0, f = 0, i = 0, p = 0x0} , rR10 = {w = 0, a = 0x0, c = 0, f = 0, i = 0, p = 0x0} , rF1 = 0, rF2 = 0, rF3 = 0, rF4 = 0, rF5 = 0, rF6 = 0 , rD1 = 0, rD2 = 0, rD3 = 0, rD4 = 0, rD5 = 0, rD6 = 0 , rXMM1 = {h = 0, l = 0} , rXMM2 = {h = 0, l = 0} , rXMM3 = {h = 0, l = 0} , rXMM4 = {h = 0, l = 0} , rXMM5 = {h = 0, l = 0} , rXMM6 = {h = 0, l = 0} , rYMM1 = {h = {h = 0, l = 0}, l = {h = 0, l = 0}} , rYMM2 = {h = {h = 0, l = 0}, l = {h = 0, l = 0}} , rYMM3 = {h = {h = 0, l = 0}, l = {h = 0, l = 0}} , rYMM4 = {h = {h = 0, l = 0}, l = {h = 0, l = 0}} , rYMM5 = {h = {h = 0, l = 0}, l = {h = 0, l = 0}} , rYMM6 = {h = {h = 0, l = 0}, l = {h = 0, l = 0}} , rZMM1 = {h = {h = {h = 0, l = 0}, l = {h = 0, l = 0}}, l = {h = {h = 0, l = 0}, l = {h = 0, l = 0}}} , rZMM2 = {h = {h = {h = 0, l = 0}, l = {h = 0, l = 0}}, l = {h = {h = 0, l = 0}, l = {h = 0, l = 0}}} , rZMM3 = {h = {h = {h = 0, l = 0}, l = {h = 0, l = 0}}, l = {h = {h = 0, l = 0}, l = {h = 0, l = 0}}} , rZMM4 = {h = {h = {h = 0, l = 0}, l = {h = 0, l = 0}}, l = {h = {h = 0, l = 0}, l = {h = 0, l = 0}}} , rZMM5 = {h = {h = {h = 0, l = 0}, l = {h = 0, l = 0}}, l = {h = {h = 0, l = 0}, l = {h = 0, l = 0}}} , rZMM6 = {h = {h = {h = 0, l = 0}, l = {h = 0, l = 0}}, l = {h = {h = 0, l = 0}, l = {h = 0, l = 0}}} , rL1 = 0, rSp = 0x0, rSpLim = 0x0, rHp = 0x0, rHpLim = 0xb08fbfff , rCCCS = 0x0, rCurrentTSO = 0xb09cc8a4, rNursery = 0x109078 , rCurrentNursery = 0xb0801f60, rCurrentAlloc = 0xb0901980, rHpAlloc = 0, rRet = 3} (gdb) ptso $2 = {header = {info = 0xb2bf43c0 }, _link = 0xb2c146d8 , global_link = 0xb2c146d8 , stackobj = 0xb09cc4f4, what_next = 1, why_blocked = 0, flags = 0 , block_info = {closure = 0xb2c146d8 , prev = 0xb2c146d8 , bh = 0xb2c146d8 , throwto = 0xb2c146d8 , wakeup = 0xb2c146d8 , fd = -1295956264 } , id = 22, saved_errno = 0, dirty = 1 , bound = 0x0, cap = 0xb2c16180 <MainCapability> , trec = 0xb2c146d0 , blocked_exceptions = 0xb2c146d8 , bq = 0xb2c146d8 , alloc_limit = 2592, tot_stack_size = 232 } (gdb) info registers r0 0xb6f73018 3069653016 r1 0xb2bf434c 2998879052 r2 0x1 1 r3 0xb09cc7c6 2963064774 r4 0xb2c16190 2999017872 r5 0xb09cc81c 2963064860 r6 0xb08fbbf4 2962209780 r7 0xb09cc938 2963065144 r8 0xb14fefa0 2974805920 r9 0x0 0 r10 0xb2bf4160 2998878560 r11 0xb09cc558 2963064152 r12 0xb2c15b8c 2999016332 sp 0xb14fcd68 0xb14fcd68 lr 0x70000000 1879048192 pc 0xb88d09d4 0xb88d09d4 cpsr 0x400f0010 1074724880 }}} The `pc` register definitetly does contain a bad value, but there's no good explanation of how it got there. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:15 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+--------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 7.10.2 Component: GHCi | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+--------------------------------- Comment (by erikd): This is definitely not statically linked (the `lld` command shows dynamically linked Haskell libraries like `haskeline`, `terminfo`, `transformers` as well as the usual C library suspects. Debugging this in GDB is *really* difficult. Firstly, I can only recreate the problem when running with `--interactive`. Secondly, GDB and `ghc- stage2 --interactive` both try to take control of the same terminals so I have to start the `ghc-stage2 --interactive` process and connect gdb to it with `gdb -p $(pidof ghc-stage2)`. The `.gchi` file contains: {{{ putStr "Continue ... " _ <- getChar putStrLn "ok" data X = Y deriving Eq Y == Y }}} The `getChar` is there to pause the `ghc-stage2` process long enough to attach GDB. Unfortunately, this means that the number of passes through the code I'm trying to debug is not deterministic which makes setting a breakpoint rather difficult. I am currently playing around with replacing `getChar` with `threadDelay` to see if I can make it determinisitc. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:17 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+--------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 7.10.2 Component: GHCi | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+--------------------------------- Comment (by erikd): Added a `static int count=0` variable that gets incremented each time the `schedule` function passes through the `ThreadRunGHC` case statement. I then added a printf: {{{ printf ("%d : %u %u\n", count, cap->no, (unsigned)cap->r.rCurrentTSO->id); }}} and running it with the `getChar` in `.ghci` I note that the last set of values printed by this statement before it segfaults is: {{{ 183 : 0 22 }}} So I run it again and attached GDB when the process is waiting for `getChar` and then set a breakpoint: {{{ (gdb) break rts/Schedule.c:490 if (count == 183 && cap->no == 0 && cap->r.rCurrentTSO->id == 22) }}} only to have the process crash in a completely different way: {{{ Program received signal SIGILL, Illegal instruction. 0xb2c56c7a in schedule (initialCapability=0xb2c96180 <MainCapability>, task=0x10faf8) at rts/Schedule.c:491 (gdb) bt #0 0xb2c56c7a in schedule (initialCapability=0xb2c96180 <MainCapability>, task=0x10faf8) at rts/Schedule.c:491 #1 0xb2c59040 in scheduleWaitThread (tso=0xb2707000, ret=0x0, pcap=0xbeffe944) at rts/Schedule.c:2408 #2 0xb2c47c98 in rts_evalLazyIO (cap=0xbeffe944, p=0xd2c30 , ret=0x0) at rts/RtsAPI.c:500 #3 0xb2c5a0fc in real_main () at rts/RtsMain.c:63 #4 0xb2c5a1d0 in hs_main (argc=3, argv=0xbeffeb04, main_closure=0xd2c30 , rts_config=...) at rts/RtsMain.c:114 #5 0x000cd2c6 in main () (gdb) dis 0xb2c56c7a warning: bad breakpoint number at or near '0xb2c56c7a' }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:19 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

Using the gdb macros on the wiki I can retrieve the following info after

#10375: arm: ghci hits an illegal instruction -------------------------------------+--------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 7.10.2 Component: GHCi | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+--------------------------------- Comment (by carter): @erikd, your register snapshot doesnt make sense, it names x86_64 registers, namely XMM*, YMM* , and ZMM* families, but your code is on ARM right? Or am I misreading the dump? Replying to [comment:15 erikd]: the segfault:

{{{ Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb14ff460 (LWP 7214)] 0xb88d09d4 in ?? () (gdb) bt #0 0xb88d09d4 in ?? () #1 0x70000000 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt

stack?)

(gdb) pregs $1 = {rR1 = {w = 0, a = 0x0, c = 0, f = 0, i = 0, p = 0x0} ,rR2 = {w = 0, a = 0x0, c = 0, f = 0, i = 0, p = 0x0} , rR3 = {w = 0, a = 0x0, c = 0, f = 0, i = 0, p = 0x0} , rR4 = {w = 0, a = 0x0, c = 0, f = 0, i = 0, p = 0x0} , rR5 = {w = 0, a = 0x0, c = 0, f = 0, i = 0, p = 0x0} , rR6 = {w = 0, a = 0x0, c = 0, f = 0, i = 0, p = 0x0} , rR7 = {w = 0, a = 0x0, c = 0, f = 0, i = 0, p = 0x0} , rR8 = {w = 0, a = 0x0, c = 0, f = 0, i = 0, p = 0x0} , rR9 = {w = 0, a = 0x0, c = 0, f = 0, i = 0, p = 0x0} , rR10 = {w = 0, a = 0x0, c = 0, f = 0, i = 0, p = 0x0} , rF1 = 0, rF2 = 0, rF3 = 0, rF4 = 0, rF5 = 0, rF6 = 0 , rD1 = 0, rD2 = 0, rD3 = 0, rD4 = 0, rD5 = 0, rD6 = 0 , rXMM1 = {h = 0, l = 0} , rXMM2 = {h = 0, l = 0} , rXMM3 = {h = 0, l = 0} , rXMM4 = {h = 0, l = 0} , rXMM5 = {h = 0, l = 0} , rXMM6 = {h = 0, l = 0} , rYMM1 = {h = {h = 0, l = 0}, l = {h = 0, l = 0}} , rYMM2 = {h = {h = 0, l = 0}, l = {h = 0, l = 0}} , rYMM3 = {h = {h = 0, l = 0}, l = {h = 0, l = 0}} , rYMM4 = {h = {h = 0, l = 0}, l = {h = 0, l = 0}} , rYMM5 = {h = {h = 0, l = 0}, l = {h = 0, l = 0}} , rYMM6 = {h = {h = 0, l = 0}, l = {h = 0, l = 0}} , rZMM1 = {h = {h = {h = 0, l = 0}, l = {h = 0, l = 0}}, l = {h = {h = 0, l = 0}, l = {h = 0, l = 0}}} , rZMM2 = {h = {h = {h = 0, l = 0}, l = {h = 0, l = 0}}, l = {h = {h = 0, l = 0}, l = {h = 0, l = 0}}} , rZMM3 = {h = {h = {h = 0, l = 0}, l = {h = 0, l = 0}}, l = {h = {h = 0, l = 0}, l = {h = 0, l = 0}}} , rZMM4 = {h = {h = {h = 0, l = 0}, l = {h = 0, l = 0}}, l = {h = {h = 0, l = 0}, l = {h = 0, l = 0}}} , rZMM5 = {h = {h = {h = 0, l = 0}, l = {h = 0, l = 0}}, l = {h = {h = 0, l = 0}, l = {h = 0, l = 0}}} , rZMM6 = {h = {h = {h = 0, l = 0}, l = {h = 0, l = 0}}, l = {h = {h = 0, l = 0}, l = {h = 0, l = 0}}} , rL1 = 0, rSp = 0x0, rSpLim = 0x0, rHp = 0x0, rHpLim = 0xb08fbfff , rCCCS = 0x0, rCurrentTSO = 0xb09cc8a4, rNursery = 0x109078 , rCurrentNursery = 0xb0801f60, rCurrentAlloc = 0xb0901980, rHpAlloc = 0, rRet = 3} (gdb) ptso $2 = {header = {info = 0xb2bf43c0 }, _link = 0xb2c146d8 , global_link = 0xb2c146d8 , stackobj = 0xb09cc4f4, what_next = 1, why_blocked = 0, flags = 0 , block_info = {closure = 0xb2c146d8 , prev = 0xb2c146d8 , bh = 0xb2c146d8 , throwto = 0xb2c146d8 , wakeup = 0xb2c146d8 , fd = -1295956264 } , id = 22, saved_errno = 0, dirty = 1 , bound = 0x0, cap = 0xb2c16180 <MainCapability> , trec = 0xb2c146d0 , blocked_exceptions = 0xb2c146d8 , bq = 0xb2c146d8 , alloc_limit = 2592, tot_stack_size = 232 } (gdb) info registers r0 0xb6f73018 3069653016 r1 0xb2bf434c 2998879052 r2 0x1 1 r3 0xb09cc7c6 2963064774 r4 0xb2c16190 2999017872 r5 0xb09cc81c 2963064860 r6 0xb08fbbf4 2962209780 r7 0xb09cc938 2963065144 r8 0xb14fefa0 2974805920 r9 0x0 0 r10 0xb2bf4160 2998878560 r11 0xb09cc558 2963064152 r12 0xb2c15b8c 2999016332 sp 0xb14fcd68 0xb14fcd68 lr 0x70000000 1879048192 pc 0xb88d09d4 0xb88d09d4 cpsr 0x400f0010 1074724880

}}}

The `pc` register definitetly does contain a bad value, but there's no good explanation of how it got there.

-- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:21 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+--------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 7.10.2 Component: GHCi | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+--------------------------------- Comment (by erikd): By hacking `mk/config.mk.in` I was able to compile with `DYNAMIC_GHC_PROGRAMS` set to `NO` and create a `ghc-stage2` compiler that dynamically links only to the C libraries. It also crashes just in a different way. Regardless of whether a `.ghci` file exists, `ghc-stage2 --interactive` crashes with `Illegal instruction` as soon as it is run and before it even diplays a prompt: {{{ $ inplace/bin/ghc-stage2 --interactive GHCi, version 7.11.20150523: http://www.haskell.org/ghc/ :? for help Illegal instruction }}} Checking the settings file, I find that the `ld.gold` linker *is* being used. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:23 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+--------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 7.10.2 Component: GHCi | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+--------------------------------- Comment (by erikd): By hacking `configure.ac` I managed to build GHC with `-fsanitize=address` passed to GCC. Absolutely, positively, 100% certain that something is wrong: {{{ $ inplace/bin/ghc-stage2 --interactive ASAN:SIGSEGV ================================================================= ==7051==ERROR: AddressSanitizer: SEGV on unknown address 0x00000000 (pc 0x00000000 sp 0xbeffe5f0 bp 0x37dffd15 T0) AddressSanitizer can not provide additional info. SUMMARY: AddressSanitizer: SEGV ??:0 ?? }}} `ghc-stage2` compiled with AddressSanitizer on `x86_64/linux` does not do the above. Unfortunately, while AddressSanitizer has found a problem, it doesn't provide any new information. I suspect it doesn't know enough about GHC's run time and calling conventions. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:25 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+--------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 7.10.2 Component: GHCi | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+--------------------------------- Comment (by erikd): I smell a rat, or maybe its a red herring. Seem to be getting SIGSEGV inside the call to `pthread_cond_init`. A bit of googling tells me that AddressSanitizer intercepts calls to `pthread_cond_init` and can get it wrong eg : https://code.google.com/p/address- sanitizer/issues/detail?id=297 . Sure enough, trivial C program: {{{ #include #include int main (void) { pthread_cond_t cond; pthread_cond_init(&cond, NULL); return 0; } }}} Works correctly when compiled and run (even under Valgrind) as: {{{ gcc -Wall cond_init.c -o cond_init && valgrind ./cond_init }}} but crashes with a SIGSEGV when compiled and run as: {{{ gcc -Wall -fsanitize=address cond_init.c -o cond_init && ./cond_init }}} Reported this as a bug in Debian's gcc : https://bugs.debian.org/cgi- bin/bugreport.cgi?bug=786850 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:27 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+--------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 7.10.2 Component: GHCi | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+--------------------------------- Comment (by erikd): Got GHC compiling with `gcc-5 -fsanitize=address` and AddressSanitizer did not find anything to complain about, but `ghc-stage2` still crashes with either a segfault or an illegal instruction depending on where it jumps to. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:29 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+--------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.2 Component: GHCi | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+--------------------------------- Changes (by erikd): * priority: normal => high -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:31 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+--------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.3 Component: GHCi | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+--------------------------------- Comment (by erikd): The first shipped version of GHC with GHCI support for ARM was ghc-7.8. I just compiled ghc-7.8.4 on ARM and the resulting GHCi fails just like it does for ghc-7.10 and ghc-7.11. For ghc-7.11 (git master) I have tried: * `DYNAMIC_GHC_PROGRAMS` set to both `YES` and `NO` and have the same crash either way. * `Unregisterised` set to `YES` and to `NO`. Both crash in the same way. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:34 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+--------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.3 Component: GHCi | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+--------------------------------- Comment (by Ansible): Just wanted to point out that on raspberry pi 2 with debian jessie, ghc and ghci are both broken. I tried the ghc binary from the ghc site, and it fails with the "Planet" test (from the top of this thread) in ghci, and in compiled code it fails "hello world". So its pretty much a show stopper for ARM. {{{ GHCi, version 7.10.2: http://www.haskell.org/ghc/ :? for help Prelude> data Planet = Mercury | Venus deriving Eq Prelude> Mercury == Mercury Illegal instruction bburdette@jessie-rpi:~$ And my hello world program: main = do putStrLn "hello" The results: bburdette@jessie-rpi:~$ ghc hello.hs [1 of 1] Compiling Main ( hello.hs, hello.o ) Linking hello ... bburdette@jessie-rpi:~$ ls bin ghc-7.10.2-arm-unknown-linux.tar.xz hello.hi hello.o code hello hello.hs bburdette@jessie-rpi:~$ ./hello Illegal instruction bburdette@jessie-rpi:~$ ghc -O2 hello.hs bburdette@jessie-rpi:~$ ./hello Illegal instruction bburdette@jessie-rpi:~$ }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:36 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+--------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.3 Component: GHCi | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+--------------------------------- Comment (by Ansible): I did have the gold linker enabled when I did that test. since then I've gone back to arch and 7.8.2. ghc --info output is below (from debian jesse on raspberry pi 2): {{{ [("Project name","The Glorious Glasgow Haskell Compilation System") ,("GCC extra via C opts"," -fwrapv") ,("C compiler command","/usr/bin/gcc") ,("C compiler flags"," -fno-stack-protector") ,("C compiler link flags"," -fuse-ld=gold -Wl,-z,noexecstack") ,("Haskell CPP command","/usr/bin/gcc") ,("Haskell CPP flags","-E -undef -traditional ") ,("ld command","/usr/bin/ld.gold") ,("ld flags"," -z noexecstack") ,("ld supports compact unwind","YES") ,("ld supports build-id","YES") ,("ld supports filelist","NO") ,("ld is GNU ld","YES") ,("ar command","/usr/bin/ar") ,("ar flags","q") ,("ar supports at file","YES") ,("touch command","touch") ,("dllwrap command","/bin/false") ,("windres command","/bin/false") ,("libtool command","libtool") ,("perl command","/usr/bin/perl") ,("cross compiling","NO") ,("target os","OSLinux") ,("target arch","ArchARM {armISA = ARMv7, armISAExt = [VFPv3,NEON], armABI = HARD}") ,("target word size","4") ,("target has GNU nonexec stack","False") ,("target has .ident directive","True") ,("target has subsections via symbols","False") ,("Unregisterised","NO") ,("LLVM llc command","/usr/bin/llc-3.5") ,("LLVM opt command","/usr/bin/opt-3.5") ,("Project version","7.10.2") ,("Project Git commit id","0da488c4438d88c9252e0b860426b8e74b5fc9e8") ,("Booter version","7.6.3") ,("Stage","2") ,("Build platform","arm-unknown-linux") ,("Host platform","arm-unknown-linux") ,("Target platform","arm-unknown-linux") ,("Have interpreter","YES") ,("Object splitting supported","NO") ,("Have native code generator","NO") ,("Support SMP","YES") ,("Tables next to code","YES") ,("RTS ways","l debug thr thr_debug thr_l thr_p dyn debug_dyn thr_dyn thr_debug_dyn l_dyn thr_l_dyn") ,("Support dynamic-too","YES") ,("Support parallel --make","YES") ,("Support reexported-modules","YES") ,("Support thinning and renaming package flags","YES") ,("Uses package keys","YES") ,("Dynamic by default","NO") ,("GHC Dynamic","YES") ,("Leading underscore","NO") ,("Debug on","False") ,("LibDir","/usr/local/lib/ghc-7.10.2") ,("Global Package DB","/usr/local/lib/ghc-7.10.2/package.conf.d") ] }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:38 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+--------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.3 Component: GHCi | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+--------------------------------- Comment (by erikd): If I configure the build with `--enable-unregistered`, built it and then re-ran my tests. The program still crashes with SIGILL, but I now get a sensible backtrace: {{{ GHCi, version 7.11.20150905: http://www.haskell.org/ghc/ :? for help Program received signal SIGILL, Illegal instruction. 0x03ff9dbc in stg_ap_v_fast () (gdb) bt #0 0x03ff9dbc in stg_ap_v_fast () #1 0x03fc6ce6 in StgRun (f=0x3ff9db4 , basereg=0x49d2090 ) at rts/StgCRun.c:81 #2 0x03fca52a in schedule (initialCapability=0x49d2080 <MainCapability>, task=0x49e62c0) at rts/Schedule.c:524 #3 0x03fcc5e6 in scheduleWaitThread (tso=0xb6c07000, ret=0x0, pcap=0xbeffeb74) at rts/Schedule.c:2429 #4 0x03fbc7e4 in rts_evalLazyIO (cap=0xbeffeb74, p=0x402d470 , ret=0x0) at rts/RtsAPI.c:500 #5 0x03fce2be in hs_main (argc=3, argv=0xbeffed64, main_closure=0x402d470 , rts_config=...) at rts/RtsMain.c:64 #6 0x00141508 in main () }}} and if I disassemble at the SIGILL I get: {{{ (gdb) disassemble Dump of assembler code for function stg_ap_v_fast: 0x03ff9db4 <+0>: push {r7, lr} 0x03ff9db6 <+2>: sub sp, #32 0x03ff9db8 <+4>: add r7, sp, #0 0x03ff9dba <+6>: ldr r3, [pc, #508] ; (0x3ff9fb8 ) => 0x03ff9dbc <+8>: ldr r3, [r3, #0] 0x03ff9dbe <+10>: and.w r3, r3, #3 }}} Now to figure out why that load instruction is illegal. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:40 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+--------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.3 Component: GHCi | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+--------------------------------- Comment (by erikd): Things to note: * Register `R0` is the argument/result/scratch register. * `stg_ap_v_fast` is generated CMM code generated by `utils/genapply`. * Since this is on Arm, the CMM code is passed throught the LLVM backend. Confirming that we are actually generating invalid Arm assembly: {{{ utils/genapply/dist/build/tmp/genapply > suspect.cmm inplace/bin/ghc-stage2 -Irts -c suspect.cmm -o suspect.o }}} and then `objdump -D suspect.o` and searching for `stg-ap_v_fast` results in: {{{ 0000853c : 853c: b480 push {r7} 853e: b087 sub sp, #28 8540: af00 add r7, sp, #0 8542: 4b76 ldr r3, [pc, #472] ; (871c ) 8544: 681b ldr r3, [r3, #0] }}} which is exactly what we found in GDB. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:42 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+--------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.3 Component: GHCi | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+--------------------------------- Comment (by erikd): Got some help from the llvm-dev mailing list. People there pointed out that this: {{{ (gdb) disassemble Dump of assembler code for function stg_ap_v_fast: 0x03ff9db4 <+0>: push {r7, lr} 0x03ff9db6 <+2>: sub sp, #32 0x03ff9db8 <+4>: add r7, sp, #0 0x03ff9dba <+6>: ldr r3, [pc, #508] ; (0x3ff9fb8 ) => 0x03ff9dbc <+8>: ldr r3, [r3, #0] 0x03ff9dbe <+10>: and.w r3, r3, # }}} show the CPU executing Thumb instructions. Thumb instructions are encoded in two btyes (and hence instructions with addresses that are even) while Arm instructions are encoded in 4 bytes (so instructions should be aligned at 4 bytes offsets). This could be that GDB getting the CPU mode incorrect. Not sure how to figure out if that is the case or even if and how that could happen. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:44 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+------------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.3 Component: Runtime System | Version: 7.10.1 (Linker) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by erikd): Readig more from #2972, I see that all the comments about PowerPC being limited to 24 bit offsets for relative branch instructions should also apply to Arm. However, enabling `USE_CONTIGUOUS_MMAP` in `rts/linker.c` for `arm_HOST_ARCH` as well as `powerpc_HOST_ARCH` doesn't fix this issue. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:46 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+------------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.3 Component: Runtime System | Version: 7.10.1 (Linker) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by erikd): Since I know that this is some problem with the function `hSetBuffering` when it is being loaded by the runtime linker I added a printf to `rts/Linker.c` to print the address of the `hSetBuffering` related functions when they are loaded. Running this now looks like: {{{ $ inplace/bin/ghc-stage2 --interactive GHCi, version 7.11.20150912: http://www.haskell.org/ghc/ :? for help lookupSymbol: value of base_GHCziIOziHandle_hSetBuffering1_entry is 0xb3f3d3cd lookupSymbol: value of base_GHCziIOziHandle_hSetBuffering1_entry is 0xb3f3d3cd lookupSymbol: value of base_GHCziIOziHandle_hSetBuffering1_info is 0xb421f738 lookupSymbol: value of base_GHCziIOziHandle_hSetBuffering1_closure is 0xb421d5b0 lookupSymbol: value of base_GHCziIOziHandle_hSetBuffering_entry is 0xb3f3d471 lookupSymbol: value of base_GHCziIOziHandle_hSetBuffering_info is 0xb421f718 lookupSymbol: value of base_GHCziIOziHandle_hSetBuffering1_entry is 0xb3f3d3cd turnOffBuffering start turnOffBuffering middle Illegal instruction }}} If I now load this into gdb, run it until it crashes I get: {{{ GHCi, version 7.11.20150912: http://www.haskell.org/ghc/ :? for help lookupSymbol: value of base_GHCziIOziHandle_hSetBuffering1_entry is 0xb3f503cd lookupSymbol: value of base_GHCziIOziHandle_hSetBuffering1_entry is 0xb3f503cd lookupSymbol: value of base_GHCziIOziHandle_hSetBuffering1_info is 0xb4232738 lookupSymbol: value of base_GHCziIOziHandle_hSetBuffering1_closure is 0xb42305b0 lookupSymbol: value of base_GHCziIOziHandle_hSetBuffering_entry is 0xb3f50471 lookupSymbol: value of base_GHCziIOziHandle_hSetBuffering_info is 0xb4232718 lookupSymbol: value of base_GHCziIOziHandle_hSetBuffering1_entry is 0xb3f503cd turnOffBuffering start turnOffBuffering middle Program received signal SIGILL, Illegal instruction. 0x04004e5c in stg_ap_v_fast () (gdb) disass base_GHCziIOziHandle_hSetBuffering1_entry Dump of assembler code for function base_GHCziIOziHandle_hSetBuffering1_entry: 0x03d423c4 <+0>: movw r3, #58688 ; 0xe540 0x03d423c8 <+4>: movt r3, #1181 ; 0x49d 0x03d423cc <+8>: ldr.w r2, [r3, #808] ; 0x328 0x03d423d0 <+12>: subs r2, #4 }}} For some reason, the address of `base_GHCziIOziHandle_hSetBuffering1_entry` in GDB is not the address printed by the runtime linker. Even more confusingly, disassembling the address the `0xb3f3d3cd` fails, possibly because its outside the memory map. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:48 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+------------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.3 Component: Runtime System | Version: 7.10.1 (Linker) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by erikd): According to `readelf -S` on the object file the sections are: {{{ Section Headers: [Nr] Name Type Addr Off Size ES Flg Lk Inf Al [ 0] NULL 00000000 000000 000000 00 0 0 0 [ 1] .text PROGBITS 00000000 000038 4c7518 00 AX 0 0 8 [ 2] .data PROGBITS 00000000 4c7550 111ab8 00 WA 0 0 8 [ 3] .bss NOBITS 00000000 5d9008 000000 00 WA 0 0 4 [ 4] .init_array INIT_ARRAY 00000000 5d9008 000004 00 WA 0 0 4 [ 5] .rodata.str1.4 PROGBITS 00000000 5d900c 00000e 01 AMS 0 0 4 [ 6] .rodata PROGBITS 00000000 5d901c 00dd40 00 A 0 0 4 [ 7] .comment PROGBITS 00000000 5e6d5c 00001d 01 MS 0 0 1 [ 8] .rel.text REL 00000000 5e6d7c 328af8 08 I 14 1 4 [ 9] .rel.data REL 00000000 90f874 0cab90 08 I 14 2 4 [10] .rel.init_array REL 00000000 9da404 000008 08 I 14 4 4 [11] .rel.rodata REL 00000000 9da40c 008820 08 I 14 6 4 [12] .note.GNU-stack PROGBITS 00000000 9e2c2c 000000 00 0 0 0 [13] .ARM.attributes ARM_ATTRIBUTES 00000000 9e2c2c 000035 00 0 0 1 [14] .symtab SYMTAB 00000000 9e2c64 29dc00 10 15 122623 4 [15] .strtab STRTAB 00000000 c80864 2dd2e5 00 0 0 1 [16] .shstrtab STRTAB 00000000 f5db49 000088 00 0 0 1 Key to Flags: W (write), A (alloc), X (execute), M (merge), S (strings) I (info), L (link order), G (group), T (TLS), E (exclude), x (unknown) O (extra OS processing required) o (OS specific), p (processor specific) }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:50 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+------------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.3 Component: Runtime System | Version: 7.10.1 (Linker) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by erikd): Tried re-producing this segfault with `--enable-unregisterised` mainly because the Unregisterised mode can be debugged in GDB. Unfortunately, the segfault doesn't happen in Unregisterised mode. Going back to registerised mode. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:52 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+------------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.3 Component: Runtime System | Version: 7.10.1 (Linker) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by erikd): Turns out the memory region its trying to execute is not actually code. Hex dump is: {{{ b6ff0000: 0000 0000 3300 0000 0800 ffb6 0800 0000 ....3........... b6ff0010: 0000 0100 0100 0000 4900 4708 f0cd e601 ........I.G..... b6ff0020: 696e 7465 7261 6374 6976 653a 4768 6369 interactive:Ghci b6ff0030: 312e 5800 3300 0000 3800 ffb6 0800 0000 1.X.3...8....... b6ff0040: 0000 0100 0100 0100 4900 4708 f0cd e601 ........I.G..... b6ff0050: 696e 7465 7261 6374 6976 653a 4768 6369 interactive:Ghci b6ff0060: 312e 5900 790f 0000 6800 ffb6 0f00 2de9 1.Y.y...h.....-. b6ff0070: 0000 9fe5 00f0 9fe5 6c00 ffb6 cceb e801 ........l....... }}} and the instruction that segfaults is at 0xb6ff0024. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:54 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+------------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.3 Component: Runtime System | Version: 7.10.1 (Linker) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by erikd): Results of previous experiments: * On ghc-7.20 branch, GHCi with Thumb code disabled still crashes. * On master branch, with SMP disabled, GHCi still crashes. Should probably continue working on master branch with SMP disabled. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:56 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+------------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.3 Component: Runtime System | Version: 7.10.1 (Linker) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by int-e): Replying to [comment:57 erikd]:

I'm not sure why GDB thinks this is within the `strlen` function. If I disassemble `strlen` I get the same assembler instructions, but at a different address:

That looks like there are several copies of `strlen` in the code, including symbol information. So the function name is found when looking for the symbol closest to `0xb6fe88ae`, but when resolving the name to an address, a different copy is used instead. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:58 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+------------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.3 Component: Runtime System | Version: 7.10.1 (Linker) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by erikd): Debugging with GDB was too un-reliable, even with Virtual Address randomization disabled in the Linux kernel and master tick timer disabled (`+RTS -V0 -RTS`). However, running with `+RTS -Da -RTS` I noticed that one of the last things the program did before crashing was to print out a `BCO` (Bycode compiled object). Unfortunately, the output with `-Ds` was simply way too verbose. Adding some custom debug which was only activated after the BCO was loaded showed that the crash (sometimes SIGILL, sometimes SIGSEGV) was happening some time after the call to `StgRun` in `rts/Schedule.c`. Since, the crash after calling `StgRun` happens immediately after loading the BCO, my suspicion is that the `StgRun` call is actually calling the recently loaded BCO. My custom debug shows something along the lines of (eliding irrelevant stuff): {{{ StgRun call 128 StgRun call 129 StgRun call 130 BCO 0: PUSH_ALTS 0xb39c2d08 2: PUSH_L 2 4: ENTER INSTRS: 6 0 2 2 46 PTRS: 0xb39c2d08 SthRun call 131 Segmentation fault }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:60 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction --------------------------------------------+------------------------------ Reporter: erikd | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.3 Component: Runtime System (Linker) | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): --------------------------------------------+------------------------------ Comment (by erikd): Tried once again to debug this using GDB. {{{ $ gdb --args inplace/bin/ghc-stage2 +RTS -V0 -RTS --interactive (gdb) break main (gdb) break main Breakpoint 1 at 0x93cd8 (gdb) r Starting program: /home/erikd/Git/ghc-upstream/inplace/lib/bin/ghc-stage2 -B/home/erikd/Git/ghc-upstream/inplace/lib +RTS -V0 -RTS --interactive [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/arm-linux- gnueabihf/libthread_db.so.1". Breakpoint 1, 0x00093cd8 in main () (gdb) while 1

x/i $pc stepi end }}}

and it crashes in a at least 3 or 4 different ways on different runs, none of them even remotely like the way it crashes outside GDB. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:62 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+------------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.3 Component: Runtime System | Version: 7.10.1 (Linker) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D1323 Wiki Page: | -------------------------------------+------------------------------------- Comment (by erikd): Haven't run the full test suite yet, but running: {{{ make test TEST="print018 print020 print021 print022 print025 ghci053 ghcirun001 ghcirun002" }}} which used to result in 8 failing tests, now results in 8 passing tests. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:64 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+------------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.3 Component: Runtime System | Version: 7.10.1 (Linker) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D1323 Wiki Page: | -------------------------------------+------------------------------------- Comment (by Erik de Castro Lopo ): In [changeset:"933adc0f31164cb651d11ecfcfe612ac429f714f/ghc" 933adc0f/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="933adc0f31164cb651d11ecfcfe612ac429f714f" Fix GHCi on Arm (#10375). Arm has two instruction sets, Arm and Thumb, and an execution mode for each. Executing Arm code in Thumb mode or vice-versa will likely result in an Illegal instruction exception. Furthermore, Haskell code compiled via LLVM was generating Arm instructions while C code compiled via GCC was generating Thumb code by default. When these two object code types were being linked by the system linker, all was fine, because the system linker knows how to jump and call from one instruction set to the other. The first problem was with GHCi's object code loader which did not know about Thumb vs Arm. When loading an object file `StgCRun` would jump into the loaded object which could change the mode causing a crash after it returned. This was fixed by forcing all C code to generate Arm instructions by passing `-marm` to GCC. The second problem was the `mkJumpToAddr` function which was generating Thumb instructions. Changing that to generate Arm instructions instead results in a working GHCi on Arm. Test Plan: validate on x86_64 and arm Reviewers: bgamari, austin, hvr Subscribers: thomie Differential Revision: https://phabricator.haskell.org/D1323 GHC Trac Issues: #10375 }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:66 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10375: arm: ghci hits an illegal instruction -------------------------------------+------------------------------------- Reporter: erikd | Owner: Type: bug | Status: merge Priority: high | Milestone: 7.10.3 Component: Runtime System | Version: 7.10.1 (Linker) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: arm Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: #10969 | Differential Rev(s): Phab:D1323 Wiki Page: | -------------------------------------+------------------------------------- Changes (by erikd): * related: => #10969 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10375#comment:68 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler