Fwd: Segmentation fault in non-dynamically linked binaries?

Forwarding this to g-h-u for archival purposes and in case anybody else has ideas (should'a done this earlier.) ---------- Forwarded message ---------- From: austin seipp Date: Wed, Jun 16, 2010 at 11:21 AM Subject: Re: Segmentation fault in non-dynamically linked binaries? To: Simon Marlow Follow up: Both of my machines are running debian x86_64 linux, both with libc6 version 2.11.2-1 (aka eglibc,) from debian sid. My work machine does not exhibit the same segfaulting behavior as my home machine. Debugging with the rts source available and watching setFullProgArgv go by does not help a lot - when stgCallocBytes is called with strlen as a parameter (to determine how much to allocate,) the code immediately crashes upon arrival into the first instruction of strlen, but the strange thing is *which* strlen it is jumping to.

From a simple 'hello world' executable on my work machine, when disassembling setFullProgArgv we get this:

... 0x0000000000432a26 :        xor    %ebp,%ebp 0x0000000000432a28 :        nopl   0x0(%rax,%rax,1) 0x0000000000432a30 :        mov    (%r14,%rbp,1),%rdi 0x0000000000432a34 :        mov    %rbp,%rbx 0x0000000000432a37 :        add 0x247f5a(%rip),%rbx        # 0x67a998 0x0000000000432a3e :        add    $0x1,%r12d 0x0000000000432a42 :        callq  0x4028b8 0x0000000000432a47 :        lea    0x1(%rax),%edi 0x0000000000432a4a :        mov    $0x4616fe,%esi 0x0000000000432a4f :        callq  0x434790 <stgMallocBytes> 0x0000000000432a54 :       mov    %rax,(%rbx) 0x0000000000432a57 :       mov    0x247f3a(%rip),%rax ... Note that it calls strlen@plt which jumps to libc's implementation of strlen. However, when looking at this same program compiled on my home machine, we instead get: ... 0x0000000000432db6 <+54>:       xor    %ebp,%ebp 0x0000000000432db8 <+56>:       nopl   0x0(%rax,%rax,1) 0x0000000000432dc0 <+64>:       mov    (%r14,%rbp,1),%rdi 0x0000000000432dc4 <+68>:       mov    %rbp,%rbx 0x0000000000432dc7 <+71>:       add    0x24898a(%rip),%rbx        # 0x67b758 0x0000000000432dce <+78>:       add    $0x1,%r12d 0x0000000000432dd2 <+82>:       callq  0x67b3c0 0x0000000000432dd7 <+87>:       lea    0x1(%rax),%edi 0x0000000000432dda <+90>:       mov    $0x46207e,%esi 0x0000000000432ddf <+95>:       callq  0x434cb0 <stgMallocBytes> 0x0000000000432de4 <+100>:      mov    %rax,(%rbx) 0x0000000000432de7 <+103>:      mov    0x24896a(%rip),%rax        # 0x67b758 ... Here we call a version of strlen which the dynamic linker enforces come from glibc v2.2.5. If we run objdump -x, we get: ... Version References:  required from librt.so.1:    0x09691973 0x00 04 GLIBC_2.3.3  required from libm.so.6:    0x09691a75 0x00 03 GLIBC_2.2.5  required from libc.so.6:    0x0d696913 0x00 05 GLIBC_2.3    0x09691a75 0x00 02 GLIBC_2.2.5 ... So there are some dependencies on older GLIBC versions for certain symbols, but I'm not sure why. I'm at work agaom so I'll follow up again here in a few hours once I get back home after doing some more reading. On Tue, Jun 15, 2010 at 3:43 PM, austin seipp wrote:

Simon,

Thanks for the reply.

One interesting thing to note is that on my work machine, also running x86_64 debian, with with a full update from apt (many updates come from sid, others are from lenny,) static and dynamic linking both seem to work fine for the same versions of GHC that affect my home machine.

On my debian machine here at work I have a libc6 which is installed from the unstable sid repository. It is libc6 version '2.11-1', which is actually a version of eglibc, not glibc (although eglibc is designed for full ABI-level compatibility - debian switched to it a while ago.) I believe my home machine with x86_64 debian has the exact same version of libc6 installed out of sid repos but I cannot confirm that at the moment.

I do not currently have access to my home machine from here. When I do I'll get a copy of the 6.12.1 source code and do a debug link and do some more investigation.

Thanks,

On Tue, Jun 15, 2010 at 5:33 AM, Simon Marlow wrote:

...

On 13/06/2010 07:26, austin seipp wrote:

...

Hello,

I am running GHC on x86_64 debian linux, and recently I have discovered that the executables generated by my GHC segfault when the linking step is not dynamic. I discovered this while attempting to install haskell-src-exts, which requires a linked version of Setup.hs when cabal builds it (and which would fail inexplicably until I did further investigation.)

Example:

link ~/t » cat hi.hs main :: IO () main = putStrLn "hi" link ~/t » ghc -dynamic hi.hs link ~/t » file ./a.out ./a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped link ~/t » ldd ./a.out        linux-vdso.so.1 =>   (0x00007fffbadff000)        libHShaskell98-1.0.1.1-ghc6.12.3.so =>

/usr/local/lib/ghc-6.12.3/haskell98-1.0.1.1/libHShaskell98-1.0.1.1-ghc6.12.3.so (0x00007fab3a4c4000)        libHSrandom-1.0.0.2-ghc6.12.3.so => /usr/local/lib/ghc-6.12.3/random-1.0.0.2/libHSrandom-1.0.0.2-ghc6.12.3.so (0x00007fab3a2af000)        libHStime-1.1.4-ghc6.12.3.so => /usr/local/lib/ghc-6.12.3/time-1.1.4/libHStime-1.1.4-ghc6.12.3.so (0x00007fab39fad000)        libHSprocess-1.0.1.3-ghc6.12.3.so =>

/usr/local/lib/ghc-6.12.3/process-1.0.1.3/libHSprocess-1.0.1.3-ghc6.12.3.so (0x00007fab39d93000)        libHSdirectory-1.0.1.1-ghc6.12.3.so =>

/usr/local/lib/ghc-6.12.3/directory-1.0.1.1/libHSdirectory-1.0.1.1-ghc6.12.3.so (0x00007fab39b77000)        libHSunix-2.4.0.2-ghc6.12.3.so => /usr/local/lib/ghc-6.12.3/unix-2.4.0.2/libHSunix-2.4.0.2-ghc6.12.3.so (0x00007fab398c6000)        librt.so.1 =>  /lib/librt.so.1 (0x00007fab396a2000)        libutil.so.1 =>  /lib/libutil.so.1 (0x00007fab3949f000)        libdl.so.2 =>  /lib/libdl.so.2 (0x00007fab3929b000)        libHSold-time-1.0.0.5-ghc6.12.3.so =>

/usr/local/lib/ghc-6.12.3/old-time-1.0.0.5/libHSold-time-1.0.0.5-ghc6.12.3.so (0x00007fab3903c000)        libHSold-locale-1.0.0.2-ghc6.12.3.so =>

/usr/local/lib/ghc-6.12.3/old-locale-1.0.0.2/libHSold-locale-1.0.0.2-ghc6.12.3.so (0x00007fab38e28000)        libHSfilepath-1.1.0.4-ghc6.12.3.so =>

/usr/local/lib/ghc-6.12.3/filepath-1.1.0.4/libHSfilepath-1.1.0.4-ghc6.12.3.so (0x00007fab38c07000)        libHSarray-0.3.0.1-ghc6.12.3.so => /usr/local/lib/ghc-6.12.3/array-0.3.0.1/libHSarray-0.3.0.1-ghc6.12.3.so (0x00007fab38992000)        libHSbase-4.2.0.2-ghc6.12.3.so => /usr/local/lib/ghc-6.12.3/base-4.2.0.2/libHSbase-4.2.0.2-ghc6.12.3.so (0x00007fab381f2000)        libHSinteger-gmp-0.2.0.1-ghc6.12.3.so =>

/usr/local/lib/ghc-6.12.3/integer-gmp-0.2.0.1/libHSinteger-gmp-0.2.0.1-ghc6.12.3.so (0x00007fab37fe1000)        libgmp.so.3 =>  /usr/lib/libgmp.so.3 (0x00007fab37da1000)        libHSghc-prim-0.2.0.0-ghc6.12.3.so =>

/usr/local/lib/ghc-6.12.3/ghc-prim-0.2.0.0/libHSghc-prim-0.2.0.0-ghc6.12.3.so (0x00007fab37b1c000)        libHSrts-ghc6.12.3.so => /usr/local/lib/ghc-6.12.3/libHSrts-ghc6.12.3.so (0x00007fab378ba000)        libm.so.6 =>  /lib/libm.so.6 (0x00007fab37637000)        libHSffi-ghc6.12.3.so => /usr/local/lib/ghc-6.12.3/libHSffi-ghc6.12.3.so (0x00007fab3742a000)        libc.so.6 =>  /lib/libc.so.6 (0x00007fab370c9000)        libpthread.so.0 =>  /lib/libpthread.so.0 (0x00007fab36eac000)        /lib64/ld-linux-x86-64.so.2 (0x00007fab3a6cb000) link ~/t » ./a.out hi link ~/t » rm ./a.out *.hi *.o link ~/t » ghc hi.hs link ~/t » file ./a.out ./a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped link ~/t » ldd ./a.out        linux-vdso.so.1 =>   (0x00007fffaafff000)        librt.so.1 =>  /lib/librt.so.1 (0x00007fdf83d77000)        libutil.so.1 =>  /lib/libutil.so.1 (0x00007fdf83b74000)        libdl.so.2 =>  /lib/libdl.so.2 (0x00007fdf8396f000)        libgmp.so.3 =>  /usr/lib/libgmp.so.3 (0x00007fdf83730000)        libm.so.6 =>  /lib/libm.so.6 (0x00007fdf834ae000)        libc.so.6 =>  /lib/libc.so.6 (0x00007fdf8314c000)        libpthread.so.0 =>  /lib/libpthread.so.0 (0x00007fdf82f30000)        /lib64/ld-linux-x86-64.so.2 (0x00007fdf83f9c000) link ~/t » ./a.out [1]    7850 segmentation fault  ./a.out link ~/t » gdb7.0 ./a.out

                                                      139 ↵ GNU gdb (GDB) 7.0 Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or laterhttp://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.  Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /home/a/t/a.out...done. (gdb) r Starting program: /home/a/t/a.out [Thread debugging using libthread_db enabled]

Program received signal SIGSEGV, Segmentation fault. 0x000000000067b3c0 in strlen@@GLIBC_2.2.5 () (gdb) bt #0  0x000000000067b3c0 in strlen@@GLIBC_2.2.5 () #1  0x0000000000432dd7 in setFullProgArgv () #2  0x000000000043464d in hs_init () #3  0x000000000043478d in startupHaskell () #4  0x0000000000433d59 in real_main () #5  0x0000000000433e87 in hs_main () #6  0x00007ffff6fccc4d in __libc_start_main () from /lib/libc.so.6 #7  0x0000000000402d49 in _start () (gdb)

What's interesting is that this occurs with all versions of GHC on my machine, which include:  * GHC 6.12.1  * GHC 6.12.3  * GHC 6.13.20100426

Executables that were statically linked with GHC before this strange behavior started occurring appear to still work, i.e. xmonad&  xmobar. I recently did an update of my machine with aptitude, but I would not think this would interfere with GHC.

Strange - I suspect there is some ABI change in glibc that means the static libraries are no longer compatible with the new version.  You'll get more information out of gdb if you compile with -debug, then try stepping through from setFullProgArgv to see what goes wrong.

If there is an ABI change, and it's going to appear in all the major Linux distros at some point, this could give us some difficulties with binary distributions.

Cheers,        Simon

-- - Austin

-- - Austin -- - Austin