[GHC] #8935: Obscure linker bug leads to crash in GHCi
#8935: Obscure linker bug leads to crash in GHCi ------------------------------------+------------------------------------- Reporter: simonmar | Owner: simonmar Type: bug | Status: new Priority: high | Milestone: 7.8.2 Component: Runtime System | Version: 7.6.3 Keywords: | Operating System: Unknown/Multiple Architecture: Unknown/Multiple | Type of failure: GHCi crash Difficulty: Rocket Science | Test Case: Blocked By: | Blocking: Related Tickets: | ------------------------------------+------------------------------------- I have a build of GHC (with `DYNAMIC_GHC_PROGRAMS=NO`) that exhibits the following crash: {{{ $ ghc -e 'System.Environment.getEnvironment' <segfault> }}} I tracked it down, eventually, to a bad reference to the symbol `environ` from `__hscore_environ` in `libraries/base/includes/HsBase.h`. Somehow, `environ` had got linked to the wrong address. Lots more investigation lead me to discover this: `internal_dlsym()` in `Linker.c` tries to look up a symbol in all the different shared libraries we have loaded so far, one by one. (see be497c202b790999c3fd0ddc4a4176b8cf6acf7e). Unfortunately, this seems to break things in my case. Here's a simple test program that works on Ubuntu 12.04: {{{ #include <dlfcn.h> #include <stdio.h> char *so = "/usr/lib/x86_64-linux-gnu/libgmp.so"; char *so2 = "/usr/lib/x86_64-linux-gnu/libpthread.so"; extern char**environ; int main(int argc, char *argv[]) { void *deflt, *hdl; deflt = dlopen(NULL, RTLD_LAZY | RTLD_GLOBAL); printf("environ = %p\n", &environ); printf("dlsym(deflt, \"environ\") = %p\n", dlsym(deflt,"environ")); hdl = dlopen(so, RTLD_LAZY); printf("dlsym(\"libgmp\", \"environ\") = %p\n", dlsym(hdl,"environ")); hdl = dlopen(so2, RTLD_LAZY); printf("dlsym(\"libpthread\", \"environ\") = %p\n", dlsym(hdl,"environ")); } }}} And the output: {{{ environ = 0x601040 dlsym(deflt, "environ") = 0x601040 dlsym("libgmp", "environ") = 0x2aaaab290568 dlsym("libpthread", "environ") = 0x601040 }}} Note that the value we get from looking up `environ` in `libgmp` is different to the others. The correct value is `0x601040`. gdb thinks that `0x2aaaab290568` is also `environ`: {{{ (gdb) p4 0x2aaaab290568 0x2aaaab290580 <buflen.9817>: 0x0 0x2aaaab290578: 0x0 0x2aaaab290570 <miss_F_GETOWN_EX>: 0x0 0x2aaaab290568 <environ>: 0x0 }}} but note that it contains zero. The real one is: {{{ (gdb) p4 0x601040 0x601058: 0x0 0x601050 <dtor_idx.6533>: 0x0 0x601048 <completed.6531>: 0x0 0x601040 <environ@@GLIBC_2.2.5>: 0x7fffffffe268 }}} In GHC we're loading `libgmp` when we load the `integer-gmp` package, and this causes future references to `environ` to go wrong. I've locally fixed this by changing `internal_dlsym` to `dlsym`, but since there was a reason to make this change in the first place I haven't pushed it to master. Suggestions welcome. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: new Priority: high | Milestone: 7.8.2 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Changes (by simonmar): * version: 7.6.3 => 7.8.1-rc2 -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:1> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: new Priority: high | Milestone: 7.8.2 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by trommler): I tried the test program on openSUSE 12.3: {{{ environ = 0x601060 dlsym(deflt, "environ") = 0x601060 dlsym("libgmp", "environ") = 0x601060 dlsym("libpthread", "environ") = 0x601060 }}} Here is my libgmp: {{{ ls /usr/lib64/libgmp.so* /usr/lib64/libgmp.so /usr/lib64/libgmp.so.10 /usr/lib64/libgmp.so.10.0.5 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:2> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: new Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by simonmar): Replying to [comment:2 trommler]:
I tried the test program on openSUSE 12.3: {{{ environ = 0x601060 dlsym(deflt, "environ") = 0x601060 dlsym("libgmp", "environ") = 0x601060 dlsym("libpthread", "environ") = 0x601060 }}} Here is my libgmp: {{{ ls /usr/lib64/libgmp.so* /usr/lib64/libgmp.so /usr/lib64/libgmp.so.10 /usr/lib64/libgmp.so.10.0.5 }}}
IIRC POSIX.1 (but not C!) reserves variable {{{environ}}}. So I'd say it is a bug in Ubuntu's libgmp.
The original distro I encountered the problem on was CentOS. I don't think it is to do with gmp specifically, IIRC there were other libraries that cause the behaviour too (but I'd need to go back and check again). My partial theory is that because `environ` is a symbol with R_COPY relocation, the linker is confused and has returned the original location (before the contents were copied and the symbol relocated). -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:4> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: new Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by dagit): I haven't tested if I have the crash on my system, but suspect you're seeing something to do with weak symbols. On my system, Fedora 19, I see that glibc defines environ as a weak symbol and gives it the same address that your libgmp check reports. I modified your C program slightly so that it takes the paths in `argv[1]` and `argv[2]`: {{{ $ ./check-environ /usr/lib64/libgmp.so /usr/lib64/libpthread.so environ = 0x601058 dlsym(deflt, "environ") = 0x601058 dlsym("libgmp", "environ") = 0x31e45bd508 dlsym("libpthread", "environ") = 0x601058 }}} My libgmp doesn't define environ so it must find it in libc, and indeed: {{{ $ readelf -Wa /usr/lib64/libc.so.6 | grep environ 00000031e45b9dd8 0000011900000006 R_X86_64_GLOB_DAT 00000031e45bd508 _environ + 0 00000031e45b9ea0 0000050000000006 R_X86_64_GLOB_DAT 00000031e45bd508 __environ + 0 281: 00000031e45bd508 8 OBJECT WEAK DEFAULT 33 _environ@@GLIBC_2.2.5 956: 00000031e45bd508 8 OBJECT WEAK DEFAULT 33 environ@@GLIBC_2.2.5 1280: 00000031e45bd508 8 OBJECT GLOBAL DEFAULT 33 __environ@@GLIBC_2.2.5 349: 00000031e45bbe70 8 OBJECT LOCAL DEFAULT 33 last_environ 1813: 0000000000000000 0 FILE LOCAL DEFAULT ABS environ.c 4539: 00000031e4238a10 1122 FUNC LOCAL DEFAULT 12 __add_to_environ 5707: 00000031e45bd508 8 OBJECT WEAK DEFAULT 33 _environ 6374: 00000031e45bd508 8 OBJECT GLOBAL DEFAULT 33 __environ 6483: 00000031e45bd508 8 OBJECT WEAK DEFAULT 33 environ }}} I suspect that the way you're searching for the symbol needs to look for a (the?) strong binding first. I hope that helps. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:5> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: new Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by dagit): I just realized my previous reply probably leaves people wondering: Since libpthread also DOES NOT define environ why does it give the right address? The difference is that libpthread depends on ld-linux.so as a shared object dependency (grep the readelf -Wa output for `NEEDED`) and ld- linux.so defines a `LOCAL` symbol for environ. As I understand it, that means the dynamic linker thinks libpthread defines environ as `LOCAL` (so it goes to look for a different definition and ends up with the right one). For libgmp it looks in libc.so and finds a `WEAK` definition and uses it because there is no other `GLOBAL` definition. To help demonstrate this, consider this example. On my system, libz.so has dependencies similar to libgmp.so: {{{ $ readelf -Wa /usr/lib64/libgmp.so.10 | grep NEEDED 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] $ readelf -Wa /usr/lib64/libz.so.1 | grep NEEDED 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] }}} And look at this: {{{ $ ./check-environ /usr/lib64/libgmp.so /usr/lib64/libz.so.1 environ = 0x601058 dlsym(deflt, "environ") = 0x601058 dlsym("libgmp", "environ") = 0x31e45bd508 dlsym("libpthread", "environ") = 0x31e45bd508 }}} I'm still fuzzy on some of the details, but it seems that `WEAK` and `LOCAL` mean very different things to the dynamic linker. `WEAK` is the same as `GLOBAL` but the link priority is lower, while `LOCAL` means the symbol should not be exported. As far as I can tell, when it finds `LOCAL` it continues the search whereas with `WEAK` it stops earlier because technically it has found an exported symbol. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:6> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: new Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Changes (by jwlato): * cc: jwlato@… (added) -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:7> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: new Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Changes (by ihameed): * cc: idhameed@… (added) -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:8> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: new Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Changes (by akio): * cc: tkn.akio@… (added) -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:9> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: new Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by trommler): Replying to [comment:6 dagit]:
The difference is that libpthread depends on ld-linux.so as a shared object dependency (grep the readelf -Wa output for `NEEDED`) and ld- linux.so defines a `LOCAL` symbol for environ. As I understand it, that means the dynamic linker thinks libpthread defines environ as `LOCAL` (so it goes to look for a different definition and ends up with the right one). For libgmp it looks in libc.so and finds a `WEAK` definition and uses it because there is no other `GLOBAL` definition.
To help demonstrate this, consider this example. On my system, libz.so has dependencies similar to libgmp.so: {{{ $ readelf -Wa /usr/lib64/libgmp.so.10 | grep NEEDED 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] $ readelf -Wa /usr/lib64/libz.so.1 | grep NEEDED 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] }}}
I'm still fuzzy on some of the details, but it seems that `WEAK` and `LOCAL` mean very different things to the dynamic linker. `WEAK` is the same as `GLOBAL` but the link priority is lower, while `LOCAL` means the symbol should not be exported. As far as I can tell, when it finds `LOCAL` it continues the search whereas with `WEAK` it stops earlier because technically it has found an exported symbol. I checked on openSUSE 13.1 (x86_64 and powerpc64) and I have the same {{{NEEDED}}} for libc.so.
{{{environ}}} is a {{{WEAK}}} symbol in my libc.so, too. Here is my libc.so.6 on powerpc64: {{{ 00000000001d7ce0 0000056600000026 R_PPC64_ADDR64 00000000001da1f0 __environ + 0 00000000001d7d00 0000056600000026 R_PPC64_ADDR64 00000000001da1f0 __environ + 0 00000000001d7ee8 0000056600000026 R_PPC64_ADDR64 00000000001da1f0 __environ + 0 00000000001d7ef0 0000056600000026 R_PPC64_ADDR64 00000000001da1f0 __environ + 0 00000000001d8018 0000056600000026 R_PPC64_ADDR64 00000000001da1f0 __environ + 0 00000000001d84a0 0000056600000026 R_PPC64_ADDR64 00000000001da1f0 __environ + 0 00000000001d84a8 0000056600000026 R_PPC64_ADDR64 00000000001da1f0 __environ + 0 00000000001d84b0 0000056600000026 R_PPC64_ADDR64 00000000001da1f0 __environ + 0 00000000001d8570 0000056600000026 R_PPC64_ADDR64 00000000001da1f0 __environ + 0 00000000001d8928 0000056600000026 R_PPC64_ADDR64 00000000001da1f0 __environ + 0 00000000001d81e8 0000013100000026 R_PPC64_ADDR64 00000000001da1f0 _environ + 0 305: 00000000001da1f0 8 OBJECT WEAK DEFAULT 32 _environ@@GLIBC_2.3 1026: 00000000001da1f0 8 OBJECT WEAK DEFAULT 32 environ@@GLIBC_2.3 1382: 00000000001da1f0 8 OBJECT GLOBAL DEFAULT 32 __environ@@GLIBC_2.3 178: 00000000001d8fa8 8 OBJECT LOCAL DEFAULT 32 last_environ 2904: 00000000001c9628 1436 FUNC LOCAL DEFAULT 28 __add_to_environ 4138: 00000000001da1f0 8 OBJECT WEAK DEFAULT 32 _environ 4843: 00000000001da1f0 8 OBJECT GLOBAL DEFAULT 32 __environ 4957: 00000000001da1f0 8 OBJECT WEAK DEFAULT 32 environ }}} On x86_64 it looks similar to comment [comment:5] except the line with environ.c is missing. Do openSUSE patch their system linker? I'll go ask on their mailing list and report back. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:10> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: infoneeded Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Changes (by trommler): * status: new => infoneeded Comment: Could you post the output of {{{LD_DEBUG=symbols ./check-environ}}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:11> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: infoneeded Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by dagit): I hope this helps! This is from the Fedora 19 system I mentioned earlier. {{{ $ LD_DEBUG=symbols ./check-environ /usr/lib64/libgmp.so /usr/lib64/libpthread.so 5823: symbol=_res; lookup in file=./check-environ [0] 5823: symbol=_res; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=_res; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=stderr; lookup in file=./check-environ [0] 5823: symbol=stderr; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=stderr; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=error_one_per_line; lookup in file=./check-environ [0] 5823: symbol=error_one_per_line; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=error_one_per_line; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__malloc_initialize_hook; lookup in file=./check- environ [0] 5823: symbol=__malloc_initialize_hook; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__malloc_initialize_hook; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_dl_starting_up; lookup in file=./check-environ [0] 5823: symbol=_dl_starting_up; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=_dl_starting_up; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_dl_starting_up; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__morecore; lookup in file=./check-environ [0] 5823: symbol=__morecore; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__morecore; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__key_encryptsession_pk_LOCAL; lookup in file =./check-environ [0] 5823: symbol=__key_encryptsession_pk_LOCAL; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__key_encryptsession_pk_LOCAL; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__progname_full; lookup in file=./check-environ [0] 5823: symbol=__progname_full; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__progname_full; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__ctype32_tolower; lookup in file=./check-environ [0] 5823: symbol=__ctype32_tolower; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__ctype32_tolower; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_environ; lookup in file=./check-environ [0] 5823: symbol=_rtld_global; lookup in file=./check-environ [0] 5823: symbol=_rtld_global; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=_rtld_global; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_rtld_global; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__progname; lookup in file=./check-environ [0] 5823: symbol=__progname; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__progname; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=argp_err_exit_status; lookup in file=./check- environ [0] 5823: symbol=argp_err_exit_status; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=argp_err_exit_status; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=mallwatch; lookup in file=./check-environ [0] 5823: symbol=mallwatch; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=mallwatch; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__rcmd_errstr; lookup in file=./check-environ [0] 5823: symbol=__rcmd_errstr; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__rcmd_errstr; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__vdso_clock_gettime; lookup in file=./check- environ [0] 5823: symbol=__vdso_clock_gettime; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__vdso_clock_gettime; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=svcauthdes_stats; lookup in file=./check-environ [0] 5823: symbol=svcauthdes_stats; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=svcauthdes_stats; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__libc_enable_secure; lookup in file=./check- environ [0] 5823: symbol=__libc_enable_secure; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__libc_enable_secure; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__libc_enable_secure; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=_res_hconf; lookup in file=./check-environ [0] 5823: symbol=_res_hconf; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=_res_hconf; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=malloc; lookup in file=./check-environ [0] 5823: symbol=malloc; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=malloc; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=getdate_err; lookup in file=./check-environ [0] 5823: symbol=getdate_err; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=getdate_err; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__tzname; lookup in file=./check-environ [0] 5823: symbol=__tzname; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__tzname; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__timezone; lookup in file=./check-environ [0] 5823: symbol=__timezone; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__timezone; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_rtld_global_ro; lookup in file=./check-environ [0] 5823: symbol=_rtld_global_ro; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=_rtld_global_ro; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_rtld_global_ro; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=optarg; lookup in file=./check-environ [0] 5823: symbol=optarg; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=optarg; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__ctype_tolower; lookup in file=./check-environ [0] 5823: symbol=__ctype_tolower; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__ctype_tolower; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__ctype_toupper; lookup in file=./check-environ [0] 5823: symbol=__ctype_toupper; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__ctype_toupper; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=svc_max_pollfd; lookup in file=./check-environ [0] 5823: symbol=svc_max_pollfd; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=svc_max_pollfd; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__ctype_b; lookup in file=./check-environ [0] 5823: symbol=__ctype_b; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__ctype_b; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=argp_program_version_hook; lookup in file=./check- environ [0] 5823: symbol=argp_program_version_hook; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=argp_program_version_hook; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__after_morecore_hook; lookup in file=./check- environ [0] 5823: symbol=__after_morecore_hook; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__after_morecore_hook; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__environ; lookup in file=./check-environ [0] 5823: symbol=__ctype32_b; lookup in file=./check-environ [0] 5823: symbol=__ctype32_b; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__ctype32_b; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__curbrk; lookup in file=./check-environ [0] 5823: symbol=__curbrk; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__curbrk; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=argp_program_version; lookup in file=./check- environ [0] 5823: symbol=argp_program_version; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=argp_program_version; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__daylight; lookup in file=./check-environ [0] 5823: symbol=__daylight; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__daylight; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__memalign_hook; lookup in file=./check-environ [0] 5823: symbol=__memalign_hook; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__memalign_hook; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__malloc_hook; lookup in file=./check-environ [0] 5823: symbol=__malloc_hook; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__malloc_hook; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__free_hook; lookup in file=./check-environ [0] 5823: symbol=__free_hook; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__free_hook; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=svc_pollfd; lookup in file=./check-environ [0] 5823: symbol=svc_pollfd; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=svc_pollfd; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_nl_domain_bindings; lookup in file=./check- environ [0] 5823: symbol=_nl_domain_bindings; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=_nl_domain_bindings; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_nl_msg_cat_cntr; lookup in file=./check-environ [0] 5823: symbol=_nl_msg_cat_cntr; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=_nl_msg_cat_cntr; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=argp_program_bug_address; lookup in file=./check- environ [0] 5823: symbol=argp_program_bug_address; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=argp_program_bug_address; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__key_decryptsession_pk_LOCAL; lookup in file =./check-environ [0] 5823: symbol=__key_decryptsession_pk_LOCAL; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__key_decryptsession_pk_LOCAL; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=h_errlist; lookup in file=./check-environ [0] 5823: symbol=h_errlist; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=h_errlist; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=program_invocation_short_name; lookup in file =./check-environ [0] 5823: symbol=program_invocation_short_name; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=program_invocation_short_name; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=optind; lookup in file=./check-environ [0] 5823: symbol=optind; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=optind; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=stdout; lookup in file=./check-environ [0] 5823: symbol=stdout; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=stdout; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=obstack_alloc_failed_handler; lookup in file =./check-environ [0] 5823: symbol=obstack_alloc_failed_handler; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=obstack_alloc_failed_handler; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=error_print_progname; lookup in file=./check- environ [0] 5823: symbol=error_print_progname; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=error_print_progname; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=optopt; lookup in file=./check-environ [0] 5823: symbol=optopt; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=optopt; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_IO_funlockfile; lookup in file=./check-environ [0] 5823: symbol=_IO_funlockfile; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=_IO_funlockfile; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=obstack_exit_failure; lookup in file=./check- environ [0] 5823: symbol=obstack_exit_failure; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=obstack_exit_failure; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=error_message_count; lookup in file=./check- environ [0] 5823: symbol=error_message_count; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=error_message_count; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=svc_fdset; lookup in file=./check-environ [0] 5823: symbol=svc_fdset; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=svc_fdset; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=program_invocation_name; lookup in file=./check- environ [0] 5823: symbol=program_invocation_name; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=program_invocation_name; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=loc1; lookup in file=./check-environ [0] 5823: symbol=loc1; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=loc1; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=free; lookup in file=./check-environ [0] 5823: symbol=free; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=free; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=loc2; lookup in file=./check-environ [0] 5823: symbol=loc2; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=loc2; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__fpu_control; lookup in file=./check-environ [0] 5823: symbol=__fpu_control; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__fpu_control; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=re_syntax_options; lookup in file=./check-environ [0] 5823: symbol=re_syntax_options; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=re_syntax_options; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=stdin; lookup in file=./check-environ [0] 5823: symbol=stdin; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=stdin; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__check_rhosts_file; lookup in file=./check- environ [0] 5823: symbol=__check_rhosts_file; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__check_rhosts_file; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=opterr; lookup in file=./check-environ [0] 5823: symbol=opterr; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=opterr; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__ctype32_toupper; lookup in file=./check-environ [0] 5823: symbol=__ctype32_toupper; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__ctype32_toupper; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__realloc_hook; lookup in file=./check-environ [0] 5823: symbol=__realloc_hook; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__realloc_hook; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_dl_argv; lookup in file=./check-environ [0] 5823: symbol=_dl_argv; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=_dl_argv; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_dl_argv; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=rpc_createerr; lookup in file=./check-environ [0] 5823: symbol=rpc_createerr; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=rpc_createerr; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_IO_2_1_stderr_; lookup in file=./check-environ [0] 5823: symbol=_IO_2_1_stderr_; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=_IO_2_1_stderr_; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_IO_2_1_stdout_; lookup in file=./check-environ [0] 5823: symbol=_IO_2_1_stdout_; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=_IO_2_1_stdout_; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_IO_2_1_stdin_; lookup in file=./check-environ [0] 5823: symbol=_IO_2_1_stdin_; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=_IO_2_1_stdin_; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=realloc; lookup in file=./check-environ [0] 5823: symbol=realloc; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=realloc; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=memset; lookup in file=./check-environ [0] 5823: symbol=memset; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=memset; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=malloc; lookup in file=./check-environ [0] 5823: symbol=malloc; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=malloc; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__tls_get_addr; lookup in file=./check-environ [0] 5823: symbol=__tls_get_addr; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__tls_get_addr; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__tls_get_addr; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=memmove; lookup in file=./check-environ [0] 5823: symbol=memmove; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=memmove; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=memalign; lookup in file=./check-environ [0] 5823: symbol=memalign; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=memalign; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=calloc; lookup in file=./check-environ [0] 5823: symbol=calloc; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=calloc; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=free; lookup in file=./check-environ [0] 5823: symbol=free; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=free; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__vdso_time; lookup in file=linux-vdso.so.1 [0] 5823: symbol=__vdso_gettimeofday; lookup in file=linux- vdso.so.1 [0] 5823: symbol=__pthread_key_create; lookup in file=./check- environ [0] 5823: symbol=__pthread_key_create; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__pthread_key_create; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__pthread_key_create; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=_ITM_deregisterTMCloneTable; lookup in file =./check-environ [0] 5823: symbol=_ITM_deregisterTMCloneTable; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=_ITM_deregisterTMCloneTable; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_ITM_deregisterTMCloneTable; lookup in file=/lib64 /ld-linux-x86-64.so.2 [0] 5823: symbol=__pthread_getspecific; lookup in file=./check- environ [0] 5823: symbol=__pthread_getspecific; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__pthread_getspecific; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__pthread_getspecific; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmon_start__; lookup in file=./check-environ [0] 5823: symbol=__gmon_start__; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmon_start__; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmon_start__; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=_rtld_global_ro; lookup in file=./check-environ [0] 5823: symbol=_rtld_global_ro; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=_rtld_global_ro; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_rtld_global_ro; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__pthread_once; lookup in file=./check-environ [0] 5823: symbol=__pthread_once; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__pthread_once; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__pthread_once; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=_libc_intl_domainname; lookup in file=./check- environ [0] 5823: symbol=_libc_intl_domainname; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=_libc_intl_domainname; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_Jv_RegisterClasses; lookup in file=./check- environ [0] 5823: symbol=_Jv_RegisterClasses; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=_Jv_RegisterClasses; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_Jv_RegisterClasses; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__environ; lookup in file=./check-environ [0] 5823: symbol=_ITM_registerTMCloneTable; lookup in file=./check- environ [0] 5823: symbol=_ITM_registerTMCloneTable; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=_ITM_registerTMCloneTable; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_ITM_registerTMCloneTable; lookup in file=/lib64 /ld-linux-x86-64.so.2 [0] 5823: symbol=__pthread_setspecific; lookup in file=./check- environ [0] 5823: symbol=__pthread_setspecific; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__pthread_setspecific; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__pthread_setspecific; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=_rtld_global; lookup in file=./check-environ [0] 5823: symbol=_rtld_global; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=_rtld_global; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_rtld_global; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__cxa_finalize; lookup in file=./check-environ [0] 5823: symbol=__cxa_finalize; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__cxa_finalize; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmon_start__; lookup in file=./check-environ [0] 5823: symbol=__gmon_start__; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmon_start__; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmon_start__; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__environ; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__environ; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_r_debug; lookup in file=./check-environ [0] 5823: symbol=_r_debug; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=_r_debug; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_r_debug; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=free; lookup in file=./check-environ [0] 5823: symbol=free; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=free; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__libc_memalign; lookup in file=./check-environ [0] 5823: symbol=__libc_memalign; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__libc_memalign; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=malloc; lookup in file=./check-environ [0] 5823: symbol=malloc; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=malloc; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__tls_get_addr; lookup in file=./check-environ [0] 5823: symbol=__tls_get_addr; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__tls_get_addr; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__tls_get_addr; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=calloc; lookup in file=./check-environ [0] 5823: symbol=calloc; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=calloc; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=realloc; lookup in file=./check-environ [0] 5823: symbol=realloc; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=realloc; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=free; lookup in file=./check-environ [0] 5823: symbol=free; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=free; lookup in file=/lib64/libc.so.6 [0] 5823: 5823: calling init: /lib64/ld-linux-x86-64.so.2 5823: 5823: 5823: calling init: /lib64/libc.so.6 5823: 5823: symbol=__vdso_clock_gettime; lookup in file=linux- vdso.so.1 [0] 5823: symbol=__vdso_getcpu; lookup in file=linux-vdso.so.1 [0] 5823: 5823: calling init: /lib64/libdl.so.2 5823: 5823: symbol=__libc_start_main; lookup in file=./check-environ [0] 5823: symbol=__libc_start_main; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__libc_start_main; lookup in file=/lib64/libc.so.6 [0] 5823: 5823: initialize program: ./check-environ 5823: 5823: 5823: transferring control: ./check-environ 5823: 5823: symbol=dlopen; lookup in file=./check-environ [0] 5823: symbol=dlopen; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=printf; lookup in file=./check-environ [0] 5823: symbol=printf; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=printf; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=dlsym; lookup in file=./check-environ [0] 5823: symbol=dlsym; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=_dl_sym; lookup in file=./check-environ [0] 5823: symbol=_dl_sym; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=_dl_sym; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=environ; lookup in file=./check-environ [0] 5823: symbol=obstack_vprintf; lookup in file=./check-environ [0] 5823: symbol=obstack_vprintf; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=obstack_vprintf; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=vfprintf; lookup in file=./check-environ [0] 5823: symbol=vfprintf; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=vfprintf; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_asprintf_memory; lookup in file=./check- environ [0] 5823: symbol=__gmp_asprintf_memory; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_asprintf_memory; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_asprintf_memory; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_asprintf_memory; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_asprintf_reps; lookup in file=./check- environ [0] 5823: symbol=__gmp_asprintf_reps; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_asprintf_reps; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_asprintf_reps; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_asprintf_reps; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_asprintf_final; lookup in file=./check- environ [0] 5823: symbol=__gmp_asprintf_final; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_asprintf_final; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_asprintf_final; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_asprintf_final; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__isoc99_fscanf; lookup in file=./check-environ [0] 5823: symbol=__isoc99_fscanf; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__isoc99_fscanf; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=fgetc; lookup in file=./check-environ [0] 5823: symbol=fgetc; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=fgetc; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=ungetc; lookup in file=./check-environ [0] 5823: symbol=ungetc; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=ungetc; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_randget_mt; lookup in file=./check-environ [0] 5823: symbol=__gmp_randget_mt; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_randget_mt; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_randget_mt; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_randget_mt; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_randclear_mt; lookup in file=./check-environ [0] 5823: symbol=__gmp_randclear_mt; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_randclear_mt; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_randclear_mt; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_randclear_mt; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_randiset_mt; lookup in file=./check-environ [0] 5823: symbol=__gmp_randiset_mt; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_randiset_mt; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_randiset_mt; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_randiset_mt; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=_ITM_deregisterTMCloneTable; lookup in file =./check-environ [0] 5823: symbol=_ITM_deregisterTMCloneTable; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=_ITM_deregisterTMCloneTable; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_ITM_deregisterTMCloneTable; lookup in file=/lib64 /ld-linux-x86-64.so.2 [0] 5823: symbol=_ITM_deregisterTMCloneTable; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=_ITM_deregisterTMCloneTable; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_ITM_deregisterTMCloneTable; lookup in file=/lib64 /ld-linux-x86-64.so.2 [0] 5823: symbol=stdout; lookup in file=./check-environ [0] 5823: symbol=stdout; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=stdout; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_reallocate_func; lookup in file=./check- environ [0] 5823: symbol=__gmp_reallocate_func; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_reallocate_func; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_reallocate_func; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_reallocate_func; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=stdin; lookup in file=./check-environ [0] 5823: symbol=stdin; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=stdin; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_fib_table; lookup in file=./check-environ [0] 5823: symbol=__gmp_fib_table; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_fib_table; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_fib_table; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_fib_table; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_fprintf_funs; lookup in file=./check-environ [0] 5823: symbol=__gmp_fprintf_funs; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_fprintf_funs; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_fprintf_funs; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_fprintf_funs; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmpz_sub; lookup in file=./check-environ [0] 5823: symbol=__gmpz_sub; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmpz_sub; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmpz_sub; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmpz_sub; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_allocate_func; lookup in file=./check- environ [0] 5823: symbol=__gmp_allocate_func; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_allocate_func; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_allocate_func; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_allocate_func; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_oddfac_table; lookup in file=./check-environ [0] 5823: symbol=__gmp_oddfac_table; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_oddfac_table; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_oddfac_table; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_oddfac_table; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_junk; lookup in file=./check-environ [0] 5823: symbol=__gmp_junk; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_junk; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_junk; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_junk; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_0; lookup in file=./check-environ [0] 5823: symbol=__gmp_0; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_0; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_0; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_0; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_asprintf_funs; lookup in file=./check- environ [0] 5823: symbol=__gmp_asprintf_funs; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_asprintf_funs; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_asprintf_funs; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_asprintf_funs; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_jacobi_table; lookup in file=./check-environ [0] 5823: symbol=__gmp_jacobi_table; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_jacobi_table; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_jacobi_table; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_jacobi_table; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_limbroots_table; lookup in file=./check- environ [0] 5823: symbol=__gmp_limbroots_table; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_limbroots_table; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_limbroots_table; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_limbroots_table; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_errno; lookup in file=./check-environ [0] 5823: symbol=__gmp_errno; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_errno; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_errno; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_errno; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmon_start__; lookup in file=./check-environ [0] 5823: symbol=__gmon_start__; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmon_start__; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmon_start__; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmon_start__; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmon_start__; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmon_start__; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_default_reallocate; lookup in file=./check- environ [0] 5823: symbol=__gmp_default_reallocate; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_default_reallocate; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_default_reallocate; lookup in file=/lib64 /ld-linux-x86-64.so.2 [0] 5823: symbol=__gmp_default_reallocate; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_digit_value_tab; lookup in file=./check- environ [0] 5823: symbol=__gmp_digit_value_tab; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_digit_value_tab; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_digit_value_tab; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_digit_value_tab; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_odd2fac_table; lookup in file=./check- environ [0] 5823: symbol=__gmp_odd2fac_table; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_odd2fac_table; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_odd2fac_table; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_odd2fac_table; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmpn_bases; lookup in file=./check-environ [0] 5823: symbol=__gmpn_bases; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmpn_bases; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmpn_bases; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmpn_bases; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_sprintf_funs; lookup in file=./check-environ [0] 5823: symbol=__gmp_sprintf_funs; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_sprintf_funs; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_sprintf_funs; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_sprintf_funs; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_binvert_limb_table; lookup in file=./check- environ [0] 5823: symbol=__gmp_binvert_limb_table; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_binvert_limb_table; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_binvert_limb_table; lookup in file=/lib64 /ld-linux-x86-64.so.2 [0] 5823: symbol=__gmp_binvert_limb_table; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_default_fp_limb_precision; lookup in file =./check-environ [0] 5823: symbol=__gmp_default_fp_limb_precision; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_default_fp_limb_precision; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_default_fp_limb_precision; lookup in file=/lib64/ld-linux-x86-64.so.2 [0] 5823: symbol=__gmp_default_fp_limb_precision; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmpn_gcdext_hook; lookup in file=./check-environ [0] 5823: symbol=__gmpn_gcdext_hook; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmpn_gcdext_hook; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmpn_gcdext_hook; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmpn_gcdext_hook; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_rands; lookup in file=./check-environ [0] 5823: symbol=__gmp_rands; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_rands; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_rands; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_rands; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_sscanf_funs; lookup in file=./check-environ [0] 5823: symbol=__gmp_sscanf_funs; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_sscanf_funs; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_sscanf_funs; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_sscanf_funs; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_fac2cnt_table; lookup in file=./check- environ [0] 5823: symbol=__gmp_fac2cnt_table; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_fac2cnt_table; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_fac2cnt_table; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_fac2cnt_table; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmpz_add; lookup in file=./check-environ [0] 5823: symbol=__gmpz_add; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmpz_add; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmpz_add; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmpz_add; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_default_allocate; lookup in file=./check- environ [0] 5823: symbol=__gmp_default_allocate; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_default_allocate; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_default_allocate; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_default_allocate; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=_Jv_RegisterClasses; lookup in file=./check- environ [0] 5823: symbol=_Jv_RegisterClasses; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=_Jv_RegisterClasses; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_Jv_RegisterClasses; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=_Jv_RegisterClasses; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=_Jv_RegisterClasses; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_Jv_RegisterClasses; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_free_func; lookup in file=./check-environ [0] 5823: symbol=__gmp_free_func; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_free_func; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_free_func; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_free_func; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_snprintf_funs; lookup in file=./check- environ [0] 5823: symbol=__gmp_snprintf_funs; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_snprintf_funs; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_snprintf_funs; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_snprintf_funs; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_default_free; lookup in file=./check-environ [0] 5823: symbol=__gmp_default_free; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_default_free; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_default_free; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_default_free; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_rands_initialized; lookup in file=./check- environ [0] 5823: symbol=__gmp_rands_initialized; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_rands_initialized; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_rands_initialized; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_rands_initialized; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=__gmp_obstack_printf_funs; lookup in file=./check- environ [0] 5823: symbol=__gmp_obstack_printf_funs; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_obstack_printf_funs; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_obstack_printf_funs; lookup in file=/lib64 /ld-linux-x86-64.so.2 [0] 5823: symbol=__gmp_obstack_printf_funs; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=_ITM_registerTMCloneTable; lookup in file=./check- environ [0] 5823: symbol=_ITM_registerTMCloneTable; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=_ITM_registerTMCloneTable; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_ITM_registerTMCloneTable; lookup in file=/lib64 /ld-linux-x86-64.so.2 [0] 5823: symbol=_ITM_registerTMCloneTable; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=_ITM_registerTMCloneTable; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=_ITM_registerTMCloneTable; lookup in file=/lib64 /ld-linux-x86-64.so.2 [0] 5823: symbol=__cxa_finalize; lookup in file=./check-environ [0] 5823: symbol=__cxa_finalize; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__cxa_finalize; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=stderr; lookup in file=./check-environ [0] 5823: symbol=stderr; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=stderr; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_fscanf_funs; lookup in file=./check-environ [0] 5823: symbol=__gmp_fscanf_funs; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_fscanf_funs; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_fscanf_funs; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_fscanf_funs; lookup in file=/usr/lib64/libgmp.so [0] 5823: 5823: calling init: /usr/lib64/libgmp.so 5823: 5823: symbol=environ; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=environ; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=free; lookup in file=./check-environ [0] 5823: symbol=free; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=free; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=environ; lookup in file=./check-environ [0] 5823: 5823: calling fini: ./check-environ [0] 5823: 5823: 5823: calling fini: /lib64/libdl.so.2 [0] 5823: 5823: symbol=__cxa_finalize; lookup in file=./check-environ [0] 5823: symbol=__cxa_finalize; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__cxa_finalize; lookup in file=/lib64/libc.so.6 [0] 5823: 5823: calling fini: /usr/lib64/libgmp.so [0] 5823: 5823: symbol=__cxa_finalize; lookup in file=./check-environ [0] 5823: symbol=__cxa_finalize; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__cxa_finalize; lookup in file=/lib64/libc.so.6 [0] environ = 0x601058 dlsym(deflt, "environ") = 0x601058 dlsym("libgmp", "environ") = 0x31e45bd508 dlsym("libpthread", "environ") = 0x601058 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:12> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: infoneeded Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by trommler): Replying to [comment:12 dagit]: Great, thanks!
{{{ 5823: symbol=environ; lookup in file=/usr/lib64/libgmp.so [0] 5823: symbol=environ; lookup in file=/lib64/libc.so.6 [0] }}}
That looks like we have found the reason, why openSUSE behaves differently here: On openSUSE {{{libgmp.so}}} is never checked for {{{environ}}} because {{{ld.so}}} looks in {{{check-environ}}} first. {{{check-environ}}} has symbol {{{environ}}}, because of the {{{extern char** environ;}}} in line 7. According to gABI [http://refspecs.linuxbase.org/elf/gabi4+/ch5.dynamic.html] this is the default behavior. To override that one can set set the flag {{{DF_SYMBOLIC}}} in the shared library's header and I suspect Fedora does that. The behavior is defined as follows in gABI: DF_SYMBOLIC If this flag is set in a shared object library, the dynamic linker's symbol resolution algorithm for references within the library is changed. Instead of starting a symbol search with the executable file, the dynamic linker starts from the shared object itself. If the shared object fails to supply the referenced symbol, the dynamic linker then searches the executable file and other shared objects as usual. Could you post the output of {{{readelf -h /usr/lib64/libgmp.so.10}}} to confirm. Replace 10 with the version on your system. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:13> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: infoneeded Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by dagit): {{{ $ readelf -h /usr/lib64/libgmp.so.10 ELF Header: Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 Class: ELF64 Data: 2's complement, little endian Version: 1 (current) OS/ABI: UNIX - System V ABI Version: 0 Type: DYN (Shared object file) Machine: Advanced Micro Devices X86-64 Version: 0x1 Entry point address: 0x31fba0c080 Start of program headers: 64 (bytes into file) Start of section headers: 488144 (bytes into file) Flags: 0x0 Size of this header: 64 (bytes) Size of program headers: 56 (bytes) Number of program headers: 7 Size of section headers: 64 (bytes) Number of section headers: 31 Section header string table index: 30 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:14> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: infoneeded Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by trommler): Replying to [comment:14 dagit]: My bad! The flags we need to check are in the dynamic section and not in the ELF header. Could you run {{{readelf -d /usr/lib64/libgmp.so.10}}} instead. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:15> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: infoneeded Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by dagit): Hmm...I still don't see the flag you're looking for. {{{ $ readelf -d /usr/lib64/libgmp.so.10 Dynamic section at offset 0x6dce0 contains 27 entries: Tag Type Name/Value 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] 0x000000000000000e (SONAME) Library soname: [libgmp.so.10] 0x000000000000000c (INIT) 0x31fba0aab8 0x000000000000000d (FINI) 0x31fba5e5f0 0x0000000000000019 (INIT_ARRAY) 0x31fbc6d8e0 0x000000000000001b (INIT_ARRAYSZ) 8 (bytes) 0x000000000000001a (FINI_ARRAY) 0x31fbc6d8e8 0x000000000000001c (FINI_ARRAYSZ) 8 (bytes) 0x000000006ffffef5 (GNU_HASH) 0x31fba001f0 0x0000000000000005 (STRTAB) 0x31fba05170 0x0000000000000006 (SYMTAB) 0x31fba01570 0x000000000000000a (STRSZ) 10633 (bytes) 0x000000000000000b (SYMENT) 24 (bytes) 0x0000000000000003 (PLTGOT) 0x31fbc6e000 0x0000000000000002 (PLTRELSZ) 8232 (bytes) 0x0000000000000014 (PLTREL) RELA 0x0000000000000017 (JMPREL) 0x31fba08a90 0x0000000000000007 (RELA) 0x31fba08070 0x0000000000000008 (RELASZ) 2592 (bytes) 0x0000000000000009 (RELAENT) 24 (bytes) 0x000000006ffffffe (VERNEED) 0x31fba08000 0x000000006fffffff (VERNEEDNUM) 1 0x000000006ffffff0 (VERSYM) 0x31fba07afa 0x000000006ffffff9 (RELACOUNT) 52 0x000000006ffffdf8 (CHECKSUM) 0xcc8edcb1 0x000000006ffffdf5 (GNU_PRELINKED) 2013-12-20T11:44:15 0x0000000000000000 (NULL) 0x0 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:16> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: infoneeded Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by simonmar): `libgmp` is not symbolic on my system either. Symbol information and relocs for the program: {{{ $ objdump --syms --dynamic-syms --reloc --dynamic-reloc ./a.out | grep envir 0000000000000000 l df *ABS* 0000000000000000 check- environ.c 0000000000601060 w O .bss 0000000000000008 environ@@GLIBC_2.2.5 0000000000601060 g O .bss 0000000000000008 __environ@@GLIBC_2.2.5 0000000000601060 w DO .bss 0000000000000008 GLIBC_2.2.5 _environ 0000000000601060 w DO .bss 0000000000000008 GLIBC_2.2.5 environ 0000000000601060 g DO .bss 0000000000000008 GLIBC_2.2.5 __environ 0000000000601060 R_X86_64_COPY __environ }}} Note there are 3 symbols at the same location (`environ`, `_environ`, and `__environ`). The first two are weak, the third is global. The third is also a copy reloc, which means its contents is copied to this location when the program starts up. Symbol information and relocs for libc: {{{ $ objdump --syms --dynamic-syms --reloc --dynamic-reloc /lib/x86_64-linux- gnu/libc-2.17.so | grep envir 00000000003c44e8 w DO .bss 0000000000000008 GLIBC_2.2.5 _environ 00000000003c44e8 w DO .bss 0000000000000008 GLIBC_2.2.5 environ 00000000003c44e8 g DO .bss 0000000000000008 GLIBC_2.2.5 __environ 00000000003c0de0 R_X86_64_GLOB_DAT _environ 00000000003c0ea8 R_X86_64_GLOB_DAT __environ }}} Again we have 3 symbols at the same location, all 8 bytes long. There are no mentions of any `environ` symbol in `libgmp`'s symbol table or relocations. `LD_DEBUG=all` shows a normal lookup like this: {{{ 23419: symbol=environ; lookup in file=./a.out [0] 23419: binding file ./a.out [0] to ./a.out [0]: normal symbol `environ' }}} but the lookup in `libgmp` looks like this: {{{ 23419: symbol=environ; lookup in file=/usr/lib/x86_64-linux- gnu/libgmp.so [0] 23419: symbol=environ; lookup in file=/lib/x86_64-linux- gnu/libc.so.6 [0] 23419: binding file /usr/lib/x86_64-linux-gnu/libgmp.so [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `environ' }}} I still have no idea why the lookup in `libgmp` is behaving differently from the others. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:17> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: infoneeded Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by dagit): Simon, did you read my explanation above about `LOCAL` vs. `GLOBAL`? -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:18> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: infoneeded Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by simonmar): I read it, but I don't think I understand the difference between local and global. So do you have an explanation for what we're seeing? The impression I got from what you wrote above is that there was still some guesswork involved. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:19> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: infoneeded Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by simonmar): Also, on my system `ld-linux.so` doesn't have a symbol for `environ`, local or otherwise. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:20> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: infoneeded Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by dagit): Simon, at the moment I'm just trying to understand this whole thing better myself. My uncertainty is in these areas: * How the `dlsym()` search should work. For example, I'm not sure if `dlsym()` is supposed to look in the `NEEDED` dependencies or not. It seems to look in them. * Whether glibc implements the above search correctly. * Finally, I have no idea what is the right way to fix ghc's linker. I assume you're already familiar with the elf linking spec, but here is the link just to make sure we're looking at the same doc: http://refspecs.linuxbase.org/elf/elf.pdf It says: * STB_LOCAL Local symbols are not visible outside the object file containing their definition. Local symbols of the same name may exist in multiple files without interfering with each other. * STB_GLOBAL Global symbols are visible to all object files being combined. One file's definition of a global symbol will satisfy another file's undefined reference to the same global symbol. * STB_WEAK Weak symbols resemble global symbols, but their definitions have lower precedence. I'd say, based on what I see on my machine that `WEAK` seems to be even lower priority than `LOCAL`. It's interesting that your ld-linux.so doesn't have `environ`, when I grep mine I get these hits: {{{ $ readelf -Wa /usr/lib64/ld-linux-x86-64.so.2 | grep environ 143: 0000000000000000 0 FILE LOCAL DEFAULT ABS dl-environ.c 179: 0000000000000000 0 FILE LOCAL DEFAULT ABS environ.c 268: 00000031e4022168 8 OBJECT LOCAL DEFAULT 21 _environ 275: 00000031e4022168 8 OBJECT LOCAL DEFAULT 21 __environ 300: 00000031e4022168 8 OBJECT LOCAL DEFAULT 21 environ }}} For your libgmp.so does `environ` get defined in any of the deps listed by `ldd`? The `dlsym` manpage has this (potentially) interesting comment:
External references in the library are resolved using the libraries in that library's dependency list and any other libraries previously opened with the RTLD_GLOBAL flag. If the executable was linked with the flag "-rdynamic" (or, synonymously, "--export-dynamic"), then the global symbols in the executable will also be used to resolve references in a dynamically loaded library.
Please let me know if there is more I can do to help. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:21> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: infoneeded Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by trommler): Let's look at comment:12 again. We see that even for symbols defined in libgmp.so the dynamic linker searches ./check-environ (and libdl and libc and ld-linux) before it finally finds the symbol in libgmp.so {{{ 5823: symbol=__gmp_default_free; lookup in file=./check-environ [0] 5823: symbol=__gmp_default_free; lookup in file=/lib64/libdl.so.2 [0] 5823: symbol=__gmp_default_free; lookup in file=/lib64/libc.so.6 [0] 5823: symbol=__gmp_default_free; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 5823: symbol=__gmp_default_free; lookup in file=/usr/lib64/libgmp.so [0] }}} And then for {{{environ}}} the search starts in {{{libgmp.so}}}. Perhaps because {{{environ}}} is a weak symbol. It seems that openSUSE ignores the weak attribute of symbols in the dynamic linker and treats them as strong and hence starts the search in {{{./check-environ}}}. This seems to be correct as Drepper [http://www.akkadia.org/drepper/dsohowto.pdf] says on p. 6: "Note that a definition in a DSO being weak has no effects. Weak definitions only play a role in static linking." It seems Ubuntu and Fedora do not agree. There is also an environment variable {{{LD_DYNAMIC_WEAK}}} (see ld.so(8)) to allow overriding weak symbols. Is that environment variable set on your systems? -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:22> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: infoneeded Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by dagit): The value of `LD_DYNAMIC_WEAK` doesn't make a difference on my system. I've also tried printing the result of `dlerror()` before/after calls to `dlsym()`. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:23> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: infoneeded Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by trommler): Replying to [comment:23 dagit]:
The value of `LD_DYNAMIC_WEAK` doesn't make a difference on my system. I've also tried printing the result of `dlerror()` before/after calls to `dlsym()`. What happens if there is no `LD_DYNAMIC_WEAK` in the environment?
-- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:24> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: infoneeded Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by trommler): Arrrg! Now I see what I was doing wrong! The libraries were specified with paths that do not exist on openSUSE. With this program I can reproduce the problem: {{{ #include <dlfcn.h> #include <stdio.h> char *so = "libgmp.so"; char *so2 = "libpthread.so"; extern char**environ; int main(int argc, char *argv[]) { void *deflt, *hdl; deflt = dlopen(NULL, RTLD_LAZY | RTLD_GLOBAL); printf("&environ = %p, environ = %p\n", &environ, environ); char ***dflt_env = dlsym(deflt,"environ"); printf("dlsym(deflt, \"environ\") = %p, *dflt_env = %p\n", dflt_env, *dflt_env ); hdl = dlopen(so, RTLD_LAZY); char ***env = dlsym(hdl,"environ"); printf("dlsym(\"libgmp\", \"environ\") = %p\n", env); printf("*env = %p\n", *env); hdl = dlopen(so2, RTLD_LAZY); printf("dlsym(\"libpthread\", \"environ\") = %p\n", dlsym(hdl,"environ")); } }}} I also added code that shows the content of the pointer returned from `dlsym` called with `libgmp.so`. {{{ peter@montebre:~/tmp> ./a.out &environ = 0x601060, environ = 0x7fffc82ce4b8 dlsym(deflt, "environ") = 0x601060, *dflt_env = 0x7fffc82ce4b8 dlsym("libgmp", "environ") = 0x7f9dcb870fd8 *env = (nil) dlsym("libpthread", "environ") = 0x601060 }}} Now, playing some more, I removed `extern char **environ;` and the print statements and the `NULL` pointer issue is gone: {{{ #include <dlfcn.h> #include <stdio.h> char *so = "libgmp.so"; char *so2 = "libpthread.so"; // extern char**environ; int main(int argc, char *argv[]) { void *deflt, *hdl; deflt = dlopen(NULL, RTLD_LAZY | RTLD_GLOBAL); // printf("&environ = %p, environ = %p\n", &environ, environ); char ***dflt_env = dlsym(deflt,"environ"); printf("dlsym(deflt, \"environ\") = %p, *dflt_env = %p\n", dflt_env, *dflt_env ); hdl = dlopen(so, RTLD_LAZY); char ***env = dlsym(hdl,"environ"); printf("dlsym(\"libgmp\", \"environ\") = %p\n", env); printf("*env = %p\n", *env); hdl = dlopen(so2, RTLD_LAZY); printf("dlsym(\"libpthread\", \"environ\") = %p\n", dlsym(hdl,"environ")); } }}} {{{ peter@montebre:~/tmp> ./a.out dlsym(deflt, "environ") = 0x7f4a0181bfd8, *dflt_env = 0x7fff2d021848 dlsym("libgmp", "environ") = 0x7f4a0181bfd8 *env = 0x7fff2d021848 dlsym("libpthread", "environ") = 0x7f4a0181bfd8 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:25> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: infoneeded Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by trommler): Adding error handling: {{{ hdl = dlopen(so2, RTLD_LAZY); if (hdl == NULL) { printf("failed to open libpthread: %s\n", dlerror()); return 1; } printf("dlsym(\"libpthread\", \"environ\") = %p\n", dlsym(hdl,"environ")); }}} I get: {{{ &environ = 0x601068, environ = 0x7fffd170d688 dlsym(deflt, "environ") = 0x601068, *dflt_env = 0x7fffd170d688 dlsym("libgmp", "environ") = 0x7fcc61a82fd8 *env = (nil) failed to open libpthread: /usr/lib64/libpthread.so: invalid ELF header }}} `libgmp.so` is not special after all. On openSUSE `libpthread.so` is a linker script. `dlopen()` cannot open it and returns `NULL`. `RTLD_DEFAULT` is defined as `(void *) 0` so `NULL` so dlsym did not fail but return the symbol from the default SO (the program itself). Trying `libpthread.so.0` instead shows the same problem as `libgmp.so`. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:26> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: infoneeded Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by trommler): I have the following fix in mind: In `internal_dlsym` search the default handle first and then the list of SOs opened so far. But there are a few things in `Linker.c` that I don't understand yet: 1. `hdl` is `dl_prog_handle` in all two call sites. Do we need that parameter or could we instead use the static variable `dl_prog_handle` directly? 1. With `hdl` being `dl_prog_handle`: In `internal_dlsym` the SO pointed to by `hdl` is searched last. Why can I not look there first? 1. In `internal_dlopen` the SO is opened with `RTLD_GLOBAL` but we want to be able to override symbols later when we reload a library. Why is `RTLD_GLOBAL` needed? -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:27> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: infoneeded Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by trommler): And another question: From `internal_dlopen()`: {{{ if (hdl == NULL) { /* dlopen failed; return a ptr to the error msg. */ errmsg = dlerror(); if (errmsg == NULL) errmsg = "addDLL: unknown error"; errmsg_copy = stgMallocBytes(strlen(errmsg)+1, "addDLL"); strcpy(errmsg_copy, errmsg); errmsg = errmsg_copy; } o_so = stgMallocBytes(sizeof(OpenedSO), "addDLL"); o_so->handle = hdl; o_so->next = openedSOs; openedSOs = o_so; }}} If dlopen() fails we are adding a `NULL` handle to `openedSOs`. Which on my system means `RTLD_DEFAULT`. Is that what we want? -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:28> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: infoneeded Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by simonmar): My apologies for making a buggy test program :-) Thanks for eliminating the strangeness there. So it seems that `dlsym` always returns a bogus result for `environ`. Your proposed fix only works if we load libraries with `RTLD_LOCAL` rather than `RTLD_GLOBAL`, I think. That is probably fine, but it needs testing. This commit that introduced `internal_dlsym` explains the rationale: be497c202b790999c3fd0ddc4a4176b8cf6acf7e I'm fine with refactoring away the `hdl` argument. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:29> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: new Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Changes (by trommler): * status: infoneeded => new Comment: Replying to [comment:29 simonmar]:
My apologies for making a buggy test program :-) Thanks for eliminating the strangeness there. No problem! I learned a lot about dynamic linking! So it seems that `dlsym` always returns a bogus result for `environ`. No, no, it does not look bad at all. The address looks strange but that is caused by the fact that it was created by the dynamic linker. Look at `*defl_env` and `*env` in the second program in comment:25. The addresses match and I checked they really point to the environment array!
In the first program the static linker creates the reference for environ because we take its address `&environ` to print it. So there is a static symbol already there and that confuses the dynamic linker when it searches in `libgmp` and then finds `environ` in `libc`. Once we have fixed this bug I will read the ELF specs again and see if giving us a defunct symbol when we load a weak symbol again is really allowed by the specification.
Your proposed fix only works if we load libraries with `RTLD_LOCAL`
rather than `RTLD_GLOBAL`, I think. That is probably fine, but it needs testing. This commit that introduced `internal_dlsym` explains the rationale: be497c202b790999c3fd0ddc4a4176b8cf6acf7e
I'm fine with refactoring away the `hdl` argument.
There is a branch on my github [https://github.com/trommler/ghc/tree/rtld-8935] where I implemented that and what I proposed above. While my AMD 64 bit and my old PowerMac G5 build ghc, I will read up on the rationale for `internal_dlsym`. If validate succeeds and I am sufficiently confident I understand what `internal_dlsym` needs to do. I'll post my patches. We can remove the info needed. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:30> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: new Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by trommler): Validating on x86_64 Linux: {{{ Unexpected failures: codeGen/should_run cgrun051 [bad exit code] (normal) ffi/should_compile cc004 [stderr mismatch] (normal) ffi/should_run T2276_ghci [bad stdout or stderr] (ghci) ghci/scripts T8696 [bad stderr] (ghci) perf/compiler T3064 [stat not good enough] (normal) perf/compiler T6048 [stat not good enough] (optasm) }}} So two dynamic linking related test failures: T2276_ghci and T8696. T2276_ghci: {{{ =====> T2276_ghci(ghci) 3154 of 3962 [0, 0, 0] cd ./ffi/should_run && $MAKE -s --no-print-directory T2276_ghci_setup T2276_ghci_c.c:4:1: warning: ‘stdcall’ attribute ignored [-Wattributes] { ^ cd ./ffi/should_run && '/local/home/peter/ghc-validate/inplace/bin/ghc- stage2' -fforce-recomp -dcore-lint -dcmm-lint -dno-debug-output -no-user- package-db -rtsopts -fno-ghci-history T2276_ghci.hs --interactive -v0 -ignore-dot-ghci +RTS -I0.1 -RTS -fobject-code T2276_ghci_c.o <T2276_ghci.genscript 1>T2276_ghci.interp.stdout 2>T2276_ghci.interp.stderr Actual stderr output differs from expected: --- /dev/null 2014-05-06 16:04:29.720620754 +0200 +++ ./ffi/should_run/T2276_ghci.run.stderr 2014-05-06 21:35:41.193750997 +0200 @@ -0,0 +1,6 @@ +ghc-stage2: panic! (the 'impossible' happened) + (GHC version 7.9.20140506 for x86_64-unknown-linux): + Loading temp shared object failed: /tmp/ghc18005_0/ghc18005_5.so: undefined symbol: test + }}} The warning about `stdcall` is strange. and T8696: {{{ =====> T8696(ghci) 3619 of 3962 [0, 0, 0] cd ./ghci/scripts && HC='/local/home/peter/ghc-validate/inplace/bin/ghc- stage2' HC_OPTS='-dcore-lint -dcmm-lint -dno-debug-output -no-user- package-db -rtsopts -fno-ghci-history ' '/local/home/peter/ghc- validate/inplace/bin/ghc-stage2' --interactive -v0 -ignore-dot-ghci -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-ghci-history <T8696.script >T8696.run.stdout 2>T8696.run.stderr Actual stderr output differs from expected: --- /dev/null 2014-05-06 16:04:29.720620754 +0200 +++ ./ghci/scripts/T8696.run.stderr 2014-05-06 21:37:54.034299041 +0200 @@ -0,0 +1,6 @@ +ghc-stage2: panic! (the 'impossible' happened) + (GHC version 7.9.20140506 for x86_64-unknown-linux): + Loading temp shared object failed: /tmp/ghc18096_0/ghc18096_4.so: undefined symbol: T8696A_a_closure + }}} The performance tests pass when I run them from the command line (I validated with `CPUS=4`) and I seem to remember that `cgrun051` exit code was discussed on `ghc-devs` recently. I have no idea what to make of `cc004` and the warning about `stdcall` calling convention not implemented on my platform. And last but not least: {{{ghc -e 'System.Environment.getEnvironment'}}} does not segfault but prints the actual environment. I'll prepare a patch and attach it to the ticket. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:31> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: patch Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Changes (by trommler): * status: new => patch Comment: I attached four patches for review. I'll check if I see the test failures also in HEAD. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:32> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: patch Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by simonmar):
No, no, it does not look bad at all. The address looks strange but that is caused by the fact that it was created by the dynamic linker. Look at *defl_env and *env in the second program in comment:25. The addresses match and I checked they really point to the environment array!
Well, every time we call `dlsym` passing the handle of a `dlopen`'d library, we get a bogus result. This happens for both `libgmp` and `libpthread` (after modifying the `libpthread` path to point to the real .so, not the linker script). {{{ #include <dlfcn.h> #include <stdio.h> #include <stdlib.h> char *so = "/usr/lib/x86_64-linux-gnu/libgmp.so"; char *so2 = "/lib/x86_64-linux-gnu/libpthread.so.0"; extern char**environ; int main(int argc, char *argv[]) { void *deflt, *hdl; char ***env; deflt = dlopen(NULL, RTLD_LAZY | RTLD_GLOBAL); printf("&environ = %p, environ = %p\n", &environ, environ); env = dlsym(deflt,"environ"); printf("dlsym(deflt, \"environ\") = %p, *env = %p\n", env, *env ); hdl = dlopen(so, RTLD_LAZY); if (hdl == NULL) { printf("%s\n", dlerror()); exit(1); } env = dlsym(hdl,"environ"); printf("dlsym(\"libgmp\", \"environ\") = %p, *env = %p\n", env, *env); hdl = dlopen(so2, RTLD_LAZY); if (hdl == NULL) { printf("%s\n", dlerror()); exit(1); } env = dlsym(hdl,"environ"); printf("dlsym(\"libpthread\", \"environ\") = %p, *env = %p\n", env, *env); } }}} output: {{{ $ ./a.out &environ = 0x601078, environ = 0x7fffc44c2008 dlsym(deflt, "environ") = 0x601078, *env = 0x7fffc44c2008 dlsym("libgmp", "environ") = 0x2ba8772b64e8, *env = (nil) dlsym("libpthread", "environ") = 0x2ba8772b64e8, *env = (nil) }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:33> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: patch Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by trommler): Replying to [comment:33 simonmar]:
No, no, it does not look bad at all. The address looks strange but that is caused by the fact that it was created by the dynamic linker. Look at *defl_env and *env in the second program in comment:25. The addresses match and I checked they really point to the environment array!
Well, every time we call `dlsym` passing the handle of a `dlopen`'d library, we get a bogus result. This happens for both `libgmp` and `libpthread` (after modifying the `libpthread` path to point to the real .so, not the linker script).
Yes, that is consistent with what I expect from our previous experiments. I don't say I understand yet why the dynamic linker gives us a broken symbol and if this behavior is correct. What I know is that the dynamic linker treats weak symbols the same as strong symbols. And then there is a conflict: In the program there is a symbol `environ` created by the static linker and now there is another symbol `environ` and that is not permitted and instead of failing we get this `NULL` pointer. I bet all libraries that depend on libc directly or indirectly will exhibit the behavior we see. But that was not my point. I wanted to say, when we look for `environ` in `RTLD_DEFAULT` we will get a properly set up pointer to the environment. My second test program (in comment:25) was meant to demonstrate that this is also true when we do not refer to `environ` in the program itself. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:34> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: patch Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by trommler): Replying to [comment:32 trommler]:
I attached four patches for review.
I'll check if I see the test failures also in HEAD. The validate on HEAD just finished.
The two failures I reported earlier are regressions. I will look into these on Thursday. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:35> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: patch Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by dagit): Here is the test program I use: {{{ #include <dlfcn.h> #include <stdio.h> #include <stdlib.h> int jason; extern char**environ; char envname[] = "environ"; char jasname[] = "jason"; char weakname[] = "weak_jason"; void dlsym_check(void *handle, char * lib, char * sym){ void * e = dlsym(handle,sym); fprintf(stderr, "dlsym(\"%s\", \"%s\") = %p\n", lib, sym, e); char * error = dlerror(); if( error != NULL ) fprintf(stderr, "Errors: %s\n", error); } int main(int argc, char *argv[]) { if( argc < 3 ){ printf("usage: ./check-environ <path to shared object> <path to shared object>\n"); exit(1); } char* so = argv[1]; char* so2 = argv[2]; void *deflt, *hdl, *hdl2; deflt = dlopen(NULL, RTLD_LAZY | RTLD_GLOBAL); dlsym_check(deflt, so, envname); hdl = dlopen(so, RTLD_LAZY | RTLD_GLOBAL); if( hdl == NULL ){ fprintf(stderr, "Error: %s", dlerror()); exit(1); } hdl2 = dlopen(so2, RTLD_LAZY | RTLD_GLOBAL); if( hdl2 == NULL ){ fprintf(stderr, "Error: %s", dlerror()); exit(1); } fprintf(stderr, "\nFind a symbol that is located here and libc\n"); dlsym_check(deflt, so, envname); dlsym_check(hdl, so, envname); dlsym_check(RTLD_DEFAULT, so, envname); dlsym_check(RTLD_NEXT, so, envname); fprintf(stderr, "\nFind a symbol that is only defined in this executable\n"); dlsym_check(deflt, so, jasname); dlsym_check(hdl, so, jasname); dlsym_check(RTLD_DEFAULT, so, jasname); dlsym_check(RTLD_NEXT, so, jasname); fprintf(stderr, "\nFind a symbol that is only defined weak in a shared object\n"); dlsym_check(deflt, so2, weakname); dlsym_check(hdl2, so2, weakname); dlsym_check(RTLD_DEFAULT, so2, weakname); dlsym_check(RTLD_NEXT, so2, weakname); } }}} That lives in check-environ.c, so I build that with: {{{ LDFLAGS=-ldl CFLAGS=-rdynamic make check-environ }}} (No actual build file required, I just use implicit rules.) I also have test.c, containing just one line: {{{ int __attribute__((weak)) weak_jason; }}} Built with: `gcc -shared -fpic test.c -o test.so` Notice I've added a symbol "jason" that is not defined anywhere but in the executable. I use `-rdynamic` as the manpage for `dlsym` suggests. The test.so shared object has a weak symbol definition for `weak_jason` (you should double check me). I've also duplicated the handle options to `dlsym` just so we can see how they change the behavior. When I run it: {{{ $ ./check-environ libgmp.so ./test.so 2>&1 dlsym("libgmp.so", "environ") = 0x31e45bd508 Find a symbol that is located here and libc dlsym("libgmp.so", "environ") = 0x31e45bd508 dlsym("libgmp.so", "environ") = 0x31e45bd508 dlsym("libgmp.so", "environ") = 0x31e45bd508 dlsym("libgmp.so", "environ") = 0x31e45bd508 Find a symbol that is only defined in this executable dlsym("libgmp.so", "jason") = 0x60208c dlsym("libgmp.so", "jason") = (nil) Errors: /lib64/libgmp.so: undefined symbol: jason dlsym("libgmp.so", "jason") = 0x60208c dlsym("libgmp.so", "jason") = (nil) Find a symbol that is only defined weak in a shared object dlsym("./test.so", "weak_jason") = 0x7f8de2f1e02c dlsym("./test.so", "weak_jason") = 0x7f8de2f1e02c dlsym("./test.so", "weak_jason") = 0x7f8de2f1e02c dlsym("./test.so", "weak_jason") = 0x7f8de2f1e02c }}} I then ran it with `LD_DEBUG=all`, and looked at the output for `environ`. {{{ Find a symbol that is located here and libc 21283: symbol=environ; lookup in file=./check-environ [0] 21283: symbol=environ; lookup in file=/lib64/libdl.so.2 [0] 21283: symbol=environ; lookup in file=/lib64/libc.so.6 [0] 21283: binding file ./check-environ [0] to /lib64/libc.so.6 [0]: normal symbol `environ' dlsym("libgmp.so", "environ") = 0x31e45bd508 21283: symbol=environ; lookup in file=/lib64/libgmp.so [0] 21283: symbol=environ; lookup in file=/lib64/libc.so.6 [0] 21283: binding file /lib64/libgmp.so [0] to /lib64/libc.so.6 [0]: normal symbol `environ' dlsym("libgmp.so", "environ") = 0x31e45bd508 21283: symbol=environ; lookup in file=./check-environ [0] 21283: symbol=environ; lookup in file=/lib64/libdl.so.2 [0] 21283: symbol=environ; lookup in file=/lib64/libc.so.6 [0] 21283: binding file ./check-environ [0] to /lib64/libc.so.6 [0]: normal symbol `environ' dlsym("libgmp.so", "environ") = 0x31e45bd508 21283: symbol=environ; lookup in file=/lib64/libdl.so.2 [0] 21283: symbol=environ; lookup in file=/lib64/libc.so.6 [0] 21283: binding file ./check-environ [0] to /lib64/libc.so.6 [0]: normal symbol `environ' dlsym("libgmp.so", "environ") = 0x31e45bd508 }}} In each cases, once it found the definition in libc it stopped looking (even though it's `WEAK`). The search for `weak_jason` is similar, but it has to keep looking until it gets to test.so. I think it's safe to say that the behavior is not specific to environ or libc: {{{ Find a symbol that is only defined weak in a shared object 21283: symbol=weak_jason; lookup in file=./check-environ [0] 21283: symbol=weak_jason; lookup in file=/lib64/libdl.so.2 [0] 21283: symbol=weak_jason; lookup in file=/lib64/libc.so.6 [0] 21283: symbol=weak_jason; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 21283: symbol=weak_jason; lookup in file=/lib64/libgmp.so [0] 21283: symbol=weak_jason; lookup in file=./test.so [0] 21283: binding file ./check-environ [0] to ./test.so [0]: normal symbol `weak_jason' dlsym("./test.so", "weak_jason") = 0x7f0e99fbd02c 21283: symbol=weak_jason; lookup in file=./test.so [0] 21283: binding file ./test.so [0] to ./test.so [0]: normal symbol `weak_jason' dlsym("./test.so", "weak_jason") = 0x7f0e99fbd02c 21283: symbol=weak_jason; lookup in file=./check-environ [0] 21283: symbol=weak_jason; lookup in file=/lib64/libdl.so.2 [0] 21283: symbol=weak_jason; lookup in file=/lib64/libc.so.6 [0] 21283: symbol=weak_jason; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 21283: symbol=weak_jason; lookup in file=/lib64/libgmp.so [0] 21283: symbol=weak_jason; lookup in file=./test.so [0] 21283: binding file ./check-environ [0] to ./test.so [0]: normal symbol `weak_jason' dlsym("./test.so", "weak_jason") = 0x7f0e99fbd02c 21283: symbol=weak_jason; lookup in file=/lib64/libdl.so.2 [0] 21283: symbol=weak_jason; lookup in file=/lib64/libc.so.6 [0] 21283: symbol=weak_jason; lookup in file=/lib64/ld- linux-x86-64.so.2 [0] 21283: symbol=weak_jason; lookup in file=/lib64/libgmp.so [0] 21283: symbol=weak_jason; lookup in file=./test.so [0] 21283: binding file ./check-environ [0] to ./test.so [0]: normal symbol `weak_jason' dlsym("./test.so", "weak_jason") = 0x7f0e99fbd02c }}} In the case of the `jason` symbol: {{{ Find a symbol that is only defined in this executable 21283: symbol=jason; lookup in file=./check-environ [0] 21283: binding file ./check-environ [0] to ./check-environ [0]: normal symbol `jason' dlsym("libgmp.so", "jason") = 0x60208c 21283: symbol=jason; lookup in file=/lib64/libgmp.so [0] 21283: symbol=jason; lookup in file=/lib64/libc.so.6 [0] 21283: symbol=jason; lookup in file=/lib64/ld-linux-x86-64.so.2 [0] 21283: /lib64/libgmp.so: error: symbol lookup error: undefined symbol: jason (fatal) dlsym("libgmp.so", "jason") = (nil) 21283: symbol=__dcgettext; lookup in file=./check-environ [0] 21283: symbol=__dcgettext; lookup in file=/lib64/libdl.so.2 [0] 21283: symbol=__dcgettext; lookup in file=/lib64/libc.so.6 [0] 21283: binding file /lib64/libdl.so.2 [0] to /lib64/libc.so.6 [0]: normal symbol `__dcgettext' [GLIBC_2.2.5] 21283: symbol=__asprintf; lookup in file=./check-environ [0] 21283: symbol=__asprintf; lookup in file=/lib64/libdl.so.2 [0] 21283: symbol=__asprintf; lookup in file=/lib64/libc.so.6 [0] 21283: binding file /lib64/libdl.so.2 [0] to /lib64/libc.so.6 [0]: normal symbol `__asprintf' [GLIBC_2.2.5] 21283: symbol=free; lookup in file=./check-environ [0] 21283: symbol=free; lookup in file=/lib64/libdl.so.2 [0] 21283: symbol=free; lookup in file=/lib64/libc.so.6 [0] 21283: binding file /lib64/libdl.so.2 [0] to /lib64/libc.so.6 [0]: normal symbol `free' [GLIBC_2.2.5] Errors: /lib64/libgmp.so: undefined symbol: jason 21283: symbol=jason; lookup in file=./check-environ [0] 21283: binding file ./check-environ [0] to ./check-environ [0]: normal symbol `jason' dlsym("libgmp.so", "jason") = 0x60208c 21283: symbol=jason; lookup in file=/lib64/libdl.so.2 [0] 21283: symbol=jason; lookup in file=/lib64/libc.so.6 [0] 21283: symbol=jason; lookup in file=/lib64/ld-linux-x86-64.so.2 [0] 21283: symbol=jason; lookup in file=/lib64/libgmp.so [0] 21283: symbol=jason; lookup in file=./test.so [0] dlsym("libgmp.so", "jason") = (nil) }}} The point though, is that `dlsym` is doing exactly what it should: It returns the first global definition it finds. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:36> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: patch Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by simonmar):
The point though, is that dlsym is doing exactly what it should: It returns the first global definition it finds.
That's right, but let me add that weak symbols are a red herring. Let me demonstrate this without using weak symbols or `environ`. foo.c: {{{ int bar = 1; int getbar(void) { return bar; } }}} Compile it like this: {{{ $ gcc -fPIC foo.c -shared -o libfoo.so }}} Now the test program, `test.c`: {{{ #include <dlfcn.h> #include <stdio.h> #include <stdlib.h> extern int bar; extern int getbar(void); int main(int argc, char *argv[]) { void *deflt, *hdl; int *pbar; int (*pgetbar)(void); bar = 2; printf("&bar = %p, bar = %d, getbar() = %d\n", &bar, bar, getbar()); deflt = dlopen(NULL, RTLD_LAZY | RTLD_GLOBAL); if (hdl == NULL) { printf("%s\n", dlerror()); exit(1); } pbar = dlsym(deflt, "bar"); printf("dlsym(deflt, \"bar\") = %p, *pbar = %d\n", pbar, *pbar); pgetbar = dlsym(deflt, "getbar"); printf("dlsym(deflt, \"getbar\") = %p, pgetbar() = %d\n", pgetbar, (*pgetbar)()); hdl = dlopen("libfoo.so", RTLD_LAZY); if (hdl == NULL) { printf("%s\n", dlerror()); exit(1); } pbar = dlsym(hdl, "bar"); printf("dlsym(\"./libfoo.so\", \"bar\") = %p, *pbar = %d\n", pbar, *pbar); pgetbar = dlsym(hdl, "getbar"); printf("dlsym(deflt, \"getbar\") = %p, pgetbar() = %d\n", pgetbar, (*pgetbar)()); } }}} Note that we have `extern` references for both the data variable `bar` and the function `getbar`. Compile and run it like this: {{{ $ gcc test.c -ldl libfoo.so $ LD_LIBRARY_PATH=. ./a.out }}} {{{ $ LD_LIBRARY_PATH=. ./a.out &bar = 0x601060, bar = 2, getbar() = 2 dlsym(deflt, "bar") = 0x601060, *pbar = 2 dlsym(deflt, "getbar") = 0x2b9809ab259c, pgetbar() = 2 dlsym("./libfoo.so", "bar") = 0x2b9809cb3010, *pbar = 1 dlsym(deflt, "getbar") = 0x2b9809ab259c, pgetbar() = 2 }}} Note that: * the address of the function `getbar` is the same, regardless of whether we look it up in the main program or `libfoo.so` * the data variable `bar` has one address in the main program, and another one in `libfoo.so`. * the version in `libfoo.so` has the initial value 1, it wasn't updated by the assignment `bar = 2` in the main program * Calling `getbar()` returns the correct value 2, because the refernce to `bar` in `libfoo.so` has been relocated to point to the version of `bar` in the main program, not the one in `libfoo.so`. So I'm pretty sure that this is all due to the copy semantics that moves data variables into the main program. It means that if you use `dlsym` to find the address of a data variable, you might get the wrong answer. In any case, the correct fix for GHC ought to be to use `RTLD_LOCAL` and look up in the main program first. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:37> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: patch Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by dagit): Great and thanks for the examples! By the way, weak not making a difference is the other half of the point I wanted to make :) I'm glad we're on the same page. I feel like I've learned something useful that I can use elsewhere. You might also look at `RTLD_DEEPBIND` for doing the initial symbol match up (when you use `RTLD_LOCAL`). I can't be certain, but I think it may help when reloading a shared object (presumably with new definitions inside). As far as I can tell, that option was added specifically for uses like this (I found some mailing list threads about it). I suspect the point with that option is that it allows consistent behavior even if something you don't have control over passes `RTLD_GLOBAL` to `dlopen`. My prediction is that if you switch to using `RTLD_LOCAL`, then you won't need `RTLD_DEEPBIND` in 99% of cases. Also, I played with `RTLD_NEXT`, to see if I could locate the last symbol loaded, but I couldn't figure out when I had reached the end and I also wasn't able to lookup the symbol in the executable that way. I wish I understood what they mean by next. Thanks! -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:38> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: patch Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by dagit): After playing with your example a bit I see what you mean about the copy semantics. I found this article: http://netwinder.osuosl.org/users/p/patb/public_html/elf_relocs.html Interestingly, [http://src.gnu-darwin.org/src/libexec/rtld- elf/amd64/reloc.c.html glibc] has these comments: {{{ /* * Process the special R_X86_64_COPY relocations in the main program. These * copy data from a shared object into a region in the main program's BSS * segment. * * Returns 0 on success, -1 on failure. */ int do_copy_relocations(Obj_Entry *dstobj) }}} And: {{{ case R_X86_64_COPY: /* * These are deferred until all other relocations have * been done. All we do here is make sure that the COPY * relocation is not in a shared library. They are allowed * only in executable files. */ }}} The elf interpreter delays processing of the `R_COPY`'s so that it gets the correct behavior. I think this supports your earlier hypothesis. At this point, I noticed that I can switch `environ` in the executable between `R_X86_64_GLOB_DAT` and `R_X86_64_COPY` by adding `-fPIC` (you probably already knew that). Specifically, without `-fPIC` I get copy and with it I get glob dat. Test program again: {{{ #define _GNU_SOURCE #include <dlfcn.h> #include <stdio.h> #include <stdlib.h> extern char**environ; char envname[] = "environ"; void dlsym_check(void *handle, char * lib, char * sym){ void * e = dlsym(handle,sym); fprintf(stderr, "dlsym(\"%s\", \"%s\") = %p\n", lib, sym, e); char * error = dlerror(); if( error != NULL ) fprintf(stderr, "Errors: %s\n", error); } int main(int argc, char *argv[]) { if( argc < 2 ){ printf("usage: ./check-environ <path to shared object>\n"); exit(1); } char* so = argv[1]; void *deflt, *hdl; fprintf(stderr, "environ = %p\n", &environ); deflt = dlopen(NULL, RTLD_LAZY | RTLD_GLOBAL); fprintf(stderr, "\nFind a symbol after passing NULL to dlopen:\n"); dlsym_check(deflt, NULL, envname); hdl = dlopen(so, RTLD_LAZY | RTLD_GLOBAL); if( hdl == NULL ){ fprintf(stderr, "Error: %s", dlerror()); exit(1); } fprintf(stderr, "\nFind a symbol that is located here and libc (handle comes from dlopen on %s)\n", so); dlsym_check(hdl, so, envname); } }}} {{{ $ ./check-environ-without-pic /usr/lib64/libgmp.so.10 environ = 0x601070 Find a symbol after passing NULL to dlopen: dlsym("(null)", "environ") = 0x601070 Find a symbol that is located here and libc (handle comes from dlopen on /usr/lib64/libgmp.so.10) dlsym("/usr/lib64/libgmp.so.10", "environ") = 0x31e45bd508 $ ./check-environ-with-pic /usr/lib64/libgmp.so.10 environ = 0x31e45bd508 Find a symbol after passing NULL to dlopen: dlsym("(null)", "environ") = 0x31e45bd508 Find a symbol that is located here and libc (handle comes from dlopen on /usr/lib64/libgmp.so.10) dlsym("/usr/lib64/libgmp.so.10", "environ") = 0x31e45bd508 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:39> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: patch Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by trommler): Thanks for the great examples! Coming back to comment:31 where I reported two failing tests: I think some changes in the creation of the SOs that ghci loads are required. Could you tell me where in the code those SOs are created? May I temporarily own the ticket for that fix? -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:40> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: trommler Type: bug | Status: patch Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Changes (by simonmar): * owner: simonmar => trommler Comment: @trommler, giving you the ticket. Thinking about it, I'm less sure that `RTLD_LOCAL` will work now, because it won't allow us to load new libraries that depend on libraries we've previously loaded, and we definitely need to be able to do that. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:41> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: trommler Type: bug | Status: patch Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by trommler): Replying to [comment:41 simonmar]:
@trommler, giving you the ticket. Thanks!
Thinking about it, I'm less sure that `RTLD_LOCAL` will work now, because it won't allow us to load new libraries that depend on libraries we've previously loaded, and we definitely need to be able to do that. For libraries (SOs) coming from installed packages everything should be fine. The have a `NEEDED` tag for all dependencies and the system linker will `dlopen()` them and resolve the symbols.
So for new libraries temporary SOs need to be produced the same way SOs in packages are, with `NEEDED` tags for all previously loaded libraries. The new library could potentially depend on all previously loaded libraries. I am thinking of something along the lines of: {{{ ld -shared temp.o ... < -l for each library loaded so far > }}} Perhaps we could remove libraries from the list that are not needed by the new library by using `--as-needed` but I want to check first if this flag is GNU ld specific. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:42> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: trommler Type: bug | Status: patch Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by simonmar):
For libraries (SOs) coming from installed packages everything should be fine. The have a NEEDED tag for all dependencies and the system linker will dlopen() them and resolve the symbols.
But won't that reload all the dependencies every time we load a library? We'll end up with multiple copies of `ghc-prim`, `base`, etc. etc. It's not just a question of wasted memory, but multiple copies of `base` causes problems because we have multiple `stdout`s and other CAFs. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:43> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: trommler Type: bug | Status: patch Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by trommler): No, the library would not be loaded again. POSIX says: Only a single copy of an object file is brought into the address space, even if dlopen() is invoked multiple times in reference to the file, and even if different pathnames are used to reference the file. The Linux man page: If the same library is loaded again with dlopen(), the same file handle is returned. The dl library maintains reference counts for library handles, so a dynamic library is not deallocated until dlclose() has been called on it as many times as dlopen() has succeeded on it. The _init() routine, if present, is called only once. But a subsequent call with RTLD_NOW may force symbol resolution for a library earlier loaded with RTLD_LAZY. So I think we will be good. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:44> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: trommler Type: bug | Status: patch Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by simonmar): I didn't believe you, so I made a test program, and now I believe you :) It's somewhat surprising that a library we loaded with `RTLD_LOCAL` can be used to satisfy a dependency of another library we loaded with `RTLD_LOCAL`, but it does indeed seem to be the case. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:45> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: trommler Type: bug | Status: patch Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by dagit): Simon, that is surprising. Do you still have your test setup? Could I take a look? -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:46> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: trommler Type: bug | Status: patch Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by simonmar): `bar.c`: {{{ static int bar = 1; int _getbar(void) { return bar; } void _setbar(int x) { bar = x; } }}} `foo.c`: {{{ extern int _getbar(void); extern void _setbar(int); int getbar(void) { return _getbar(); } int setbar(int x) { _setbar(x); } }}} `test.c`: {{{ #include <dlfcn.h> #include <stdio.h> #include <stdlib.h> extern int getbar(void); extern void setbar(int); int main(int argc, char *argv[]) { void *deflt, *hdl; int (*pgetbar)(void); int (*psetbar)(int); setbar(2); printf("getbar() = %d\n", getbar()); deflt = dlopen(NULL, RTLD_LAZY | RTLD_GLOBAL); if (deflt == NULL) { printf("%s\n", dlerror()); exit(1); } pgetbar = dlsym(deflt, "getbar"); printf("dlsym(deflt, \"getbar\") = %p, pgetbar() = %d\n", pgetbar, (*pgetbar)()); hdl = dlopen("libbar.so", RTLD_LOCAL | RTLD_LAZY); if (hdl == NULL) { printf("%s\n", dlerror()); exit(1); } pgetbar = dlsym(hdl, "_getbar"); psetbar = dlsym(hdl, "_setbar"); printf("dlsym(deflt, \"_getbar\") = %p, pgetbar() = %d\n", pgetbar, (*pgetbar)()); (*psetbar)(3); printf("(*psetbar)(3); pgetbar() = %d\n", (*pgetbar)()); hdl = dlopen("libfoo.so", RTLD_LOCAL | RTLD_LAZY); if (hdl == NULL) { printf("%s\n", dlerror()); exit(1); } pgetbar = dlsym(hdl, "getbar"); psetbar = dlsym(hdl, "setbar"); printf("dlsym(deflt, \"getbar\") = %p, pgetbar() = %d\n", pgetbar, (*pgetbar)()); (*psetbar)(4); printf("(*psetbar)(4); pgetbar() = %d\n", (*pgetbar)()); } }}} {{{ $ gcc -shared -fPIC bar.c -o libbar.so $ gcc -shared -fPIC foo.c -o libfoo.so -L. -lbar $ gcc test.c -L. -lfoo -lbar -ldl -g $ LD_LIBRARY_PATH=. ./a.out getbar() = 2 dlsym(deflt, "getbar") = 0x2b4660dcc61c, pgetbar() = 2 dlsym(deflt, "_getbar") = 0x2b46615a658c, pgetbar() = 2 (*psetbar)(3); pgetbar() = 3 dlsym(deflt, "getbar") = 0x2b4660dcc61c, pgetbar() = 3 (*psetbar)(4); pgetbar() = 4 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:47> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: trommler Type: bug | Status: patch Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by dagit): I think you test depends on how you link the program and libraries. I'm using your setup, but I made the following tweaks. `test.c`, in this version `dlopen` of libbar is `RTLD_GLOBAL`: {{{ #include <dlfcn.h> #include <stdio.h> #include <stdlib.h> extern int getbar(void); extern void setbar(int); int main(int argc, char *argv[]) { void *deflt, *hdl; int (*pgetbar)(void); int (*psetbar)(int); //setbar(2); //fprintf(stderr,"getbar() = %d\n", getbar()); deflt = dlopen(NULL, RTLD_LAZY | RTLD_GLOBAL); if (deflt == NULL) { fprintf(stderr,"%s\n", dlerror()); exit(1); } //pgetbar = dlsym(deflt, "getbar"); //fprintf(stderr,"dlsym(deflt, \"getbar\") = %p, pgetbar() = %d\n", pgetbar, (*pgetbar)()); hdl = dlopen("libbar.so", RTLD_GLOBAL | RTLD_LAZY); if (hdl == NULL) { fprintf(stderr,"%s\n", dlerror()); exit(1); } pgetbar = dlsym(hdl, "_getbar"); char* e; e = dlerror(); if( e != NULL ){ fprintf(stderr,"%s\n", e); } psetbar = dlsym(hdl, "_setbar"); e = dlerror(); if( e != NULL ){ fprintf(stderr,"%s\n", e); } fprintf(stderr,"dlsym(deflt, \"_getbar\") = %p, pgetbar() = %d\n", pgetbar, (*pgetbar)()); (*psetbar)(3); fprintf(stderr,"(*psetbar)(3); pgetbar() = %d\n", (*pgetbar)()); hdl = dlopen("libfoo.so", RTLD_LOCAL | RTLD_LAZY); if (hdl == NULL) { fprintf(stderr,"%s\n", dlerror()); exit(1); } pgetbar = dlsym(hdl, "getbar"); e = dlerror(); if( e != NULL ){ fprintf(stderr,"%s\n", e); } psetbar = dlsym(hdl, "setbar"); e = dlerror(); if( e != NULL ){ fprintf(stderr,"%s\n", e); } fprintf(stderr,"dlsym(deflt, \"getbar\") = %p, pgetbar() = %d\n", pgetbar, (*pgetbar)()); (*psetbar)(4); fprintf(stderr,"(*psetbar)(4); pgetbar() = %d\n", (*pgetbar)()); return 0; } }}} Here, I don't link together the libraries at compile time, not even for `test.c` (I'm trying to be careful not to call this static linking, because as we'll see later it's not): {{{ gcc -shared -fPIC bar.c -o libbar.so gcc -shared -fPIC foo.c -o libfoo.so gcc -Wall test.c -ldl -g LD_LIBRARY_PATH=. ./a.out }}} Now when I run the program I get: {{{ dlsym(deflt, "_getbar") = 0x7fddcc3a5698, pgetbar() = 1 (*psetbar)(3); pgetbar() = 3 dlsym(deflt, "getbar") = 0x7fddcc1a3728, pgetbar() = 3 (*psetbar)(4); pgetbar() = 4 }}} Okay, as we would expect libfoo uses the definition in libbar. Now, switch the `dlopen` to `RTLD_LOCAL`, and run it again: {{{ dlsym(deflt, "_getbar") = 0x7f056f698698, pgetbar() = 1 (*psetbar)(3); pgetbar() = 3 ./a.out: symbol lookup error: ./libfoo.so: undefined symbol: _getbar }}} Now if we link libbar and the final program: {{{ gcc -shared -fPIC bar.c -o libbar.so gcc -shared -fPIC foo.c -o libfoo.so gcc -Wall test.c -L. -lbar -ldl -g LD_LIBRARY_PATH=. ./a.out dlsym(deflt, "_getbar") = 0x7f75b9914698, pgetbar() = 1 (*psetbar)(3); pgetbar() = 3 dlsym(deflt, "getbar") = 0x7f75b96f9728, pgetbar() = 3 (*psetbar)(4); pgetbar() = 4 }}} Now it finds the symbol again. The difference seems to be this: {{{ readelf -Wa a.out.withbar | grep NEEDED 0x0000000000000001 (NEEDED) Shared library: [libbar.so] 0x0000000000000001 (NEEDED) Shared library: [libdl.so.2] 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] readelf -Wa a.out.withoutbar | grep NEEDED 0x0000000000000001 (NEEDED) Shared library: [libdl.so.2] 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] }}} In other words, mentioning libbar on the gcc line for `test.c` adds an entry that the elf interpreter sees and it resolves the symbol, lazily, before we get into main. I would expect the same behavior if libfoo had been linked to libbar instead of with the main application. And indeed, that's what we get: {{{ gcc -shared -fPIC bar.c -o libbar.so gcc -shared -fPIC foo.c -o libfoo.so -L. -lbar gcc -Wall test.c -ldl -g -o a.out.withoutbar LD_LIBRARY_PATH=. ./a.out.withoutbar dlsym(deflt, "_getbar") = 0x7fc75e77c698, pgetbar() = 1 (*psetbar)(3); pgetbar() = 3 dlsym(deflt, "getbar") = 0x7fc75e57a728, pgetbar() = 3 (*psetbar)(4); pgetbar() = 4 readelf -Wa libfoo.so | grep NEEDED 0x0000000000000001 (NEEDED) Shared library: [libbar.so] 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:48> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: trommler Type: bug | Status: patch Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by simonmar): @dagit: that's right, the point of that test is to show that a library loaded with `RTLD_LOCAL` is used to satisfy a dependency of another library loaded with `RTLD_LOCAL`. The dependency is expressed by the `DT_NEEDED` tag, and is set up when you link the library. @trommler's point earlier was that if we use `RTLD_LOCAL` then we'll need to ensure that all the dependencies are specified with `DT_NEEDED` tags. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:49> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: trommler Type: bug | Status: patch Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by dagit): Excellent. I must have missed trommler saying that. Thanks! -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:50> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: patch Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Changes (by trommler): * owner: trommler => simonmar Comment: The attached patches change the way shared libraries are loaded (dlopen) by the runtime system and the way temporary shared libraries are linked for files compiled to object code in GHCi. The first patch [https://ghc.haskell.org/trac/ghc/attachment/ticket/8935/0001-Fix-GHCi- crash-on-access-to-process-environment.patch] implements the changes for Linux and I validated those on x86_64 (openSUSE 13.1). The second patch [https://ghc.haskell.org/trac/ghc/attachment/ticket/8935/0001-Fix-GHCi- linking-of-files-compiled-to-object-code.patch] might be needed for OS X. This patch is not validated as I do not have access to an OS X machine at the moment. Simon, please have a look at the patches and the commit messages and let me know what could be improved. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:51> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: patch Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by Simon Marlow <marlowsd@…>): In [changeset:"2f8b4c9330b455d4cb31c186c747a7db12a69251/ghc"]: {{{ #!CommitTicketReference repository="ghc" revision="2f8b4c9330b455d4cb31c186c747a7db12a69251" Fix obscure problem with using the system linker (#8935) See Note [RTLD_LOCAL] for a summary of the problem and solution, and }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:52> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: merge Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Changes (by simonmar): * status: patch => merge Comment: Committed with a few changes and comments by me. I'm not sure whether the second patch is required, but I refactored that bit anyway. Please re- open if there's still a problem. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:53> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: closed Priority: high | Milestone: 7.8.3 Component: Runtime System | Version: 7.8.1-rc2 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Changes (by thoughtpolice): * status: merge => closed * resolution: => fixed Comment: Merged. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:54> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: Type: bug | Status: new Priority: high | Milestone: 7.8.4 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: #9186 -------------------------------------+------------------------------------ Changes (by thoughtpolice): * owner: simonmar => * status: closed => new * resolution: fixed => * related: => #9186 * milestone: 7.8.3 => 7.8.4 Comment: Re-opening, the patch to fix this (2f8b4c9330b455d4cb31c186c747a7db12a69251) caused the failure in #9186, which has been reverted on the GHC 7.8 branch. I'll be reverting it on HEAD as well soon. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:55> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: new Priority: high | Milestone: 7.8.4 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: #9186 -------------------------------------+------------------------------------ Changes (by thoughtpolice): * owner: => simonmar -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:56> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------ Reporter: simonmar | Owner: simonmar Type: bug | Status: new Priority: high | Milestone: 7.8.4 Component: Runtime System | Version: 7.8.1-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: GHCi crash | Difficulty: Rocket Science Test Case: | Blocked By: Blocking: | Related Tickets: #9186 -------------------------------------+------------------------------------ Comment (by Austin Seipp <austin@…>): In [changeset:"aed1723f97e0539d5ab35222b180c1552a5f4cfc/ghc"]: {{{ #!CommitTicketReference repository="ghc" revision="aed1723f97e0539d5ab35222b180c1552a5f4cfc" Revert "Fix obscure problem with using the system linker (#8935)" This reverts commit 2f8b4c9330b455d4cb31c186c747a7db12a69251. Signed-off-by: Austin Seipp <austin@well-typed.com> }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:57> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------- Reporter: simonmar | Owner: simonmar Type: bug | Status: new Priority: high | Milestone: 7.8.4 Component: Runtime | Version: 7.8.1-rc2 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Rocket Science Unknown/Multiple | Blocked By: Type of failure: GHCi crash | Related Tickets: #9186, #9480 Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Changes (by rwbarton): * related: #9186 => #9186, #9480 Comment: I think the patch to fix this (which has been reverted) would fix #9480, too. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:58> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------- Reporter: simonmar | Owner: simonmar Type: bug | Status: new Priority: high | Milestone: 7.8.4 Component: Runtime | Version: 7.8.1-rc2 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Rocket Science Unknown/Multiple | Blocked By: Type of failure: GHCi crash | Related Tickets: #9186, #9480 Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Changes (by trommler): * cc: trommler (added) -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:59> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------- Reporter: simonmar | Owner: trommler Type: bug | Status: new Priority: high | Milestone: 7.10.1 Component: Runtime | Version: 7.8.1-rc2 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Rocket Science Unknown/Multiple | Blocked By: Type of failure: GHCi crash | Related Tickets: #9186, #9480 Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Changes (by trommler): * owner: simonmar => trommler Comment: I have a fix for [changeset:2f8b4c9330b455d4cb31c186c747a7db12a69251] that validates and also does not introduce the regression reported in #9186. I'll clean up the patch and post it on phabricator. Analysis: On some systems the order of linker options and especially the order libraries are passed to `ld` matters. That behaviour is caused by a patch binutils to change the default of `--no-as-needed` to `--as-needed`. The library ordering issue is the reason for the regression we saw in #9186. On openSUSE you can control the default behaviour of `ld` by setting `SUSE_ASNEEDED=1` in the environment. Given the name of the environment variable I doubt it works on other distributions :-) -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:61> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------- Reporter: simonmar | Owner: trommler Type: bug | Status: patch Priority: high | Milestone: 7.10.1 Component: Runtime | Version: 7.8.1-rc2 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Rocket Science Unknown/Multiple | Blocked By: Type of failure: GHCi crash | Related Tickets: #9186, #9480 Test Case: | Blocking: | Differential Revisions: Phab:D349 | -------------------------------------+------------------------------------- Changes (by trommler): * status: new => patch * differential: => Phab:D349 Comment: Here is my [Phab:D349 patch]. I validated on openSUSE 13.1 running on x86_64. Additionally I back ported the patch to 7.8.3 built, all of Haskell Platform and pandoc and verified I can build compdata-0.8.1.0 -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:62> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------- Reporter: simonmar | Owner: Type: bug | Status: patch Priority: high | Milestone: 7.10.1 Component: Runtime | Version: 7.8.1-rc2 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Rocket Science Unknown/Multiple | Blocked By: Type of failure: GHCi crash | Related Tickets: #9186, #9480 Test Case: | Blocking: | Differential Revisions: Phab:D349 | -------------------------------------+------------------------------------- Changes (by trommler): * owner: trommler => -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:63> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------- Reporter: simonmar | Owner: Type: bug | Status: patch Priority: high | Milestone: 7.10.1 Component: Runtime | Version: 7.8.1-rc2 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Rocket Science Unknown/Multiple | Blocked By: Type of failure: GHCi crash | Related Tickets: #9186, #9480 Test Case: | Blocking: | Differential Revisions: Phab:D349 | -------------------------------------+------------------------------------- Comment (by Austin Seipp <austin@…>): In [changeset:"383733b9191a36e2d3f757700842dbc3855911d9/ghc"]: {{{ #!CommitTicketReference repository="ghc" revision="383733b9191a36e2d3f757700842dbc3855911d9" Fix obscure problem with using the system linker (#8935) Summary: In a statically linked GHCi symbol `environ` resolves to NULL when called from a Haskell script. When resolving symbols in a Haskell script we need to search the executable program and its dependent (DT_NEEDED) shared libraries first and then search the loaded libraries. We want to be able to override functions in loaded libraries later. Libraries must be opened with local scope (RTLD_LOCAL) and not global. The latter adds all symbols to the executable program's symbols where they are then searched in loading order. We want reverse loading order. When libraries are loaded with local scope the dynamic linker cannot use symbols in that library when resolving the dependencies in another shared library. This changes the way files compiled to object code must be linked into temporary shared libraries. We link with the last temporary shared library created so far if it exists. Since each temporary shared library is linked to the previous temporary shared library the dynamic linker finds the latest definition of a symbol by following the dependency chain. See also Note [RTLD_LOCAL] for a summary of the problem and solution. Cherry-picked commit 2f8b4c Changed linker argument ordering On some ELF systems GNU ld (and others?) default to --as-needed and the order of libraries in the link matters. The last temporary shared library, must appear before all other libraries. Switching the position of extra_ld_inputs and lib_path_objs does that. Fixes #8935 and #9186 Reviewers: austin, hvr, rwbarton, simonmar Reviewed By: simonmar Subscribers: thomie, carter, simonmar Differential Revision: https://phabricator.haskell.org/D349 GHC Trac Issues: #8935, #9186, #9480 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:64> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------- Reporter: simonmar | Owner: Type: bug | Status: closed Priority: high | Milestone: 7.10.1 Component: Runtime | Version: 7.8.1-rc2 System | Keywords: Resolution: fixed | Architecture: Unknown/Multiple Operating System: | Difficulty: Rocket Science Unknown/Multiple | Blocked By: Type of failure: GHCi crash | Related Tickets: #9186, #9480 Test Case: | Blocking: | Differential Revisions: Phab:D349 | -------------------------------------+------------------------------------- Changes (by thoughtpolice): * status: patch => closed * resolution: => fixed Comment: OK, marking as fixed (crosses fingers). -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:65> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#8935: Obscure linker bug leads to crash in GHCi -------------------------------------+------------------------------------- Reporter: simonmar | Owner: Type: bug | Status: closed Priority: high | Milestone: 7.10.1 Component: Runtime System | Version: 7.8.1-rc2 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: GHCi crash | Unknown/Multiple Blocked By: | Test Case: Related Tickets: #9186, #9480 | Blocking: | Differential Revisions: Phab:D349 -------------------------------------+------------------------------------- Comment (by Austin Seipp <austin@…>): In [changeset:"0fcc454329c4e3e0dc4474412bff599d0e9bdfcd/ghc"]: {{{ #!CommitTicketReference repository="ghc" revision="0fcc454329c4e3e0dc4474412bff599d0e9bdfcd" Dynamically link all loaded packages in new object Summary: As a result of fixing #8935 we needed to open shared libraries with RTLD_LOCAL and so symbols from packages loaded earlier cannot be found anymore. We need to include in the link all packages loaded so far. This fixes #10058 Test Plan: validate Reviewers: hvr, simonmar, austin Reviewed By: austin Subscribers: rwbarton, thomie Differential Revision: https://phabricator.haskell.org/D676 GHC Trac Issues: #10058 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8935#comment:66> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
participants (1)
-
GHC