#7602: Threaded RTS performing badly on recent OS X (10.8?) ---------------------------------+------------------------------------------ Reporter: simonmar | Owner: Type: bug | Status: new Priority: normal | Milestone: _|_ Component: Runtime System | Version: 7.6.1 Keywords: | Os: Unknown/Multiple Architecture: Unknown/Multiple | Failure: None/Unknown Difficulty: Unknown | Testcase: Blockedby: | Blocking: Related: | ---------------------------------+------------------------------------------ Changes (by tibbe): * cc: johan.tibell@… (added) -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/7602#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#7602: Threaded RTS performing badly on recent OS X (10.8?) ---------------------------------+------------------------------------------ Reporter: simonmar | Owner: Type: bug | Status: new Priority: normal | Milestone: _|_ Component: Runtime System | Version: 7.6.1 Keywords: | Os: Unknown/Multiple Architecture: Unknown/Multiple | Failure: None/Unknown Difficulty: Unknown | Testcase: Blockedby: | Blocking: Related: | ---------------------------------+------------------------------------------ Comment(by simonmar): Does the support for TLS compile down to calls to `pthread_getspecific`/`pthread_getspecific`? If so, that won't help much - we could remove some hacks from the code, but it will still perform badly. Someone with a Mac will need to do some measurements to be sure. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/7602#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#7602: Threaded RTS performing badly on recent OS X (10.8?) ---------------------------------+------------------------------------------ Reporter: simonmar | Owner: Type: bug | Status: new Priority: normal | Milestone: _|_ Component: Runtime System | Version: 7.6.1 Keywords: | Os: Unknown/Multiple Architecture: Unknown/Multiple | Failure: None/Unknown Difficulty: Unknown | Testcase: Blockedby: | Blocking: Related: | ---------------------------------+------------------------------------------ Comment(by thoughtpolice): Ah, I misspoke in earnest without looking more deeply. It does look like the ```callq *(%rdi)``` is the indirection we're still hitting in some sense, although I'm not sure why it's different from apple's implementation in libc: with -O2, this is minimized to a simple load and call, but it's still a significant overhead for the access, at a glance. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/7602#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#7602: Threaded RTS performing badly on recent OS X (10.8?) ---------------------------------+------------------------------------------ Reporter: simonmar | Owner: Type: bug | Status: new Priority: normal | Milestone: _|_ Component: Runtime System | Version: 7.6.1 Keywords: | Os: Unknown/Multiple Architecture: Unknown/Multiple | Failure: None/Unknown Difficulty: Unknown | Testcase: Blockedby: | Blocking: Related: | ---------------------------------+------------------------------------------ Changes (by chak): * cc: chak@… (added) Comment: Let me just add that llvm-gcc will disappear from Xcode (command line tools) soon — i.e., we will need to use clang. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/7602#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#7602: Threaded RTS performing badly on recent OS X (10.8?) ---------------------------------+------------------------------------------ Reporter: simonmar | Owner: Type: bug | Status: new Priority: normal | Milestone: _|_ Component: Runtime System | Version: 7.6.1 Keywords: | Os: Unknown/Multiple Architecture: Unknown/Multiple | Failure: None/Unknown Difficulty: Unknown | Testcase: Blockedby: | Blocking: Related: | ---------------------------------+------------------------------------------ Comment(by thoughtpolice): Just as a note, I don't think GCC will help anymore; using GCC 4.7.2 on OSX 10.8, compiling my same 'foo' example earlier with ```gcc -O3```, I get: {{{ (lldb) disassemble -m -n main a.out`main a.out[0x100000f00]: cmpl $1, %edi a.out[0x100000f03]: pushq %rbx a.out[0x100000f04]: jle 0x100000f3f ; main + 63 a.out[0x100000f06]: movq 8(%rsi), %rdi a.out[0x100000f0a]: callq 0x100000f54 ; symbol stub for: atoi a.out[0x100000f0f]: leaq 330(%rip), %rdi ; __emutls_v.foo a.out[0x100000f16]: movl %eax, %ebx a.out[0x100000f18]: callq 0x100000f66 ; symbol stub for: __emutls_get_address a.out[0x100000f1d]: movl %ebx, (%rax) a.out[0x100000f1f]: leaq 314(%rip), %rdi ; __emutls_v.foo a.out[0x100000f26]: callq 0x100000f66 ; symbol stub for: __emutls_get_address a.out[0x100000f2b]: leaq 114(%rip), %rdi ; "foo = %d\n" a.out[0x100000f32]: movl (%rax), %esi a.out[0x100000f34]: xorl %eax, %eax a.out[0x100000f36]: callq 0x100000f60 ; symbol stub for: printf a.out[0x100000f3b]: xorl %eax, %eax a.out[0x100000f3d]: popq %rbx a.out[0x100000f3e]: ret a.out[0x100000f3f]: leaq 282(%rip), %rdi ; __emutls_v.foo a.out[0x100000f46]: callq 0x100000f66 ; symbol stub for: __emutls_get_address a.out[0x100000f4b]: movl $10, (%rax) a.out[0x100000f51]: jmp 0x100000f1f ; main + 31 }}} So I think for now we're going to have to just bite the bullet on this one, and make sure the build is solid with Clang on modern OS X anyway. Maybe we can do something evil here later to recover the loss :/ David Peixotto's original patches made the RTS build with clang at first. I'll run a test against HEAD using Clang and see what I find. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/7602#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#7602: Threaded RTS performing badly on recent OS X (10.8?) ---------------------------------+------------------------------------------ Reporter: simonmar | Owner: Type: bug | Status: new Priority: normal | Milestone: _|_ Component: Runtime System | Version: 7.6.1 Keywords: | Os: Unknown/Multiple Architecture: Unknown/Multiple | Failure: None/Unknown Difficulty: Unknown | Testcase: Blockedby: | Blocking: Related: | ---------------------------------+------------------------------------------ Comment(by thoughtpolice): Ah, you're very correct. I shouldn't comment on tickets when it's absurdly late at night and I'm really tired... Apple has advertised OS X as having TLS since Lion I believe. I think this really is TLS support. It's just slower than Linux - on Ubuntu 12.10, the example above moves values directly into ```%fs:0xfffffffffffffff0```. GCC is just weird here because it's calling into ```libgcc``` first from my digging (which would maybe mean it's even slower. I haven't tested.) Modern GCC is at least easily obtainable on OS X, so perhaps this isn't as bad and it's not worth doing something evil. We'll probably need to fix other things, so I'll still look into it. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/7602#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#7602: Threaded RTS performing badly on recent OS X (10.8?) ---------------------------------+------------------------------------------ Reporter: simonmar | Owner: Type: bug | Status: new Priority: normal | Milestone: _|_ Component: Runtime System | Version: 7.6.1 Keywords: | Os: Unknown/Multiple Architecture: Unknown/Multiple | Failure: None/Unknown Difficulty: Unknown | Testcase: Blockedby: | Blocking: Related: | ---------------------------------+------------------------------------------ Comment(by thoughtpolice): Alright, I think my patch is almost working, but in the mean time I've verified with a small snippet the behavior I think we want. Simon, can you please tell me if this approach would be OK? Essentially, there is a small set of predefined TLS keys in the OS X C library for various Apple-internal things. There are about 100 of these special keys. With them, it's possible to use very special inline variants of ```pthread_getspecific``` and ```pthread_setspecific``` that directly write into an offset block of the ```%gs``` register. Performance-wise, this should be very close to Linux's implementation. One of these things on modern OS X and its libc is WebKit. pthread has a specific range of keys (5 to be exact) dedicated to WebKit. These are used in JavaScriptCore's FastMalloc allocator for performance critical sections - likely for their GC! But only a single key is used by WebKit at all, and there are 0 references to it elsewhere that I can find on the internet. You can see this here: http://www.opensource.apple.com/source/Libc/Libc-825.25/pthreads/pthread_mac... This defines the inline get/set routines for special TLS keys. If you scroll down a little you can see the ```JavaScriptCore``` keys (keys 90-94 to be exact.) Now, look here: http://code.google.com/codesearch#mcaWan7Aaio/trunk/WebKit-r115846/Source/WTF/wtf/FastMalloc.cpp&q=__PTK_FRAMEWORK_JAVASCRIPTCORE_KEY0&type=cs&l=453 And you can see there's a special stubbed out ```pthread_getspecific``` and ```pthread_setspecific``` routine for this exact purpose. Therefore, I propose we steal one of the high TLS keys that dedicated to WebKit's JS engine for the GC. Unfortunately, ```pthread_machdep.h``` is not installed by default in modern variants of XCode, so we must inline the definitions ourselves for the necessary architectures. The following example demonstrates the use of these special keys: {{{ #include #include #include /** Snipped from pthread_machdep.h */ #define __PTK_FRAMEWORK_JAVASCRIPTCORE_KEY4 94 __inline__ void * _pthread_getspecific_direct(unsigned long slot) { void* ret; #if defined(__i386__) || defined(__x86_64__) __asm__("mov %%gs:%1, %0" : "=r" (ret) : "m" (*(void **)(slot * sizeof(void *)))); #else #error "No definition of pthread_getspecific_direct!" #endif return ret; } /* To be used with static constant keys only */ __inline__ static int _pthread_setspecific_direct(unsigned long slot, void * val) { #if defined(__x86_64__) /* PIC is free and cannot be disabled, even with: gcc -mdynamic-no-pic ... */ __asm__("movq %1,%%gs:%0" : "=m" (*(void **)(slot * sizeof(void *))) : "rn" (val)); #else #error "No definition of pthread_setspecific_direct!" #endif return 0; } /** End snippets */ static const pthread_key_t fooKey = __PTK_FRAMEWORK_JAVASCRIPTCORE_KEY4; #define GET_FOO() ((int)(_pthread_getspecific_direct(fooKey))) #define SET_FOO(to) (_pthread_setspecific_direct(fooKey, to)) int main(int ac, char* av[]) { if (ac < 2) SET_FOO((void*)10); else SET_FOO((void*)atoi(av[1])); printf("foo = %d\n", GET_FOO()); return 0; } }}} This is pretty close to what the GC does now. And compiling: {{{ $ clang -O3 tls2.c $ lldb ./a.out Current executable set to './a.out' (x86_64). (lldb) disassemble -m -n main a.out`main a.out[0x100000ef0]: pushq %rbp a.out[0x100000ef1]: movq %rsp, %rbp a.out[0x100000ef4]: cmpl $1, %edi a.out[0x100000ef7]: jg 0x100000f08 ; main + 24 a.out[0x100000ef9]: movq $10, %gs:752 a.out[0x100000f06]: jmp 0x100000f1d ; main + 45 a.out[0x100000f08]: movq 8(%rsi), %rdi a.out[0x100000f0c]: callq 0x100000f38 ; symbol stub for: atoi a.out[0x100000f11]: movslq %eax, %rax a.out[0x100000f14]: movq %rax, %gs:752 a.out[0x100000f1d]: movq %gs:752, %rsi a.out[0x100000f26]: leaq 59(%rip), %rdi ; "foo = %d\n" a.out[0x100000f2d]: xorb %al, %al a.out[0x100000f2f]: callq 0x100000f3e ; symbol stub for: printf a.out[0x100000f34]: xorl %eax, %eax a.out[0x100000f36]: popq %rbp a.out[0x100000f37]: ret (lldb) r Process 67488 launched: './a.out' (x86_64) foo = 10 Process 67488 exited with status = 0 (0x00000000) (lldb) ^D $ }}} This will probably only work on modern versions of XCode and OS X (10.8 etc.) In part, older libcs have very different implementations of ```pthread_setspecific_direct```, which means this could be very wrong on older machines. I'm not sure how much older, so if we had any 10.7 users who could try this that would be awesome. The build system will need modifications to check for that, and fall back to the much slower routines otherwise I suppose. Simon, does this approach sound OK? I think it will recover the performance loss here and we can just go ahead and use Clang, which is the easiest for everybody I think. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/7602#comment:13 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#7602: Threaded RTS performing badly on recent OS X (10.8?) -------------------------------+-------------------------------------------- Reporter: simonmar | Owner: thoughtpolice Type: bug | Status: new Priority: normal | Milestone: _|_ Component: Runtime System | Version: 7.7 Keywords: | Os: MacOS X Architecture: x86_64 (amd64) | Failure: None/Unknown Difficulty: Unknown | Testcase: Blockedby: 7678 | Blocking: Related: | -------------------------------+-------------------------------------------- Comment(by simonmar): Ok, so an inline `pthread_getspecific` is good, but it's still not ideal, because the compiler can't see the code inside the inline asm. Multiple references to the TLS variable should not cause repeated reads, but with the inline `pthread_getspecific`, they will. However, this can be worked around in the code by loading the TLS variable into a local once at the start of the function. I wouldn't object to this, it's less invasive than passing around the TLS variable as a parameter everywhere. But we should peer at the asm before and after to make sure it's doing what we expect. Stealing the WebKit-reserved slot would probably work for now, until some library developer has the same idea, and then we have a bizarre bug waiting to happen. I suppose I slightly prefer to use gcc for now, since we know it works, has good performance, and doesn't have this infelicity. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/7602#comment:16 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#7602: Threaded RTS performing badly on recent OS X (10.8?) -------------------------------+-------------------------------------------- Reporter: simonmar | Owner: thoughtpolice Type: bug | Status: new Priority: normal | Milestone: _|_ Component: Runtime System | Version: 7.7 Keywords: | Os: MacOS X Architecture: x86_64 (amd64) | Failure: None/Unknown Difficulty: Unknown | Testcase: Blockedby: 7678 | Blocking: Related: | -------------------------------+-------------------------------------------- Comment(by chak): Any news here? Did you check out the Xcode 5 DP? It's now clang or nothing. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/7602#comment:18 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

Sigh. I haven't seen XCode 5. The fact they EOL'd gcc finally isn't totally surprising.

The summary is that I have a patch for this and it should help the

#7602: Threaded RTS performing badly on recent OS X (10.8?) -------------------------------+-------------------------------------------- Reporter: simonmar | Owner: thoughtpolice Type: bug | Status: new Priority: normal | Milestone: _|_ Component: Runtime System | Version: 7.7 Keywords: | Os: MacOS X Architecture: x86_64 (amd64) | Failure: None/Unknown Difficulty: Unknown | Testcase: Blockedby: 7678 | Blocking: Related: | -------------------------------+-------------------------------------------- Comment(by chak): Replying to [comment:19 thoughtpolice]: performance loss quite a bit, but the blocker here is #7678 - clang's preprocessor doesn't like the way we use RULES, among other things.

This is work-around-able, but probably not in a very nice way

unfortunately. Our use of CPP was always a dirty hack and it was only a matter of time, before it was going to bite us. Can't we just disable cpp in modules with rules? -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/7602#comment:20 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#7602: Threaded RTS performing badly on recent OS X (10.8?) -------------------------------+-------------------------------------------- Reporter: simonmar | Owner: thoughtpolice Type: bug | Status: new Priority: normal | Milestone: _|_ Component: Runtime System | Version: 7.7 Keywords: | Os: MacOS X Architecture: x86_64 (amd64) | Failure: None/Unknown Difficulty: Unknown | Testcase: Blockedby: 7678 | Blocking: Related: | -------------------------------+-------------------------------------------- Comment(by thoughtpolice): Well, technically I think that under GCC's -traditional-cpp mode, clang SHOULD respect our code. The problem is I just don't think the "be insensitive to leading whitespace" rule is implemented. :( I don't think turning one or the other off is an option. Base in particular has a few modules with both CPP and Rules, but if you disable one or the other, things will break. Either the preprocessor directives become a syntax error, or you disable the RULES and they are ignored - but the preprocessor would error on them anyway because it runs first. There is another problem. We can change GHC and all of its dependent libraries to remove leading whitespace on lines that would be ambiguous to clang. Unfortunately, 3rd party libraries do this kind of formatting to, so this could break user programs for a very bizarre reason. I think it should be possible to fix this in a transparent way, but it won't be pretty. Basically, we'll need to preemptively strip out lines that begin with whitespace, where the first non whitespace character is #. Something like "s/^\s+\#(.*)/#$1/" or whatever in regex-ese. In the mean time I think it should be possible to get clang building by reformatting a few of libraries using RULES improperly, which should not be too much work I think. Finally, the various TLS models makes no difference on the example using __thread I posted above. I posit that TLS models are silently ignored on OS X, where Apple is free to do what they wish. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/7602#comment:22 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#7602: Threaded RTS performing badly on recent OS X (10.8?) -------------------------------+-------------------------------------------- Reporter: simonmar | Owner: thoughtpolice Type: bug | Status: new Priority: high | Milestone: 7.8.1 Component: Runtime System | Version: 7.7 Keywords: | Os: MacOS X Architecture: x86_64 (amd64) | Failure: None/Unknown Difficulty: Unknown | Testcase: Blockedby: 7678 | Blocking: Related: | -------------------------------+-------------------------------------------- Comment(by thoughtpolice): Right, understood. I'll put this on my plate and try to tackle it tonight. Just getting clang to build and recovering the performance I think won't be too hard. The preprocessing hack can happen afterwords, and I can file a separate ticket for it. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/7602#comment:24 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#7602: Threaded RTS performing badly on recent OS X (10.8?) -------------------------------+-------------------------------------------- Reporter: simonmar | Owner: thoughtpolice Type: bug | Status: new Priority: high | Milestone: 7.8.1 Component: Runtime System | Version: 7.7 Keywords: | Os: MacOS X Architecture: x86_64 (amd64) | Failure: None/Unknown Difficulty: Unknown | Testcase: Blockedby: 7678 | Blocking: Related: | -------------------------------+-------------------------------------------- Changes (by lelf): * cc: anton.nik@… (added) -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/7602#comment:26 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler