[GHC] #10229: setThreadAffinity assumes a certain CPU virtual core layout

2 Apr 2015

      #10229: setThreadAffinity assumes a certain CPU virtual core layout
-------------------------------------+-------------------------------------
              Reporter:  nh2         |             Owner:  simonmar
                  Type:  bug         |            Status:  new
              Priority:  normal      |         Milestone:
             Component:  Runtime     |           Version:  7.10.1
  System                             |  Operating System:  Unknown/Multiple
              Keywords:              |   Type of failure:  Runtime
          Architecture:              |  performance bug
  Unknown/Multiple                   |        Blocked By:
             Test Case:              |   Related Tickets:
              Blocking:              |
Differential Revisions:              |
-------------------------------------+-------------------------------------
 The {{{RTS -qa}}} option that can set thread affinity was implemented in
 https://git.haskell.org/ghc.git/commitdiff/31caec794c3978d55d79f715f21fb7294...

 {{{
 // Schedules the thread to run on CPU n of m.  m may be less than the
 // number of physical CPUs, in which case, the thread will be allowed
 // to run on CPU n, n+m, n+2m etc.
 void
 setThreadAffinity (nat n, nat m)
 }}}

 Today I discovered that on some machines, this option helps parallel
 performance (e.g. {{{+RTS -N4}}}) a lot, while on others it doesn't.

 Together with thomie on #ghc, I found out the reason:

 Lets assume I have 4 real cores with hyperthreading, so 8 virtual cores.

 The mapping of hyperthreading cores to physical cores is different across
 machines.

 On my one machine (Intel i5), the layout is 11223344, meaning that the
 first two vCPUs (hyperthreads) that the OS announces (visible e.g. in
 HTOP) map to the first physical core in the system, and so on.

 On my other machine (Intel Xeon), the layout is 12341234; here the 1st and
 the 5th vCPU map to the same physical core.

 This layout can be (on Linux) observed by running:

 {{{
 cat /proc/cpuinfo|egrep "processor|physical id|core id" |sed
 's/^processor/\nprocessor/g'
 }}}

 I do not know whether this layout is dictated by the processor, chosen by
 the OS, or even changing across reboots; what is clear is that the layout
 can vary across machines.

 Now, as explained by thomie:

 {{{
 -qa will set your 4 capabilities to cores [(1,5), (2,6), (3,7), (4,8)],
 and then the os randomly chooses out of those tuples
 }}}

 This strategy is optimal for the 12341234 layout; for example, when
 running with -N4, it ensures that two threads are not scheduled onto vCPUs
 that are on the same physical core. The possible {{{+RTS -aq}}} choice
 {{{1__4_23_}}} is a great assignment in this case, as is {{{1234____}}}
 ({{{_}}} means the vCPU is not chosen).

 But for the 11223344, the choice {{{1234____}}} isn't good, because it
 uses only 2 of our 4 physical cores; our program now takes twice as long
 to run.

 ----

 It seems likely to me that {{{setThreadAffinity}}} was written on a
 machine with 12341234 layout, and with the assumption that all machines
 have this layout.

 It would be great if we could change it to take the actual layout into
 account.

--
Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10229
GHC http://www.haskell.org/ghc/
The Glasgow Haskell Compiler

GHC

GHC

GHC

GHC

GHC

GHC

GHC

tags

participants (1)