On 27/03/2012 01:14, Sajith T S wrote:

Hi Simon,

Thanks for the reply. It seems that forwarding the message here was a very good idea!

Simon Marlow wrote:

...

...

-- From a very recent discussion on parallel-haskell [4], we learn that RTS' NUMA support could be improved. The hypothesis is that allocating nurseries per Capability might be a better plan than using global pool. We might borrow/steal ideas from hwloc [5] for this.

I like this idea too (since I suggested it :-).

I guess you will also be available for eventual pestering about this stuff, then? :)

Sure. Do you have a NUMA machine to test on? Cheers, Simon

On March 28, 2012 04:41:16 Simon Marlow wrote:

Sure. Do you have a NUMA machine to test on?

My understanding is non-NUMA machines went away when the AMD and Intel moved away from frontside buses (FSB) and integrated the memory controllers on die. Intel is more recent to this game. I believe AMD's last non-NUMA machines where the Athalon XP series and Intel's the Core 2 series. An easy way to see what you've got is to see what 'numactl --hardware' says. If the node distance matrix is not uniform, you have NUMA hardware. As an example, on a 8 socket Opteron machine (32 cores) you get $ numactl --hardware available: 8 nodes (0-7) node 0 size: 16140 MB node 0 free: 3670 MB node 1 size: 16160 MB node 1 free: 3472 MB node 2 size: 16160 MB node 2 free: 4749 MB node 3 size: 16160 MB node 3 free: 4542 MB node 4 size: 16160 MB node 4 free: 3110 MB node 5 size: 16160 MB node 5 free: 1963 MB node 6 size: 16160 MB node 6 free: 1715 MB node 7 size: 16160 MB node 7 free: 2862 MB node distances: node 0 1 2 3 4 5 6 7 0: 10 20 20 20 20 20 20 20 1: 20 10 20 20 20 20 20 20 2: 20 20 10 20 20 20 20 20 3: 20 20 20 10 20 20 20 20 4: 20 20 20 20 10 20 20 20 5: 20 20 20 20 20 10 20 20 6: 20 20 20 20 20 20 10 20 7: 20 20 20 20 20 20 20 10 On our more traditional NUMA there are 64 nodes and the numbers range from 10-37. But it's an older SGI Itanium solution, so that comes with its own set of problems, and most modern machines are already out performing it. Cheers! -Tyson

On March 28, 2012 12:40:02 Sajith T S wrote:

Tyson Whitehead wrote:

...

Intel is more recent to this game. I believe AMD's last non-NUMA machines where the Athalon XP series and Intel's the Core 2 series.

An easy way to see what you've got is to see what 'numactl --hardware' says.

Ah, thanks. I trust this one qualifies?

$ numactl --hardware available: 4 nodes (0-3) node 0 cpus: 0 4 8 12 16 20 24 28 node 0 size: 16370 MB node 0 free: 14185 MB node 1 cpus: 1 5 9 13 17 21 25 29 node 1 size: 16384 MB node 1 free: 10071 MB node 2 cpus: 2 6 10 14 18 22 26 30 node 2 size: 16384 MB node 2 free: 14525 MB node 3 cpus: 3 7 11 15 19 23 27 31 node 3 size: 16384 MB node 3 free: 13598 MB node distances: node 0 1 2 3 0: 10 20 20 20 1: 20 10 20 20 2: 20 20 10 20 3: 20 20 20 10

Yup. For sure. Here's an example from a 4 socket 16 core non-NUMA Intel Xeon $ numactl --hardware available: 1 nodes (0) node 0 size: 129260 MB node 0 free: 1304 MB node distances: node 0 0: 10 On a NUMA system I believe you should be able to get an idea of the worst case penality a program experiences due to having to do it's memory acesses all go across the QuickPath Interconnect (Intel)/HyperTransport (AMD) by forcing it to execute on one socket while using the memory on another. $ numactl --cpunodebind=0 --membind=1 <program> There is also some good information under /proc/<PID>/numa_maps. See the man page for details, but basically it tells you how many pages are associated with each node for each part of the programs address space.. Note that for file backed pages, they don't always reside in the limited node due to the system already having mapped them into memory on another node for an earlier process. Appoligies if you are already familair with these items. Cheers! -Tyson PS: That looks like a pretty sweet box you've got going there. :)

On 28/03/2012 16:57, Tyson Whitehead wrote:

On March 28, 2012 04:41:16 Simon Marlow wrote:

...

Sure. Do you have a NUMA machine to test on?

My understanding is non-NUMA machines went away when the AMD and Intel moved away from frontside buses (FSB) and integrated the memory controllers on die.

Intel is more recent to this game. I believe AMD's last non-NUMA machines where the Athalon XP series and Intel's the Core 2 series.

An easy way to see what you've got is to see what 'numactl --hardware' says. If the node distance matrix is not uniform, you have NUMA hardware.

As an example, on a 8 socket Opteron machine (32 cores) you get

$ numactl --hardware available: 8 nodes (0-7) node 0 size: 16140 MB node 0 free: 3670 MB node 1 size: 16160 MB node 1 free: 3472 MB node 2 size: 16160 MB node 2 free: 4749 MB node 3 size: 16160 MB node 3 free: 4542 MB node 4 size: 16160 MB node 4 free: 3110 MB node 5 size: 16160 MB node 5 free: 1963 MB node 6 size: 16160 MB node 6 free: 1715 MB node 7 size: 16160 MB node 7 free: 2862 MB node distances: node 0 1 2 3 4 5 6 7 0: 10 20 20 20 20 20 20 20 1: 20 10 20 20 20 20 20 20 2: 20 20 10 20 20 20 20 20 3: 20 20 20 10 20 20 20 20 4: 20 20 20 20 10 20 20 20 5: 20 20 20 20 20 10 20 20 6: 20 20 20 20 20 20 10 20 7: 20 20 20 20 20 20 20 10

Well, you learn something new every day! On the new 32-core Opteron box we have here: available: 8 nodes (0-7) node 0 cpus: 0 4 8 12 node 0 size: 8182 MB node 0 free: 1994 MB node 1 cpus: 16 20 24 28 node 1 size: 8192 MB node 1 free: 2783 MB node 2 cpus: 3 7 11 15 node 2 size: 8192 MB node 2 free: 2961 MB node 3 cpus: 19 23 27 31 node 3 size: 8192 MB node 3 free: 5359 MB node 4 cpus: 2 6 10 14 node 4 size: 8192 MB node 4 free: 3030 MB node 5 cpus: 18 22 26 30 node 5 size: 8192 MB node 5 free: 4667 MB node 6 cpus: 1 5 9 13 node 6 size: 8192 MB node 6 free: 3240 MB node 7 cpus: 17 21 25 29 node 7 size: 8192 MB node 7 free: 4031 MB node distances: node 0 1 2 3 4 5 6 7 0: 10 16 16 22 16 22 16 22 1: 16 10 16 22 22 16 22 16 2: 16 16 10 16 16 16 16 22 3: 22 22 16 10 16 16 22 16 4: 16 22 16 16 10 16 16 16 5: 22 16 16 16 16 10 22 22 6: 16 22 16 22 16 22 10 16 7: 22 16 22 16 16 22 16 10 The node distances on this box are less uniform than yours. Cheers, Simon