How to run parallel on dual socket machines ?

[grobalt]

I have several dual socket epyc nodes and some single socket. All same CPU, multiple TB of memory.
The 2 Socket servers run 240 threads and the 1 socket server run 120 threads.
Dual socket finishes in ~ 330 seconds in average and the single socket in ~ 280 seconds.

How can i assign 2 parallel threads, one to each socket on the 2 socket nodes ?

[grobalt]

i think i figured it out how it works on ubuntu LTS 20.04
example to run on CPU 1 (64 physical cores)
numactl -C +0-63,128-191 --membind=0 /home/*/bladebit/build/bladebit -m -n 1 -t 120 -f * -w -c * --no-cpu-affinity /media/storage

example to run on CPU 2 (64 physical cores)
numactl -C +64-127,192-255 --membind=1/home/*/bladebit/build/bladebit -m -n 1 -t 120 -f * -w -c * --no-cpu-affinity /media/storage

previously one process using all ressources between 5 and 7 minutes.
with numactl assignment each job finished under 5, so more than doubling the performance, 2 plots finished every 5 minutes

[grobalt]

Confirmed, working flawless: (2 processes in that log, so 1 plots per ~ 140 seconds)
*@chia12:~$ cat nohup.out | grep minutes
Finished plotting in 276.27 seconds (4.60 minutes).
Finished plotting in 273.68 seconds (4.56 minutes).
Finished plotting in 277.13 seconds (4.62 minutes).
Finished plotting in 273.94 seconds (4.57 minutes).
Finished plotting in 280.38 seconds (4.67 minutes).
Finished plotting in 279.67 seconds (4.66 minutes).
Finished plotting in 277.71 seconds (4.63 minutes).
Finished plotting in 274.00 seconds (4.57 minutes).
Finished plotting in 275.60 seconds (4.59 minutes).
Finished plotting in 280.05 seconds (4.67 minutes).
Finished plotting in 277.65 seconds (4.63 minutes).
Finished plotting in 279.64 seconds (4.66 minutes).
Finished plotting in 274.21 seconds (4.57 minutes).
Finished plotting in 276.93 seconds (4.62 minutes).
Finished plotting in 274.19 seconds (4.57 minutes).
Finished plotting in 276.62 seconds (4.61 minutes).
Finished plotting in 273.71 seconds (4.56 minutes).
Finished plotting in 277.50 seconds (4.63 minutes).
Finished plotting in 274.43 seconds (4.57 minutes).
Finished plotting in 277.96 seconds (4.63 minutes).
Finished plotting in 274.08 seconds (4.57 minutes).
Finished plotting in 277.14 seconds (4.62 minutes).
Finished plotting in 278.02 seconds (4.63 minutes).
Finished plotting in 280.41 seconds (4.67 minutes).

[harold-b]

Looks like you managed it on your own 🙂. Great timings, BTW

[grobalt]

yes, thanks.

i decided to map just 60 threads, 2 per socket, 4 per machine:

crazy numbers:

Finished plotting in 509.66 seconds (8.49 minutes).
Finished plotting in 506.80 seconds (8.45 minutes).
Finished plotting in 506.58 seconds (8.44 minutes).
Finished plotting in 504.49 seconds (8.41 minutes).
Finished plotting in 501.03 seconds (8.35 minutes).
Finished plotting in 502.83 seconds (8.38 minutes).
Finished plotting in 502.77 seconds (8.38 minutes).
Finished plotting in 507.51 seconds (8.46 minutes).
Finished plotting in 512.32 seconds (8.54 minutes).
Finished plotting in 501.67 seconds (8.36 minutes).
Finished plotting in 499.88 seconds (8.33 minutes).
Finished plotting in 504.13 seconds (8.40 minutes).
Finished plotting in 501.61 seconds (8.36 minutes).
Finished plotting in 500.75 seconds (8.35 minutes).

roughly every 10 second per plot :) (13 nodes)

nohup sudo numactl -C +0-31,128-159 --membind=0 /home/gts-data/bladebit/build/bladebit -m -n 2000 -t 60 -f *-w -c *--no-cpu-affinity /media/nvme &
sleep 120s
nohup sudo numactl -C +32-63,160-191 --membind=0 /home/gts-data/bladebit/build/bladebit -m -n 2000 -t 60 -f *-w -c *--no-cpu-affinity /media/nvme &
sleep 120s
nohup sudo numactl -C +64-95,192-223 --membind=1 /home/gts-data/bladebit/build/bladebit -m -n 2000 -t 60 -f *-w -c *--no-cpu-affinity /media/nvme &
sleep 120s
nohup sudo numactl -C +96-127,224-255 --membind=1 /home/gts-data/bladebit/build/bladebit -m -n 2000 -t 60 -f *-w -c *--no-cpu-affinity /media/nvme &

[tigana0210]

That's really fast, what's your cpu model? Do you overclock it?

[bladeuserpi]

Testing 2xE5-2680v4 with 1TB Ram (2400MHz) running two bladebits parallel

Each CPU has 14 physical cores (28HT): total 56 Threads 
lscpu |cat -n
     1	Architecture:        x86_64
     2	CPU op-mode(s):      32-bit, 64-bit
     3	Byte Order:          Little Endian
     4	CPU(s):              56
     5	On-line CPU(s) list: 0-55
     6	Thread(s) per core:  2
     7	Core(s) per socket:  14
     8	Socket(s):           2
     9	NUMA node(s):        2
    10	Vendor ID:           GenuineIntel
    11	BIOS Vendor ID:      Intel(R) Corporation
    12	CPU family:          6
    13	Model:               79
    14	Model name:          Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
    15	BIOS Model name:     Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
    16	Stepping:            1
    17	CPU MHz:             2893.321
    18	CPU max MHz:         3300.0000
    19	CPU min MHz:         1200.0000
    20	BogoMIPS:            4788.94
    21	L1d cache:           32K
    22	L1i cache:           32K
    23	L2 cache:            256K
    24	L3 cache:            35840K
    25	NUMA node0 CPU(s):   0-13,28-41
    26	NUMA node1 CPU(s):   14-27,42-55


Test 1) Baseline
================
One bladebit with 56 threads:
3635.25238.out:Finished plotting in 548.07 seconds (9.13 minutes).
That is 158 PPD (plots per day).

Test 2) Two parallel 
====================
a) numactl -C14-27,42-55  -m1  bladebit -t 28 --no-cpu-affinity  ...
b) sleep 30 ; numactl -C0-13,28-41 -m0 bladebit -t 28 --no-cpu-affinity ...

Result:
55238.11899.out:Finished plotting in 1095.08 seconds (18.25 minutes).
55326.24032.out:Finished plotting in 1095.82 seconds (18.26 minutes).

That is the same throughput as Test1: 2 plots each 18.2 minutes = 158PPD
It was expected the throughput should improve, but it stayed the same (at least did not drop)
 
Test 3) One Socket 
==================
Run only one job on a single socket:
numactl -C0-13,28-41 -m0 bladebit -t 28 --no-cpu-affinity

Result:
59035.15280.out:Finished plotting in 1064.80 seconds (17.75 minutes).

The run showed high RAMWatt for 2nd CPU when observing with:
watch -d -n0 turbostat -i1 -n1 --quiet --show Package,Core,CPU,Avg_MHz,TSC_MHz,CoreTmp,PkgTmp,PkgWatt,RAMWatt,PKG_%,RAM_%
Package Core    CPU     Avg_MHz TSC_MHz CoreTmp PkgTmp  PkgWatt RAMWatt PKG_%   RAM_%
0       0       0       2893    2394    75      79      85.87   31.48   0.00    0.00
1	0 	14	59      2394    38	46	30.89   35.81   0.00    0.00
It was expected the 2nd CPU+RAM should be idle.

And indeed bladebit is using RAM in interleaved mode from both sockets (see below)
despite using "numactl --membind ...".
That means there is memory transfer over QPI link which is bottleneck now.

Test 4) No-Numa to the rescue
=============================
Use "--no-numa" setting (note: older bladebit versions had a bug where it was ignored)
Two parallel jobs
a) numactl -C14-27,42-55  -m1  bladebit -t 28 --no-cpu-affinity  --no-numa ...
b) sleep 30 ; numactl -C0-13,28-41 -m0 bladebit -t 28 --no-cpu-affinity --no-numa ...

Result:
In 15.7 minutes you get two plots; that means 7.85 minutes per plot
Total Throughput: 183 PPD (+15% compared to Test1)

28649.24842.out:Finished plotting in 941.89 seconds (15.70 minutes).
28649.24842.out:Finished plotting in 932.82 seconds (15.55 minutes).
28649.24842.out:Finished plotting in 960.54 seconds (16.01 minutes). #including final write
28751.27173.out:Finished plotting in 950.29 seconds (15.84 minutes).
28751.27173.out:Finished plotting in 941.85 seconds (15.70 minutes).
28751.27173.out:Finished plotting in 968.70 seconds (16.15 minutes). #including final write

Discussion of result:
=====================
- The --no-numa option is needed when running two parallel jobs for my configuration
- That improves throughput from 158 to 183 PPD (+15%) when ruunning two jobs each 28 threads

Open question when comparing to Epyc results above
===========================================
@grobalt: From your report is seems you get good runtime even without --no-numa despite
it would run with interleaved memory. Can you confirm you did not use --no-numa /retest with --no-numa 
with your configuration ?


Appendix: strace to document how numa is implemented in bladbit
========
Default bladebit calls mbind(...MPOL_INTERLEAVE,..)
53977 write(1, "Allocating buffers.", 19) = 19
53977 write(1, "\n", 1)                 = 1
53977 mmap(NULL, 17179873280, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f964cc6f000
53977 get_mempolicy(NULL, NULL, 0, NULL, 0) = 0
53977 mbind(0x7f964cc70000, 17179869184, MPOL_INTERLEAVE, [

With "--no-numa" the numactl setting is kept, no mbind() is used.
54244 write(1, "Allocating buffers.", 19) = 19
54244 write(1, "\n", 1)                 = 1
54244 mmap(NULL, 17179873280, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fec4b7f9000
54244 mmap(NULL, 34359742464, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe44b7f8000

[grobalt]

as you see above, i used -m

[Unnameless]

@grobalt I wonder if for your use case wouldn't it be better to run 4 threads assigned evenly per each CPU with half t count ofc

[grobalt]

That would be the worst decision you can make. The connection between the 2 CPUs to access the memory of the distant socket is compared to local memory extremly slow.

[Unnameless]

but isn't that what you actually did above (didn't saw at the time of my original comment), when you said: "i decided to map just 60 threads, 2 per socket, 4 per machine:"? You ran 4 bladebit threads per machine right!