It turns out that this machine can be used for a piece of land in 18 minutes. Now I don't know why it is slow. The log is as follows. Memory footprint is only 270G

[hadesya]

Creating 3 plots:
Output path : /mnt/chia
Thread count : 48
Warm start enabled : false
Farmer public key : b499221905e5745915ce80cfe8691bac03ef4e80c65fa43d7a45ab25508cc40b524c8fa7e2527c91c83f27b71c35e708
Pool public key : 972484c7c81ef427337b6514848690d8f3822b9cda8ceb0952ad438265e67ef3e2955fbbc57a7189637df302a6fc1bb8

System Memory: 502/503 GiB.
Memory required: 416 GiB.
Allocating buffers.
Generating plot 1 / 3: 5c2851a0a57de3c40654c2a39cccef8e8bf3f3c407d1f3a3572d479ea186b4e4

Running Phase 1
Generating F1...
Finished F1 generation in 26.21 seconds.
Sorting F1...
Finished F1 sort in 228.39 seconds.
Forward propagating to table 2...
Pairing L/R groups...
Finished pairing L/R groups in 69.2310 seconds. Created 4294967296 pairs.
Average of 236.1406 pairs per group.
Computing Fx...
Finished computing Fx in 66.6400 seconds.
Sorting entries...

Finished sorting in 1697.84 seconds.
Finished forward propagating table 2 in 1836.98 seconds.
Forward propagating to table 3...
Pairing L/R groups...
Finished pairing L/R groups in 60.4030 seconds. Created 4294938242 pairs.
Average of 236.1390 pairs per group.
Computing Fx...
Finished computing Fx in 84.9330 seconds.
Sorting entries...

[harold-b]

Were you plotting without issue before?

[hadesya]

There is no problem with the earliest version. This machine can be used for one field in 18 minutes. . Recently I cleaned up the machine, rebuilt the system and compiled the software, and that’s it

[harold-b]

Try building version 1.0.0 (git checkout 1.0.0), then rebuild and run it to see if you see any difference.

[hadesya]

OK thank you

[harold-b]

Please let me know if you see any changes with that version.

[bladeuserpi]

It seems sensitive about NUMA and Linux aggressive use of buffer cache (also causing swap storms)
-Avoid mv/cp and others before/while it is running (dd with directio is OK)
-Does performance recover with this? Enter as root:
time echo 3 >/proc/sys/vm/drop_caches ; time sync
If it does not help, show "top" output.