Total CPU time increasing for high numbers of futures workers

[sciaba]

I noticed that if I run the AGC workload for increasing numbers of Python futures workers on a node with 128 CPU cores and reading from the local file system, the total CPU time for the entire process tree as measured by PrMon starts increasing for high numbers of workers (~30 or more). Ideally, it should remain constant, if the CPU time depended only on the amount of data to process. A similar thing happens with the RDF version, so it's not specific to Coffea, at least qualitatively (the causes might be different). Any idea about possible causes for this overhead?
I am attaching a plot comparing the CPU time to the processtime (as reported by Coffea) and to the wallclock time of the entire AGC workload multiplied by the number of workers:

[lgray]

@sciaba My first inclination is to think this has something to do with dask (if that's what you're using!) and the way to communicates with workers. It might be worth cross post on their github (with a bunch of details about the setup) and see if anyone comes back with some immediate answers.

[lgray]

Crank the CPU affinity way up and see what happens?

[lgray]

Data cache thrashing does make a bunch of sense.

[lgray]

Could also be TLB misses from context switching all the time. Pinning processes to cores would fix that.

[sciaba]

I'd be happy to try but I don't know how... (in case you can help me with that send me a DM not to pollute the thread!).

[lgray]

@sciaba taskset -pc <cpu-number> <pid> :-)