[Native] Make PrestoServer create and pass connectorCpuExecutor to Connector #24259
Conversation
| const auto numConnectorCpuThreads = std::max<size_t>( | ||
| SystemConfig::instance()->connectorNumCpuThreadsHwMultiplier() * | ||
| std::thread::hardware_concurrency(), | ||
| 0); |
There was a problem hiding this comment.
Can you guys help confirm if we still need this multiplier mechanism to derive # of cpu threads?
I will double check and handle the config names later. It's being referenced and set to 0 in prod.
gggrace14
changed the title
| @@ -1172,6 +1180,20 @@ std::vector<std::string> PrestoServer::registerConnectors( | |||
| const fs::path& configDirectoryPath) { | |||
| static const std::string kPropertiesExtension = ".properties"; | |||
|
|
|||
| const auto numConnectorCpuThreads = std::max<size_t>( | |||
There was a problem hiding this comment.
you might want to track the connector cpu executor stats in followup. Thanks!
There was a problem hiding this comment.
Will check with you about what stats will be useful to expose offline
gggrace14
force-pushed
the
cpuexec
branch
2 times, most recently
from
62e07af to
024fa28
Compare
…nnector Create a CPUThreadPoolExecutor data member, connectorCpuExecutor_, for PrestoServer. Pass it to every created Connector. Add a new config `connector.num-cpu-threads-hw-multiplier` to control how many threads would be used for the executor. `connector.num-cpu-threads-hw-multiplier` will set connectorCpuExecutor_ to nullptr. Make a process-wise managed CPUThreadPoolExecutor instance available to all connectors. Connector could schedule CPU-bound async operators to it so that they will not occupy the driver thread pool.
|
The test failure is "Execution of 'actual' query failed" as opposed to different results. Looks like not related to this change but a framework issue. The failed tests are |
Description
Create a CPUThreadPoolExecutor data member, connectorCpuExecutor_, for PrestoServer.
Pass it to every created Connector. Add a new config
connector.num-cpu-threads-hw-multiplierto control how many threads would be used for the executor.
connector.num-cpu-threads-hw-multiplierwill set connectorCpuExecutor_ to nullptr.Motivation and Context
Make a process-wise managed CPUThreadPoolExecutor instance available to all connectors.
Connector could schedule CPU-bound async operators to it so that they will not occupy
the driver thread pool.
Impact
When the new config
connector.num-cpu-threads-hw-multiplieris set to positive,a process-wise CPUThreadPoolExecutor will be created.
Test Plan
Release Notes