Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Change default KvikIO parameters in cuDF: set the thread pool size to 4, and compatibility mode to ON #17185

Merged
merged 9 commits into from
Nov 1, 2024
Merged

Change default KvikIO parameters in cuDF: set the thread pool size to 4, and compatibility mode to ON #17185

merged 9 commits into from
Nov 1, 2024

Conversation

kingcrimsontianyu
Copy link
Contributor

@kingcrimsontianyu kingcrimsontianyu commented Oct 25, 2024 •
edited
Loading

Description

This PR adjusts the default KvikIO parameters in light of recent discussions.

  • Set KvikIO compatibility mode to ON (previously unspecified). This is to avoid the overhead of KvikIO validating the cuFile library when most of the time clients are not using cuFile/GDS.
  • Set KvikIO thread pool size to 4 (previous 8). See the reason below.

In addition, this PR updates the documentation on LIBCUDF_CUFILE_POLICY.

It is reported that Dask-cuDF on a 8-GPU node with Lustre file system has major performance regression when the KvikIO thread pool size is 8.

KVIKIO_NTHREADS 8 4 2 1
Dask-cuDF time [s] 16 3.9 4.0 4.3
cuDF time [s] 3.4 3.4 3.8 4.9

Additional benchmark on Grace Hopper (Parquet, CSV) indicates no performance regression by switching the thread pool size from 8 to 4. For the time being, we choose 4 as an empirical sweet spot.

Closes #16512

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@github-actions github-actions bot added the libcudf Affects libcudf (C++/CUDA) code. label Oct 25, 2024
@kingcrimsontianyu kingcrimsontianyu self-assigned this Oct 25, 2024
@kingcrimsontianyu kingcrimsontianyu added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Oct 25, 2024
@kingcrimsontianyu kingcrimsontianyu changed the title Change the KvikIO thread pool size from 8 to 4 due to the performance issue with Dask-cuDF Change default KvikIO parameters in cuDF: set the thread pool size to 4, and compatibility mode to ON Oct 30, 2024
@kingcrimsontianyu kingcrimsontianyu marked this pull request as ready for review October 30, 2024 18:10
@kingcrimsontianyu kingcrimsontianyu requested a review from a team as a code owner October 30, 2024 18:10
bdice
bdice approved these changes Oct 30, 2024
View reviewed changes
cpp/src/io/utilities/config_utils.cpp Outdated Show resolved Hide resolved
cpp/src/io/utilities/config_utils.cpp Outdated Show resolved Hide resolved
@kingcrimsontianyu
Copy link
Contributor Author

/merge

@rapids-bot rapids-bot bot merged commit 6ce9ea4 into rapidsai:branch-24.12 Nov 1, 2024
102 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vuule vuule vuule approved these changes

@ttnghia ttnghia ttnghia approved these changes

@bdice bdice bdice approved these changes

Assignees

@kingcrimsontianyu kingcrimsontianyu

Labels
improvement Improvement / enhancement to an existing function libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change
Projects
libcudf
Status: No status
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEA] Adjust libcudf to not load cuFile by default
4 participants
@kingcrimsontianyu @bdice @ttnghia @vuule