Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

improve benchmark scripts and add a nightly job #2281

Merged
merged 1 commit into from
Nov 6, 2024
Merged

improve benchmark scripts and add a nightly job #2281

merged 1 commit into from
Nov 6, 2024

Conversation

pbalcer
Copy link
Contributor

@pbalcer pbalcer commented Nov 4, 2024

This patch:

  • Adds a benchmark suite abstraction and moves all existing benchmarks into it. This makes individual benchmark types self-contained and allows us to skip over benchmarks that don't have their dependencies met.
  • Makes sycl and ur optional, and they are no longer positional arguments.
  • Creates a benchmark history that stores benchmark runs. This then enables us to do comparisions not just with the latest result but also against e.g., a historical average.
  • adds a nightly job to store baseline results.

This is all in preparation for adding an HTML output that will generate a static HTML with historical benchmark data.

@pbalcer pbalcer requested a review from a team as a code owner November 4, 2024 14:41
@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 4, 2024

Compute Benchmarks level_zero run (with params: ):
https://github.com/oneapi-src/unified-runtime/actions/runs/11666492180

@github-actions github-actions bot added the ci/cd Continuous integration/devliery label Nov 4, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 4, 2024

Compute Benchmarks level_zero run ():
https://github.com/oneapi-src/unified-runtime/actions/runs/11666492180
Job status: failure. Test status: failure.

@pbalcer
Copy link
Contributor Author

pbalcer commented Nov 4, 2024

The workflow is failing because I changed the arguments:

Run numactl -N 0 /home/test-user/shared-actions-runner/_work/unified-runtime/unified-runtime/ur-repo/scripts/benchmarks/main.py  ~/bench_workdir  /home/test-user/shared-actions-runner/_work/unified-runtime/unified-runtime/sycl_build /home/test-user/shared-actions-runner/_work/unified-runtime/unified-runtime/ur_install level_zero 
usage: main.py [-h] [--sycl SYCL] [--ur UR] [--adapter ADAPTER] [--no-rebuild]
               [--env ENV] [--save SAVE] [--compare COMPARE]
               [--iterations ITERATIONS] [--timeout TIMEOUT] [--filter FILTER]
               [--epsilon EPSILON] [--verbose] [--exit-on-failure]
               [--compare-type {latest,average,median}]
               [--compare-max COMPARE_MAX]
               benchmark_directory
main.py: error: unrecognized arguments: /home/test-user/shared-actions-runner/_work/unified-runtime/unified-runtime/sycl_build /home/test-user/shared-actions-runner/_work/unified-runtime/unified-runtime/ur_install level_zero

But I ran this by hand and it was working fine.

@pbalcer pbalcer requested a review from igchor November 4, 2024 15:09
lukaszstolarczuk
lukaszstolarczuk reviewed Nov 4, 2024
View reviewed changes
.github/workflows/benchmarks.yml Outdated Show resolved Hide resolved
.github/workflows/benchmarks.yml Show resolved Hide resolved
.github/workflows/benchmarks_reusable.yml Outdated Show resolved Hide resolved
@lukaszstolarczuk
Copy link
Contributor

The workflow is failing because I changed the arguments:
...

FYI, if you'd really like to test this workflow you'd have to push your branch to upstream (e.g. pbalcer/test) and use workflow file from that branch

image

@pbalcer pbalcer force-pushed the benchmark-improvements branch 3 times, most recently from 92ebc6d to e9cd3da Compare November 6, 2024 10:45
@pbalcer
Copy link
Contributor Author

pbalcer commented Nov 6, 2024

The workflow is failing because I changed the arguments:
...

FYI, if you'd really like to test this workflow you'd have to push your branch to upstream (e.g. pbalcer/test) and use workflow file from that branch

image

I tried that, unfortunately, if you are changing the workflow dispatch files, they need to be on the main branch.

Example: https://pbalcer.github.io/unified-runtime/benchmark_results.html

@pbalcer pbalcer force-pushed the benchmark-improvements branch 3 times, most recently from 0dd49ea to bd10616 Compare November 6, 2024 11:26
lukaszstolarczuk
lukaszstolarczuk reviewed Nov 6, 2024
View reviewed changes
.github/workflows/docs.yml Outdated Show resolved Hide resolved
scripts/benchmarks/history.py Show resolved Hide resolved
.github/workflows/benchmarks.yml
- gpu
pr_no:
description: PR number (if 0, it'll run on the main)
type: number
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't this be string now?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This affects the form input. But no, it doesn't have to be a string.

.github/workflows/benchmarks-reusable.yml Outdated Show resolved Hide resolved
@pbalcer pbalcer force-pushed the benchmark-improvements branch 2 times, most recently from 9c25dac to e23ca1b Compare November 6, 2024 12:52
lukaszstolarczuk
lukaszstolarczuk reviewed Nov 6, 2024
View reviewed changes
Copy link
Contributor

@lukaszstolarczuk lukaszstolarczuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

.github/workflows/benchmarks-nightly.yml Show resolved Hide resolved
third_party/benchmark_requirements.txt
lxml==4.9.3
Mako==1.3.0
MarkupSafe==2.1.5
packaging==24.1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, the issue we had with six installation, was because packaging was too old (<20.0)

@pbalcer
improve benchmark scripts and add a nightly job
aa3c170 
This patch:
- Adds a benchmark suite abstraction and moves all existing benchmarks
  into it. This makes individual benchmark types self-contained and
  allows us to skip over benchmarks that don't have their dependencies
  met.
- Makes sycl and ur optional, and they are no longer positional
  arguments.
- Creates a benchmark history that stores benchmark runs. This then
  enables us to do comparisions not just with the latest result but
  also against e.g., a historical average.
- adds a nightly job to store baseline results.
- adds HTML output
@pbalcer pbalcer merged commit 3712a7f into main Nov 6, 2024
7 of 37 checks passed
@pbalcer pbalcer deleted the benchmark-improvements branch November 6, 2024 14:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lukaszstolarczuk lukaszstolarczuk lukaszstolarczuk left review comments

@igchor igchor igchor approved these changes

@RossBrunton RossBrunton RossBrunton approved these changes

Assignees
No one assigned
Labels
ci/cd Continuous integration/devliery
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@pbalcer @lukaszstolarczuk @RossBrunton @igchor