-
Notifications
You must be signed in to change notification settings - Fork 603
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add workflow to automate durations files updates #6247
Changes from 13 commits
0a62b11
c6b2aa8
57084b3
bef53ab
a6bf30e
de63a86
709fedc
c86c54f
90bd488
f007892
0c969c3
f7f7126
a8750e9
1d70fa9
b34d336
2606785
8262496
9a2ee1e
4c20c98
7abe0fa
be9467e
866fdd4
a16316b
dbb1797
6cc4a30
b7de0ec
16e87bc
147d7c5
9e64081
e63d5fa
7fc4168
e4168ed
d809823
d912927
3e491cd
ad51d95
df6d126
9bb77e7
afb83b3
c65f601
98636f4
e453629
9d8bfa5
fd3b8cb
5e66e05
a5ccbf9
f1d55fa
2586021
4cd618c
1445476
f5f07c4
46c2652
2fe0f0f
341c5bd
711ec21
888c2f5
b5d8f57
0dde67a
037dc88
2411a8b
bdbd37c
6423644
0e38df3
c0146c4
f85ebda
9a02bea
0b01495
446312c
05d1f31
8e4ea49
46df546
64f2175
3b2f5cb
83182a1
1d6b859
e2903e6
4f31588
4f40f5a
6d48958
c80a877
300da86
eaef556
420ceb8
d0f124c
7dac0d8
f3b6d3a
b6216ee
3584d94
081b58d
6d7eb0b
221afdb
e9c829d
e4b0bbe
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -11,7 +11,7 @@ on: | |
required: true | ||
type: string | ||
pipeline_mode: | ||
description: The pipeline mode can be unit-tests, benchmarks, or reference-benchmarks | ||
description: The pipeline mode can be unit-tests, update-durations, benchmarks, or reference-benchmarks | ||
required: false | ||
type: string | ||
default: 'unit-tests' | ||
|
@@ -42,11 +42,6 @@ on: | |
required: false | ||
type: string | ||
default: "False" | ||
pytest_store_durations: | ||
description: Whether to store artifacts for test durations | ||
required: false | ||
type: boolean | ||
default: false | ||
Comment on lines
-45
to
-49
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Removed this in favour of the new |
||
|
||
jobs: | ||
setup-ci-load: | ||
|
@@ -212,7 +207,7 @@ jobs: | |
|| fromJSON(needs.setup-ci-load.outputs.matrix-max-parallel).default | ||
}} | ||
matrix: | ||
group: [1, 2, 3] | ||
group: ${{ inputs.pipeline_mode == 'update-durations' && fromJSON('[1]') || fromJSON('[1, 2, 3]') }} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We don't split the jobs for tensorflow, jax, and core tests when updating durations. This way, rather than generating multiple artifacts for each job, we generate a single unified one that can be used as the updated durations file. |
||
python-version: >- | ||
${{ | ||
fromJSON(needs.setup-ci-load.outputs.python-version).tf-tests | ||
|
@@ -233,8 +228,7 @@ jobs: | |
pytest_coverage_flags: ${{ inputs.pytest_coverage_flags }} | ||
pytest_markers: tf and not qcut and not finite-diff and not param-shift | ||
pytest_additional_args: --splits 3 --group ${{ matrix.group }} | ||
pytest_durations_file_path: '.github/workflows/tf_tests_durations.json' | ||
pytest_store_durations: ${{ inputs.pytest_store_durations }} | ||
pytest_durations_file_path: '.github/durations/tf_tests_durations.json' | ||
additional_pip_packages: pytest-split | ||
requirements_file: ${{ github.event_name == 'schedule' && strategy.job-index == 0 && 'tf.txt' || '' }} | ||
disable_new_opmath: ${{ inputs.disable_new_opmath }} | ||
|
@@ -250,7 +244,7 @@ jobs: | |
|| fromJSON(needs.setup-ci-load.outputs.matrix-max-parallel).default | ||
}} | ||
matrix: | ||
group: [1, 2, 3, 4, 5] | ||
group: ${{ inputs.pipeline_mode == 'update-durations' && fromJSON('[1]') || fromJSON('[1, 2, 3, 4, 5]') }} | ||
python-version: >- | ||
${{ | ||
fromJSON(needs.setup-ci-load.outputs.python-version).jax-tests | ||
|
@@ -271,8 +265,7 @@ jobs: | |
pytest_coverage_flags: ${{ inputs.pytest_coverage_flags }} | ||
pytest_markers: jax and not qcut and not finite-diff and not param-shift | ||
pytest_additional_args: --splits 5 --group ${{ matrix.group }} | ||
pytest_durations_file_path: '.github/workflows/jax_tests_durations.json' | ||
pytest_store_durations: ${{ inputs.pytest_store_durations }} | ||
pytest_durations_file_path: '.github/durations/jax_tests_durations.json' | ||
additional_pip_packages: pytest-split | ||
requirements_file: ${{ github.event_name == 'schedule' && strategy.job-index == 0 && 'jax.txt' || '' }} | ||
disable_new_opmath: ${{ inputs.disable_new_opmath }} | ||
|
@@ -288,7 +281,7 @@ jobs: | |
|| fromJSON(needs.setup-ci-load.outputs.matrix-max-parallel).default | ||
}} | ||
matrix: | ||
group: [1, 2, 3, 4, 5] | ||
group: ${{ inputs.pipeline_mode == 'update-durations' && fromJSON('[1]') || fromJSON('[1, 2, 3, 4, 5]') }} | ||
python-version: >- | ||
${{ | ||
fromJSON(needs.setup-ci-load.outputs.python-version).core-tests | ||
|
@@ -309,8 +302,7 @@ jobs: | |
pytest_coverage_flags: ${{ inputs.pytest_coverage_flags }} | ||
pytest_markers: core and not qcut and not finite-diff and not param-shift | ||
pytest_additional_args: --splits 5 --group ${{ matrix.group }} | ||
pytest_durations_file_path: '.github/workflows/core_tests_durations.json' | ||
pytest_store_durations: ${{ inputs.pytest_store_durations }} | ||
pytest_durations_file_path: '.github/durations/core_tests_durations.json' | ||
additional_pip_packages: pytest-split | ||
requirements_file: ${{ github.event_name == 'schedule' && strategy.job-index == 0 && 'core.txt' || '' }} | ||
disable_new_opmath: ${{ inputs.disable_new_opmath }} | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -25,7 +25,7 @@ on: | |
type: string | ||
default: '3.10' | ||
pipeline_mode: | ||
description: The pipeline mode can be unit-tests, benchmarks, or reference-benchmark | ||
description: The pipeline mode can be unit-tests, update-durations, benchmarks, or reference-benchmark | ||
required: false | ||
type: string | ||
default: 'unit-tests' | ||
|
@@ -94,11 +94,6 @@ on: | |
required: false | ||
type: string | ||
default: '' | ||
pytest_store_durations: | ||
description: Whether to store artifacts for test durations | ||
required: false | ||
type: boolean | ||
default: false | ||
additional_pip_packages: | ||
description: Additional packages to install. Values will be passed to pip install {value} | ||
required: false | ||
|
@@ -129,7 +124,7 @@ jobs: | |
repository: PennyLaneAI/pennylane | ||
|
||
- name: Determine benchmark name | ||
if: ${{ inputs.pipeline_mode != 'unit-tests' }} | ||
if: ${{ !contains(fromJSON('["unit-tests", "update-durations"]'), inputs.pipeline_mode) }} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Skip benchmarks pipeline related steps when not needed. |
||
id: benchmark_name | ||
run: | | ||
job_name="${{ inputs.job_name }}" | ||
|
@@ -138,7 +133,7 @@ jobs: | |
echo "benchmark_name=$_benchmark_name" >> $GITHUB_OUTPUT | ||
|
||
- name: Cache reference benchmarks | ||
if: ${{ inputs.pipeline_mode != 'unit-tests' }} | ||
if: ${{ !contains(fromJSON('["unit-tests", "update-durations"]'), inputs.pipeline_mode) }} | ||
id: benchmark-cache | ||
uses: actions/cache@v3 | ||
with: | ||
|
@@ -149,7 +144,7 @@ jobs: | |
id: continue | ||
run: >- | ||
echo "confirm=${{ | ||
contains(fromJSON('["unit-tests", "benchmarks"]'), inputs.pipeline_mode) | ||
contains(fromJSON('["unit-tests", "benchmarks", "update-durations"]'), inputs.pipeline_mode) | ||
|| (inputs.pipeline_mode == 'reference-benchmarks' | ||
&& steps.benchmark-cache.outputs.cache-hit != 'true' )}}" >> $GITHUB_OUTPUT | ||
|
||
|
@@ -174,14 +169,16 @@ jobs: | |
PYTEST_COVERAGE_ARGS: ${{ inputs.pytest_coverage_flags }} | ||
PYTEST_PARALLELISE_ARGS: -n auto | ||
PYTEST_BENCHMARKS_ARGS: --benchmark-enable --benchmark-only --benchmark-json=benchmarks.json | ||
PYTEST_ADDITIONAL_ARGS: ${{ inputs.pytest_additional_args }} | ||
PYTEST_ADDITIONAL_ARGS: ${{ inputs.pipeline_mode == 'update-durations' && '' || inputs.pytest_additional_args }} | ||
PYTEST_DURATIONS_ARGS: ${{ inputs.pytest_durations_file_path != '' && format('--durations-path="{0}"', inputs.pytest_durations_file_path) || '' }} | ||
PYTEST_STORE_ARGS: ${{ inputs.pytest_store_durations == true && '--store-durations --clean-durations' || '' }} | ||
PYTEST_STORE_ARGS: ${{ inputs.pipeline_mode == 'update-durations' && '--store-durations --clean-durations' || '' }} | ||
run: | | ||
if [[ "$PIPELINE_MODE" =~ .*"benchmarks".* ]]; then | ||
echo "args=$PYTEST_BENCHMARKS_ARGS $PYTEST_ADDITIONAL_ARGS" >> $GITHUB_OUTPUT | ||
elif [[ "$PIPELINE_MODE" == "update-durations" ]]; then | ||
echo "args=$PYTEST_PARALLELISE_ARGS $PYTEST_DURATIONS_ARGS $PYTEST_STORE_ARGS" >> $GITHUB_OUTPUT | ||
else | ||
echo "args=$PYTEST_COVERAGE_ARGS $PYTEST_PARALLELISE_ARGS $PYTEST_ADDITIONAL_ARGS $PYTEST_DURATIONS_ARGS $PYTEST_STORE_ARGS" >> $GITHUB_OUTPUT | ||
echo "args=$PYTEST_COVERAGE_ARGS $PYTEST_PARALLELISE_ARGS $PYTEST_ADDITIONAL_ARGS $PYTEST_DURATIONS_ARGS" >> $GITHUB_OUTPUT | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If using the |
||
fi | ||
|
||
- name: Run PennyLane Unit Tests | ||
|
@@ -196,10 +193,10 @@ jobs: | |
run: python -m pytest ${{ inputs.pytest_test_directory }} ${{ steps.pytest_args.outputs.args }} ${{ env.PYTEST_MARKER }} --disable-opmath=${{ inputs.disable_new_opmath }} | ||
|
||
- name: Upload Durations file as artifact | ||
if: ${{ inputs.pytest_store_durations == true }} | ||
if: ${{ inputs.pipeline_mode == 'update-durations' }} | ||
uses: actions/upload-artifact@v4 | ||
with: | ||
name: ${{ inputs.job_name }}-durations.json | ||
name: durations-${{ inputs.job_name }} | ||
path: ${{ inputs.pytest_durations_file_path }} | ||
include-hidden-files: true | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Only upload durations as artifacts if using the |
||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,102 @@ | ||
name: Update Durations Files | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This workflow will use the The workflow will run on a weekly schedule, and will also be available using a manual trigger. |
||
on: | ||
# Scheduled trigger on the 15th of every month at 2:47am UTC | ||
# schedule: | ||
# - cron: '47 2 15 * *' | ||
pull_request: | ||
# types: | ||
# - opened | ||
# - reopened | ||
# - synchronize | ||
# - ready_for_review | ||
workflow_dispatch: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I added a |
||
|
||
concurrency: | ||
group: unit-tests-${{ github.ref }} | ||
cancel-in-progress: true | ||
|
||
env: | ||
DURATIONS_BRANCH: bot/durations-update | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Branch from which durations file update PR will be opened. |
||
|
||
jobs: | ||
update-durations: | ||
# if: >- | ||
# ${{ | ||
# github.event_name == 'schedule' || | ||
# github.event_name == 'workflow_dispatch' || | ||
# ( | ||
# github.event_name == 'pull_request' && | ||
# github.event.pull_request.draft == false && | ||
# contains(github.event.pull_request.labels.*.name, 'ci:update-durations') && | ||
# github.event.pull_request.state == 'open' | ||
# ) | ||
# }} | ||
uses: ./.github/workflows/interface-unit-tests.yml | ||
secrets: | ||
codecov_token: ${{ secrets.CODECOV_TOKEN }} | ||
with: | ||
branch: ${{ github.ref }} | ||
|
||
# Run a 'lightened' version of the CI on Pull Requests by default | ||
# Unless the label `ci:run-full-test-suite` is attached to the PR. | ||
# Always runs the full suite for push events. | ||
pipeline_mode: 'update-durations' | ||
run_lightened_ci: true | ||
skip_ci_test_jobs: 'torch-tests,autograd-tests,all-interfaces-tests,external-libraries-tests,qcut-tests,qchem-tests,gradients-tests,data-tests,device-tests' | ||
Comment on lines
+22
to
+24
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We run lightened CI so that we only run the tests with python 3.10, and the listed tests are skipped (everything except for |
||
|
||
# Add job for downloading artifacts and opening PR | ||
upload-durations-files: | ||
needs: update-durations | ||
runs-on: ubuntu-latest | ||
steps: | ||
- name: Checkout | ||
uses: actions/checkout@v3 | ||
with: | ||
fetch-depth: 0 | ||
ref: master | ||
# ref: ${{ github.ref }} | ||
|
||
- name: Prepare local repo | ||
run: | | ||
git config user.name "GitHub Actions Bot" | ||
git config user.email "<>" | ||
if git ls-remote --exit-code origin "refs/heads/${{ env.DURATIONS_BRANCH }}"; then | ||
git checkout "${{ env.DURATIONS_BRANCH }}" | ||
else | ||
git checkout -b "${{ env.DURATIONS_BRANCH }}" | ||
fi | ||
|
||
- name: Download artifacts | ||
uses: actions/download-artifact@v4 | ||
with: | ||
pattern: durations-* | ||
path: .github/durations/ | ||
merge-multiple: true | ||
|
||
- name: Stage changes | ||
run: | | ||
git add .github/durations/ | ||
git commit -m "Update durations files" | ||
git push -f --set-upstream origin "${{ env.DURATIONS_BRANCH }}" | ||
|
||
# Create PR to master | ||
- name: Create pull request | ||
uses: repo-sync/pull-request@v2 | ||
with: | ||
source_branch: "${{ env.DURATIONS_BRANCH }}" | ||
destination_branch: "master" | ||
github_token: "${{ secrets.GITHUB_TOKEN }}" | ||
pr_title: "Update test duration files" | ||
# if updating the pr_body, please also update the same text in the docs.yml file | ||
pr_body: | | ||
Automatic update of test duration files. | ||
|
||
Because bots are not able to trigger CI on their own, please do so by pushing an empty commit to this branch using the following command: | ||
|
||
``` | ||
git commit --allow-empty -m 'trigger ci' | ||
``` | ||
|
||
Alternatively, wait for this branch to be out-of-date with master, then just use the "Update branch" button! | ||
pr_allow_empty: false | ||
pr_draft: false | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This new pipeline mode is used to selectively run jobs that are needed for updating the durations files.