Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add workflow to automate durations files updates #6247

Closed
wants to merge 93 commits into from
Closed
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
93 commits
Select commit Hold shift + click to select a range
0a62b11
Tweak test actions; add durations update action
mudit2812 Sep 9, 2024
c6b2aa8
Add commented sections for workflow triggers
mudit2812 Sep 10, 2024
57084b3
Fix syntax
mudit2812 Sep 10, 2024
bef53ab
Merge branch 'master' into durations-automation
mudit2812 Sep 10, 2024
a6bf30e
Fix matrix for interface-unit-tests
mudit2812 Sep 10, 2024
de63a86
Testing workflow using pull_request trigger
mudit2812 Sep 10, 2024
709fedc
Rename step
mudit2812 Sep 10, 2024
c86c54f
Add codecov token secret
mudit2812 Sep 10, 2024
90bd488
Fix usage of pipeline_mode
mudit2812 Sep 10, 2024
f007892
Add pipeline_mode info
mudit2812 Sep 10, 2024
0c969c3
Remove max-parallel for update-durations pipeline
mudit2812 Sep 10, 2024
f7f7126
Trigger CI
mudit2812 Sep 10, 2024
a8750e9
Update pytest args
mudit2812 Sep 10, 2024
1d70fa9
[skip ci] Update pytest args; remove pull-request trigger
mudit2812 Sep 10, 2024
b34d336
Use github CLI, add back pull_request trigger to test
mudit2812 Sep 10, 2024
2606785
Trying to fix syntax; revert cli use in test workflow
mudit2812 Sep 10, 2024
8262496
Fix gh syntax more; add core team as reviewer
mudit2812 Sep 10, 2024
9a2ee1e
Skip pytest while debugging for fast testing
mudit2812 Sep 10, 2024
4c20c98
Skip tests properly
mudit2812 Sep 10, 2024
7abe0fa
Trigger CI
mudit2812 Sep 10, 2024
be9467e
Trigger CI
mudit2812 Sep 10, 2024
866fdd4
Run tests only if not using update-durations pipeline
mudit2812 Sep 10, 2024
a16316b
Merge branch 'master' into durations-automation
mudit2812 Sep 10, 2024
dbb1797
Change PR body; skip dep installation while testing
mudit2812 Sep 10, 2024
6cc4a30
Change quotes
mudit2812 Sep 10, 2024
b7de0ec
Adding updates to duration files
mudit2812 Sep 10, 2024
16e87bc
Add github token
mudit2812 Sep 10, 2024
147d7c5
Testing push
mudit2812 Sep 10, 2024
9e64081
Trying fix
mudit2812 Sep 10, 2024
e63d5fa
testing one more time
mudit2812 Sep 10, 2024
7fc4168
Final try
mudit2812 Sep 10, 2024
e4168ed
Dummy commits to files
mudit2812 Sep 10, 2024
d809823
Add empty body
mudit2812 Sep 10, 2024
d912927
Update reviewer
mudit2812 Sep 10, 2024
3e491cd
Update reviewer again
mudit2812 Sep 10, 2024
ad51d95
Try again after closing PR
mudit2812 Sep 10, 2024
df6d126
Removed reviewer
mudit2812 Sep 11, 2024
9bb77e7
[skip ci] Final workflow
mudit2812 Sep 11, 2024
afb83b3
Adding back test skip for final PR opening step
mudit2812 Sep 11, 2024
c65f601
Update quotes
mudit2812 Sep 11, 2024
98636f4
Remove body
mudit2812 Sep 11, 2024
e453629
Make changes
mudit2812 Sep 11, 2024
9d8bfa5
Change PR script
mudit2812 Sep 11, 2024
fd3b8cb
Add change
mudit2812 Sep 11, 2024
5e66e05
Finalize workflow
mudit2812 Sep 11, 2024
a5ccbf9
[skip ci] Remove pull request trigger
mudit2812 Sep 11, 2024
f1d55fa
Trigger CI
mudit2812 Sep 13, 2024
2586021
[no ci] bump nightly version
ringo-but-quantum Sep 11, 2024
4cd618c
[no ci] bump nightly version
ringo-but-quantum Sep 12, 2024
1445476
Remove default qubit jax (#6209)
albi3ro Sep 12, 2024
f5f07c4
Fix failing `test_diagonalize_all_measurements` for legacy opmath (#6…
astralcai Sep 12, 2024
46c2652
RC sync for `v0.38.1` to `master` (#6265)
mudit2812 Sep 12, 2024
2fe0f0f
Deprecate`QNode.gradient_fn` (#6244)
albi3ro Sep 12, 2024
341c5bd
[no ci] bump nightly version
ringo-but-quantum Sep 13, 2024
711ec21
deprecate set_shots (#6250)
albi3ro Sep 13, 2024
888c2f5
Fix FABLE template to return the correct result in JIT mode (#6263)
willjmax Sep 13, 2024
b5d8f57
Fix failing test in legacy opmath (#6272)
astralcai Sep 13, 2024
0dde67a
remove default.qubit.autograd (#6210)
albi3ro Sep 13, 2024
037dc88
Deprecate top level access to legacy device base classes (#6238)
albi3ro Sep 13, 2024
2411a8b
[Program Capture] Add pytree support to captured `qml.grad` and `qml.…
dwierichs Sep 15, 2024
bdbd37c
[no ci] bump nightly version
ringo-but-quantum Sep 16, 2024
6423644
Pennylane is compatible with numpy 2.0 (#6061)
EmilianoG-byte Sep 16, 2024
0e38df3
Clean up how `interface` is handled in `QNode` and `qml.execute` (#6225)
astralcai Sep 16, 2024
c0146c4
Updating torch to 2.3.0 (#6258)
PietropaoloFrisoni Sep 16, 2024
f85ebda
[no ci] bump nightly version
ringo-but-quantum Sep 17, 2024
9a02bea
Add `reference.qubit` for testing and reference (#6181)
astralcai Sep 17, 2024
0b01495
Deprecation QubitStateVector (#6172)
KetpuntoG Sep 17, 2024
446312c
PennyLane is compatible with JAX 0.4.28 (#6255)
PietropaoloFrisoni Sep 17, 2024
05d1f31
`qinfo` module deprecations and removals (#5911)
isaacdevlugt Sep 17, 2024
8e4ea49
Miscellaneous deprecations (#6277)
mudit2812 Sep 17, 2024
46df546
Remove `decomp_depth` from `qml.device` constructor (#6234)
albi3ro Sep 17, 2024
64f2175
[no ci] bump nightly version
ringo-but-quantum Sep 18, 2024
3b2f5cb
Remove `simplify` argument from `Hamiltonian` and `LinearCombination`…
mudit2812 Sep 18, 2024
83182a1
Remove some end-to-end `default_qubit_native_mcm` tests (#6246)
obliviateandsurrender Sep 18, 2024
1d6b859
Deprecate basis state preparation (#6116)
KetpuntoG Sep 18, 2024
e2903e6
Methods to commute terms in Fermi objects (#6196)
willjmax Sep 18, 2024
4f31588
Add `work_wires` parameter to `qml.MultiControlledX` docstring signat…
justinpickering Sep 18, 2024
4f40f5a
[no ci] bump nightly version
ringo-but-quantum Sep 19, 2024
6d48958
Remove DefaultQubitLegacy 🎉 (#6266)
albi3ro Sep 19, 2024
c80a877
Unit tests for `simulate_one_shot_native_mcm (#6124)
astralcai Sep 19, 2024
300da86
Add `VnEntanglementEntropyMP` measurement process (#6270)
mudit2812 Sep 19, 2024
eaef556
`devices.qubit.measure` uses `csr_dot_products` only when it is usabl…
astralcai Sep 19, 2024
420ceb8
Add unit tests for `simulate_tree_mcm` (#6231)
obliviateandsurrender Sep 19, 2024
d0f124c
Correct some typos across the codebase (#6280)
dwierichs Sep 20, 2024
7dac0d8
[no ci] bump nightly version
ringo-but-quantum Sep 20, 2024
f3b6d3a
Create a workflow to test PennyLane with NumPy 1.26 (#6275)
PietropaoloFrisoni Sep 20, 2024
b6216ee
Add emery and Haldane model Hamiltonians (#6201)
ddhawan11 Sep 20, 2024
3584d94
Add additional lattice models (#6237)
ddhawan11 Sep 20, 2024
081b58d
Added Kitaev model Hamiltonian (#6174)
ddhawan11 Sep 20, 2024
6d7eb0b
Update casting for `apply_operation` (#6268)
mudit2812 Sep 20, 2024
221afdb
Update durations (#6289)
mudit2812 Sep 20, 2024
e9c829d
Build hamiltonians from custom lattices (#6226)
austingmhuang Sep 20, 2024
e4b0bbe
[no ci] bump nightly version
ringo-but-quantum Sep 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 7 additions & 15 deletions .github/workflows/interface-unit-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ on:
required: true
type: string
pipeline_mode:
description: The pipeline mode can be unit-tests, benchmarks, or reference-benchmarks
description: The pipeline mode can be unit-tests, update-durations, benchmarks, or reference-benchmarks
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new pipeline mode is used to selectively run jobs that are needed for updating the durations files.

required: false
type: string
default: 'unit-tests'
Expand Down Expand Up @@ -42,11 +42,6 @@ on:
required: false
type: string
default: "False"
pytest_store_durations:
description: Whether to store artifacts for test durations
required: false
type: boolean
default: false
Comment on lines -45 to -49
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed this in favour of the new update-durations pipeline mode.


jobs:
setup-ci-load:
Expand Down Expand Up @@ -212,7 +207,7 @@ jobs:
|| fromJSON(needs.setup-ci-load.outputs.matrix-max-parallel).default
}}
matrix:
group: [1, 2, 3]
group: ${{ inputs.pipeline_mode == 'update-durations' && fromJSON('[1]') || fromJSON('[1, 2, 3]') }}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't split the jobs for tensorflow, jax, and core tests when updating durations. This way, rather than generating multiple artifacts for each job, we generate a single unified one that can be used as the updated durations file.

python-version: >-
${{
fromJSON(needs.setup-ci-load.outputs.python-version).tf-tests
Expand All @@ -233,8 +228,7 @@ jobs:
pytest_coverage_flags: ${{ inputs.pytest_coverage_flags }}
pytest_markers: tf and not qcut and not finite-diff and not param-shift
pytest_additional_args: --splits 3 --group ${{ matrix.group }}
pytest_durations_file_path: '.github/workflows/tf_tests_durations.json'
pytest_store_durations: ${{ inputs.pytest_store_durations }}
pytest_durations_file_path: '.github/durations/tf_tests_durations.json'
additional_pip_packages: pytest-split
requirements_file: ${{ github.event_name == 'schedule' && strategy.job-index == 0 && 'tf.txt' || '' }}
disable_new_opmath: ${{ inputs.disable_new_opmath }}
Expand All @@ -250,7 +244,7 @@ jobs:
|| fromJSON(needs.setup-ci-load.outputs.matrix-max-parallel).default
}}
matrix:
group: [1, 2, 3, 4, 5]
group: ${{ inputs.pipeline_mode == 'update-durations' && fromJSON('[1]') || fromJSON('[1, 2, 3, 4, 5]') }}
python-version: >-
${{
fromJSON(needs.setup-ci-load.outputs.python-version).jax-tests
Expand All @@ -271,8 +265,7 @@ jobs:
pytest_coverage_flags: ${{ inputs.pytest_coverage_flags }}
pytest_markers: jax and not qcut and not finite-diff and not param-shift
pytest_additional_args: --splits 5 --group ${{ matrix.group }}
pytest_durations_file_path: '.github/workflows/jax_tests_durations.json'
pytest_store_durations: ${{ inputs.pytest_store_durations }}
pytest_durations_file_path: '.github/durations/jax_tests_durations.json'
additional_pip_packages: pytest-split
requirements_file: ${{ github.event_name == 'schedule' && strategy.job-index == 0 && 'jax.txt' || '' }}
disable_new_opmath: ${{ inputs.disable_new_opmath }}
Expand All @@ -288,7 +281,7 @@ jobs:
|| fromJSON(needs.setup-ci-load.outputs.matrix-max-parallel).default
}}
matrix:
group: [1, 2, 3, 4, 5]
group: ${{ inputs.pipeline_mode == 'update-durations' && fromJSON('[1]') || fromJSON('[1, 2, 3, 4, 5]') }}
python-version: >-
${{
fromJSON(needs.setup-ci-load.outputs.python-version).core-tests
Expand All @@ -309,8 +302,7 @@ jobs:
pytest_coverage_flags: ${{ inputs.pytest_coverage_flags }}
pytest_markers: core and not qcut and not finite-diff and not param-shift
pytest_additional_args: --splits 5 --group ${{ matrix.group }}
pytest_durations_file_path: '.github/workflows/core_tests_durations.json'
pytest_store_durations: ${{ inputs.pytest_store_durations }}
pytest_durations_file_path: '.github/durations/core_tests_durations.json'
additional_pip_packages: pytest-split
requirements_file: ${{ github.event_name == 'schedule' && strategy.job-index == 0 && 'core.txt' || '' }}
disable_new_opmath: ${{ inputs.disable_new_opmath }}
Expand Down
3 changes: 1 addition & 2 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,12 +38,11 @@ jobs:
&& !contains(github.event.pull_request.labels.*.name, 'ci:run-full-test-suite')
|| false
}}
pytest_store_durations: false

upload-stable-deps:
needs: tests
runs-on: ubuntu-latest
if: ${{ github.event_name == 'push' }}
if: ${{ github.event_name == 'schedule' }}
steps:
- name: Checkout
uses: actions/checkout@v3
Expand Down
25 changes: 11 additions & 14 deletions .github/workflows/unit-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ on:
type: string
default: '3.10'
pipeline_mode:
description: The pipeline mode can be unit-tests, benchmarks, or reference-benchmark
description: The pipeline mode can be unit-tests, update-durations, benchmarks, or reference-benchmark
required: false
type: string
default: 'unit-tests'
Expand Down Expand Up @@ -94,11 +94,6 @@ on:
required: false
type: string
default: ''
pytest_store_durations:
description: Whether to store artifacts for test durations
required: false
type: boolean
default: false
additional_pip_packages:
description: Additional packages to install. Values will be passed to pip install {value}
required: false
Expand Down Expand Up @@ -129,7 +124,7 @@ jobs:
repository: PennyLaneAI/pennylane

- name: Determine benchmark name
if: ${{ inputs.pipeline_mode != 'unit-tests' }}
if: ${{ !contains(fromJSON('["unit-tests", "update-durations"]'), inputs.pipeline_mode) }}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skip benchmarks pipeline related steps when not needed.

id: benchmark_name
run: |
job_name="${{ inputs.job_name }}"
Expand All @@ -138,7 +133,7 @@ jobs:
echo "benchmark_name=$_benchmark_name" >> $GITHUB_OUTPUT

- name: Cache reference benchmarks
if: ${{ inputs.pipeline_mode != 'unit-tests' }}
if: ${{ !contains(fromJSON('["unit-tests", "update-durations"]'), inputs.pipeline_mode) }}
id: benchmark-cache
uses: actions/cache@v3
with:
Expand All @@ -149,7 +144,7 @@ jobs:
id: continue
run: >-
echo "confirm=${{
contains(fromJSON('["unit-tests", "benchmarks"]'), inputs.pipeline_mode)
contains(fromJSON('["unit-tests", "benchmarks", "update-durations"]'), inputs.pipeline_mode)
|| (inputs.pipeline_mode == 'reference-benchmarks'
&& steps.benchmark-cache.outputs.cache-hit != 'true' )}}" >> $GITHUB_OUTPUT

Expand All @@ -174,14 +169,16 @@ jobs:
PYTEST_COVERAGE_ARGS: ${{ inputs.pytest_coverage_flags }}
PYTEST_PARALLELISE_ARGS: -n auto
PYTEST_BENCHMARKS_ARGS: --benchmark-enable --benchmark-only --benchmark-json=benchmarks.json
PYTEST_ADDITIONAL_ARGS: ${{ inputs.pytest_additional_args }}
PYTEST_ADDITIONAL_ARGS: ${{ inputs.pipeline_mode == 'update-durations' && '' || inputs.pytest_additional_args }}
PYTEST_DURATIONS_ARGS: ${{ inputs.pytest_durations_file_path != '' && format('--durations-path="{0}"', inputs.pytest_durations_file_path) || '' }}
PYTEST_STORE_ARGS: ${{ inputs.pytest_store_durations == true && '--store-durations --clean-durations' || '' }}
PYTEST_STORE_ARGS: ${{ inputs.pipeline_mode == 'update-durations' && '--store-durations --clean-durations' || '' }}
run: |
if [[ "$PIPELINE_MODE" =~ .*"benchmarks".* ]]; then
echo "args=$PYTEST_BENCHMARKS_ARGS $PYTEST_ADDITIONAL_ARGS" >> $GITHUB_OUTPUT
elif [[ "$PIPELINE_MODE" == "update-durations" ]]; then
echo "args=$PYTEST_PARALLELISE_ARGS $PYTEST_DURATIONS_ARGS $PYTEST_STORE_ARGS" >> $GITHUB_OUTPUT
else
echo "args=$PYTEST_COVERAGE_ARGS $PYTEST_PARALLELISE_ARGS $PYTEST_ADDITIONAL_ARGS $PYTEST_DURATIONS_ARGS $PYTEST_STORE_ARGS" >> $GITHUB_OUTPUT
echo "args=$PYTEST_COVERAGE_ARGS $PYTEST_PARALLELISE_ARGS $PYTEST_ADDITIONAL_ARGS $PYTEST_DURATIONS_ARGS" >> $GITHUB_OUTPUT
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If using the update-durations pipeline, don't include codecov arguments or benchmarks arguments.

fi

- name: Run PennyLane Unit Tests
Expand All @@ -196,10 +193,10 @@ jobs:
run: python -m pytest ${{ inputs.pytest_test_directory }} ${{ steps.pytest_args.outputs.args }} ${{ env.PYTEST_MARKER }} --disable-opmath=${{ inputs.disable_new_opmath }}

- name: Upload Durations file as artifact
if: ${{ inputs.pytest_store_durations == true }}
if: ${{ inputs.pipeline_mode == 'update-durations' }}
uses: actions/upload-artifact@v4
with:
name: ${{ inputs.job_name }}-durations.json
name: durations-${{ inputs.job_name }}
path: ${{ inputs.pytest_durations_file_path }}
include-hidden-files: true
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only upload durations as artifacts if using the update-durations pipeline mode. Also changed the artifact name to start with durations- to be easier to search for when downloading the artifacts


Expand Down
102 changes: 102 additions & 0 deletions .github/workflows/update-durations.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
name: Update Durations Files
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This workflow will use the update-durations pipeline mode for running the tests. Core, tf, and jax tests will be run (since those are the only ones with durations files associated with them), the durations files will be stored as artifacts, and then the workflow will open a PR with the updated durations files.

The workflow will run on a weekly schedule, and will also be available using a manual trigger.

on:
# Scheduled trigger on the 15th of every month at 2:47am UTC
# schedule:
# - cron: '47 2 15 * *'
pull_request:
# types:
# - opened
# - reopened
# - synchronize
# - ready_for_review
workflow_dispatch:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a workflow_dispatch trigger so that we can manually trigger the workflow if any PR massively imbalances the load.


concurrency:
group: unit-tests-${{ github.ref }}
cancel-in-progress: true

env:
DURATIONS_BRANCH: bot/durations-update
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Branch from which durations file update PR will be opened.


jobs:
update-durations:
# if: >-
# ${{
# github.event_name == 'schedule' ||
# github.event_name == 'workflow_dispatch' ||
# (
# github.event_name == 'pull_request' &&
# github.event.pull_request.draft == false &&
# contains(github.event.pull_request.labels.*.name, 'ci:update-durations') &&
# github.event.pull_request.state == 'open'
# )
# }}
uses: ./.github/workflows/interface-unit-tests.yml
secrets:
codecov_token: ${{ secrets.CODECOV_TOKEN }}
with:
branch: ${{ github.ref }}

# Run a 'lightened' version of the CI on Pull Requests by default
# Unless the label `ci:run-full-test-suite` is attached to the PR.
# Always runs the full suite for push events.
pipeline_mode: 'update-durations'
run_lightened_ci: true
skip_ci_test_jobs: 'torch-tests,autograd-tests,all-interfaces-tests,external-libraries-tests,qcut-tests,qchem-tests,gradients-tests,data-tests,device-tests'
Comment on lines +22 to +24
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We run lightened CI so that we only run the tests with python 3.10, and the listed tests are skipped (everything except for tf, jax, and core).


# Add job for downloading artifacts and opening PR
upload-durations-files:
needs: update-durations
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
with:
fetch-depth: 0
ref: master
# ref: ${{ github.ref }}

- name: Prepare local repo
run: |
git config user.name "GitHub Actions Bot"
git config user.email "<>"
if git ls-remote --exit-code origin "refs/heads/${{ env.DURATIONS_BRANCH }}"; then
git checkout "${{ env.DURATIONS_BRANCH }}"
else
git checkout -b "${{ env.DURATIONS_BRANCH }}"
fi

- name: Download artifacts
uses: actions/download-artifact@v4
with:
pattern: durations-*
path: .github/durations/
merge-multiple: true

- name: Stage changes
run: |
git add .github/durations/
git commit -m "Update durations files"
git push -f --set-upstream origin "${{ env.DURATIONS_BRANCH }}"

# Create PR to master
- name: Create pull request
uses: repo-sync/pull-request@v2
with:
source_branch: "${{ env.DURATIONS_BRANCH }}"
destination_branch: "master"
github_token: "${{ secrets.GITHUB_TOKEN }}"
pr_title: "Update test duration files"
# if updating the pr_body, please also update the same text in the docs.yml file
pr_body: |
Automatic update of test duration files.

Because bots are not able to trigger CI on their own, please do so by pushing an empty commit to this branch using the following command:

```
git commit --allow-empty -m 'trigger ci'
```

Alternatively, wait for this branch to be out-of-date with master, then just use the "Update branch" button!
pr_allow_empty: false
pr_draft: false
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Checkout out pennylane
  • If DURATIONS_BRANCH exists, checkout to it, otherwise create DURATIONS_BRANCH and checkout to it.
  • Download durations files to the .github/durations directory
  • Commit changes and publish the branch.
  • Open a PR

Loading