Skip to content

Commit

Permalink
Add ability to benchmark pull requests (#6274)
Browse files Browse the repository at this point in the history
**Context:** Add ability to benchmark a pull request

**Description of the Change:**
This pull request adds the ability to benchmark a PR.

When a user with either `admin` or `maintain` permissions comments
`/benchmark` on a pull request, this workflow will trigger a webhook to
benchmarks repo with sufficient details to benchmark the PR, it is then
up to the benchmarks action to report back with results.

Once the webhook is sent, a 👍 reaction is added to the comment to
indicate it was successful.

**Benefits:**
New feature.

**Possible Drawbacks:**
This is an MVP product, we may want to iterate on this to add further
information or functionality. However, the current state is sufficient
for the basic needs of benchmarking.

**Related GitHub Issues:**
None.
[sc-73314](https://app.shortcut.com/xanaduai/story/73314/add-webhook-capability-to-pennylane-prs-to-trigger-benchmark-build)
  • Loading branch information
rashidnhm authored Sep 23, 2024
1 parent ce43f0e commit 9af9fd8
Showing 1 changed file with 100 additions and 0 deletions.
100 changes: 100 additions & 0 deletions .github/workflows/trigger_benchmarks.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
name: Trigger Benchmarks
on:
issue_comment:
types:
- created

env:
BENCHMARK_TRIGGER_COMMAND: /benchmark

jobs:
trigger_benchmark_webhook:
runs-on: ubuntu-latest

# Only run for pull request comments (newly created)
if: github.event.issue.pull_request

steps:
- name: Check Comment Body
id: comment_body
run: echo "triggers_benchmark=${{ github.event.comment.body == env.BENCHMARK_TRIGGER_COMMAND }}" >> $GITHUB_OUTPUT

- name: Set Repository Info
id: repo_info
env:
REPO_OWNER: ${{ github.repository_owner }}
run: |
echo "REPO_OWNER=$REPO_OWNER" >> $GITHUB_OUTPUT
echo "REPO_NAME=${GITHUB_REPOSITORY#$REPO_OWNER/}" >> $GITHUB_OUTPUT
- name: Check if commenter has write access
id: comment_user
if: steps.comment_body.outputs.triggers_benchmark == 'true'
uses: actions/github-script@v7
env:
REPO_OWNER: ${{ steps.repo_info.outputs.REPO_OWNER }}
REPO_NAME: ${{ steps.repo_info.outputs.REPO_NAME }}
with:
retries: 3
result-encoding: string
script: |
const user = context.payload.sender.login;
console.log(`Checking permissions for '${user}' on repository ${process.env.REPO_OWNER}/${process.env.REPO_NAME}}`);
const { data } = await github.rest.repos.getCollaboratorPermissionLevel({
owner: process.env.REPO_OWNER,
repo: process.env.REPO_NAME,
username: user
});
const userPermissions = data.user.permissions;
const permission = userPermissions.admin || userPermissions.maintain;
console.log(`User '${user}' has admin/maintain access => ${permission}`);
return permission;
- name: Trigger Benchmark Webhook
if: steps.comment_body.outputs.triggers_benchmark == 'true' && steps.comment_user.outputs.result == 'true'
uses: peter-evans/repository-dispatch@v3
env:
# The repo that is triggering the source webhook
WEBHOOK_METADATA_SOURCE_REPO: ${{ steps.repo_info.outputs.REPO_OWNER }}/${{ steps.repo_info.outputs.REPO_NAME }}

# The URL of the actual actions run that triggered the webhook
WEBHOOK_METADATA_ACTIONS_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}

# The username of the person who commented the trigger command on the pull request
WEBHOOK_METADATA_GITHUB_ACTOR: ${{ github.event.comment.user.login }}
WEBHOOK_METADATA_COMMENT_LINK: ${{ github.event.comment.html_url }}

# Information regarding the pull request
WEBHOOK_METADATA_PULL_REQUEST_NUMBER: ${{ github.event.issue.number }}
WEBHOOK_METADATA_PULL_REQUEST_URL: ${{ github.event.issue.html_url }}
with:
token: ${{ secrets.PENNYLANE_BENCHMARKS_WEBHOOK_TOKEN }}
repository: PennyLaneAI/${{ secrets.PENNYLANE_BENCHMARKS_REPOSITORY_NAME }}
event-type: pull-request-benchmark
client-payload: |
{
"source_repo": "${{ env.WEBHOOK_METADATA_SOURCE_REPO }}",
"actions_run_url": "${{ env.WEBHOOK_METADATA_ACTIONS_RUN_URL }}",
"github_actor": "${{ env.WEBHOOK_METADATA_GITHUB_ACTOR }}",
"pull_request_number": ${{ env.WEBHOOK_METADATA_PULL_REQUEST_NUMBER }},
"pull_request_url": "${{ env.WEBHOOK_METADATA_PULL_REQUEST_URL }}",
"comment_link": "${{ env.WEBHOOK_METADATA_COMMENT_LINK }}"
}
- name: React to original comment indicating webhook sent
if: steps.comment_user.outputs.result == 'true'
uses: actions/github-script@v7
env:
REPO_OWNER: ${{ steps.repo_info.outputs.REPO_OWNER }}
REPO_NAME: ${{ steps.repo_info.outputs.REPO_NAME }}
with:
script: |
await github.rest.reactions.createForIssueComment({
owner: process.env.REPO_OWNER,
repo: process.env.REPO_NAME,
comment_id: context.payload.comment.id,
content: '+1'
});

0 comments on commit 9af9fd8

Please sign in to comment.