Add GitHub workflow for continuous benchmarking. #304
Conversation
plietar
force-pushed
the
continuous-benchmarking
branch
from
5ec1393 to
0397412
Compare
plietar
changed the base branch from
dqrng-release
to
continuous-benchmarking-base
plietar
force-pushed
the
continuous-benchmarking
branch
from
d5a5ad5 to
01ea03c
Compare
plietar
force-pushed
the
continuous-benchmarking-base
branch
from
0397412 to
dfc943f
Compare
plietar
force-pushed
the
continuous-benchmarking
branch
from
01ea03c to
498b021
Compare
plietar
force-pushed
the
continuous-benchmarking
branch
2 times, most recently
from
19a04fc to
3fd40f1
Compare
plietar
changed the title
|
See plietar#9 for an example interaction. The link at the end of the bot's comment points to more detailed plots. The workflow file needs to be merged into master, since that is what GitHub looks for to react to PR comments, which is why this PR targets that branch. (Coincidentally I forgot I had that other performance improvement branch lying around that I should probably open a real PR for, since the numbers on it look pretty good...) |
|
PR #311 |
The workflow runs on-demand by commenting `/benchmark` on any pull request. It runs a few iterations of both the base branch and the PR and posts the results back to the pull request as a comment. Additionally, detailed plots are available as images in the workflow artifacts. Unfortunately GitHub doesn't have a way for actions to upload images in comments. At the moment the workflow runs two benchmarks, one at a 10k population size and 10k time steps, and second one at 1M population size and 1k time steps. The workflow currently takes on the order of 1 hour to run. We can easily tweak the set of benchmarks in the future if we want to run more scenarios, or conversly if we want to reduce the total time. There is no long-term tracking of benchmark results yet. Doing so is tricky as the performance of the underlying CI is unlikely to be stable long-term.
It was merged into touchstone upstream.
plietar
force-pushed
the
continuous-benchmarking
branch
from
faeb56d to
142d857
Compare
I'm getting issues with 24.04 where pak installs some system packages with apt-get, some system services are upgraded and restarted and the CI runner is canceled. Not sure what the reason is, but 22.04 doesn't have this problem.
|
Is this good to go? I would really like this for #300 |
Add GitHub workflow for continuous benchmarking.
|
Yes! Sorry I'd gotten sidetracked on some other stuff. I've manually cherry-picked and pushed the changes onto the dev branch. You'll need to rebase your PR onto that (the PR branch needs the touchstone directory), after which the / benchmark should work! |
The workflow runs on-demand by commenting
/benchmarkon any pull request. It runs a few iterations of both the base branch and the PR and posts the results back to the pull request as a comment. Additionally, detailed plots are available as images in the workflow artifacts. Unfortunately GitHub doesn't have a way for actions to upload images in comments.At the moment the workflow runs two benchmarks, one at a 10k population size and 10k time steps, and second one at 1M population size and 1k time steps.
The workflow currently takes on the order of 1 hour to run. We can easily tweak the set of benchmarks in the future if we want to run more scenarios, or conversely if we want to reduce the total time.
There is no long-term tracking of benchmark results yet. Doing so is tricky as the performance of the underlying CI is unlikely to be stable long-term.