Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add regression workflow #131

Merged
merged 7 commits into from
Sep 12, 2024
Merged
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 84 additions & 0 deletions .github/workflows/regression.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
on:
pull_request:
branches: [ "master" ]

name: "Regression"

jobs:
compare-results:
runs-on: ubuntu-latest

steps:
- name: Checkout PR branch
uses: actions/checkout@v4
with:
fetch-depth: 2

- uses: dtolnay/rust-toolchain@stable

- name: Download LAZ file
env:
REGRESSION_LAZ_FILE_URL: ${{ secrets.REGRESSION_LAZ_FILE_URL }}
run: curl -L "$REGRESSION_LAZ_FILE_URL" -o test_file.laz
antbern marked this conversation as resolved.
Show resolved Hide resolved

- name: Run code on PR branch
id: run_pr_branch
run: |
echo "Running code on PR branch"
compare -list resource
cargo run --release -- test_file.laz
mv pullautus.png pullautus_pr.png
mv pullautus_depr.png pullautus_depr_pr.png
rm -rf temp/

- name: Download, extract and run latest release
run: |
# in a new folder to make sure we have a clean environment
mkdir latest_release && cd latest_release
curl -L https://github.com/rphlo/karttapullautin/releases/latest/download/karttapullautin-x86_64-linux.tar.gz | tar xvz
./pullauta ../test_file.laz
mv pullautus.png ../pullautus_main.png
mv pullautus_depr.png ../pullautus_depr_main.png

- name: Compare results (pngcomp)
id: compare
run: |
# Compare the results from both branches, output results as step summary
sudo apt install pngnq
printf "\n\n##### Comparing without depressions:\n" | tee -a $GITHUB_STEP_SUMMARY
pngcomp pullautus_main.png pullautus_pr.png | tee -a pngcomp.txt $GITHUB_STEP_SUMMARY
printf "\n\n##### Comparing with depressions:\n" | tee -a $GITHUB_STEP_SUMMARY
pngcomp pullautus_depr_main.png pullautus_depr_pr.png | tee -a pngcomp_depr.txt $GITHUB_STEP_SUMMARY

- name: Upload results
uses: actions/upload-artifact@v4
with:
name: regression-results
path: |
pullautus_main.png
pullautus_pr.png
pngcomp.txt
pullautus_depr_main.png
pullautus_depr_pr.png
pngcomp_depr.txt
overwrite: true # we need subsequent runs to overwrite the result

- name: Check for differences
run: |
# Analyze metrics: if there is a change this step should fail!

# Here we could check any of the metrics output by pngcomp. for now we check the
# percentage of overlapping/correct pixels and fail the job if it is less than 100%
value=$(sed -rn 's/Percentage correct pixels: ([0-9]+\.[0-9]+)$/\1/p' pngcomp.txt)
value_depr=$(sed -rn 's/Percentage correct pixels: ([0-9]+\.[0-9]+)$/\1/p' pngcomp_depr.txt)

if (( $(echo "$value < 100.0" | bc -l) )); then
echo "Percentage of correct pixels (without depressions) is $value < 100%. Output has changed!" | tee -a $GITHUB_STEP_SUMMARY
exit 1
fi

if (( $(echo "$value_depr < 100.0" | bc -l) )); then
echo "Percentage of correct pixels (with depressions) is $value_depr < 100%. Output has changed!" | tee -a $GITHUB_STEP_SUMMARY
exit 1
fi

Loading