Enable PerformanceTests #882

[basilevs]

Enable PerformanceTests for Pull Requests labeled "performance".
See discussion on why tests can't be enabled on a permanent basis.

[laeubi]

@basilevs just in case you are interested, I could think about triggering a workflow by adding a label to a PR that then additionally executes the performance tests on demand.

[basilevs]

@laeubi that would be very nice indeed. I suspect this PR can't be used as test ground anyway because by default Github does not use workflows from unverified PRs.

[laeubi]

Yes one need to develop such workflow on the a fork as it requires to merge things to master (can be reset later on of course) and have additional permissions, but as you seem very enthusiastic on this maybe you like to go in that direction as well 👍

[laeubi]

Even though I approved the Run it seems the Github actions failed to execute, do you like to take a look?

[basilevs]

I've missed a colon in step declaration. Would be nice to get an error from that.

[laeubi]

I've missed a colon in step declaration. Would be nice to get an error from that.

yes that a bit unfortunate but you should receive an email if there is a fatal error in the action, also you can edit the file with github UI and then gets a validation.

[github-actions]

Test Results

     302 files  +  3       302 suites  +3   17m 2s ⏱️ + 11m 28s
  4 102 tests +  7    4 094 ✔️ +  7    8 💤 ±0  0 ❌ ±0 
12 218 runs  +21  12 144 ✔️ +20  74 💤 +1  0 ❌ ±0 

Results for commit 158e4c6. ± Comparison against base commit f677076.

♻️ This comment has been updated with latest results.

[basilevs]

Yes one need to develop such workflow on the a fork as it requires to merge things to master (can be reset later on of course) and have additional permissions, but as you seem very enthusiastic on this maybe you like to go in that direction as well 👍

@laeubi could you please mark my pull requests with performance label?

[laeubi]

@basilevs have you tested this with your fork? I think even if I aprove the workflow run your changes are not picked up maybe because you are not a cmoitter on the repo.

[basilevs]

@basilevs have you tested this with your fork?

Here is my test build

I'm still not sure it it breaks artifact collection. And something strange is going on with MacOS.

[basilevs]

API Tools fail the build. What is the workflow here? Last time I've tried to bump versions, it did not end well.

On a positive note I see my additional step in the action logs.

[iloveeclipse]

[ERROR] Failed to execute goal org.eclipse.tycho.extras:tycho-p2-extras-plugin:4.0.4:compare-version-with-baselines (compare-attached-artifacts-with-release) on project org.eclipse.swt: Only qualifier changed for (org.eclipse.swt/3.124.200.v20231127-1131). Expected to have bigger x.y.z than what is available in baseline (3.124.200.v20231113-1355)

This means, this is first change in the release in this bundle but manifest version is not updated.
Depending on the change you have to bump either last version segment (if the API os not changed) or second one (if API is affected).

[basilevs]

[ERROR] Failed to execute goal org.eclipse.tycho.extras:tycho-p2-extras-plugin:4.0.4:compare-version-with-baselines (compare-attached-artifacts-with-release) on project org.eclipse.swt: Only qualifier changed for (org.eclipse.swt/3.124.200.v20231127-1131). Expected to have bigger x.y.z than what is available in baseline (3.124.200.v20231113-1355)

This means, this is first change in the release in this bundle but manifest version is not updated. Depending on the change you have to bump either last version segment (if the API os not changed) or second one (if API is affected).

No code was changed (the change modifies only .github directory).
I've created #909 to bump the version

[basilevs]

This works, but one test takes a a lot of time on MacOS. Looks like a performance bug. I've created #912

[basilevs]

@akurtakov I see that performance test assertions were disabled three years ago. Now they can never fail. Should this be fixed? I assume lots of tests are using these assertions in vain now.

eclipse-platform/eclipse.platform.releng@071af51#diff-55cfefb6d6ca7ae136d8057fc7ea93d8b62af5db44fc77cec0f961fc68d2bdbdL85-L107

[akurtakov]

@basilevs 3 years ago perf tests have been migrated from storing both (current and latest release) timings in a db (this database server was a nightmare to maintain and historical data with OS updates and etc. proved to be useless) to using the test files which contain timings to show the diffs in perfermance . This also degraded over time:

• 4.27 (March 2023) - working for a few perf tests https://download.eclipse.org/eclipse/downloads/drops4/R-4.27-202303020300/performance/performance.php . SWT tests worked (to some extend) https://download.eclipse.org/eclipse/downloads/drops4/R-4.27-202303020300/performance/basicPerformance.php?name=org.eclipse.swt&build=I20230302-0300&baseline=R-4.26-202211231800
• 4.28 (June 2023) - only baseline results https://download.eclipse.org/eclipse/downloads/drops4/R-4.28-202306050440/performance/performance.php
• 4.29 (September 2023) - no perf tests data at all

Long story short perf tests were running (minus infra/releng failures) till March this year . The way these perf tests worked was to run the same test twice - once with previous release and once with latest I-build and compare timings. The disabled assertions were actually failing based on "timing" criteria from DB content which as said previously is not that useful with fast changing environments nowadays.

[basilevs]

Skipped the slow test on MacOS with Assume.

[basilevs]

@laeubi this works fine. Please merge.