incubator-kie-issues/issues#697 Disable flaky tests in incubator-kie-kogito-examples #1830

Rikkola · 2023-11-15T17:24:42Z

How to replicate CI configuration locally?

Build Chain tool does "simple" maven build(s), the builds are just Maven commands, but because the repositories relates and depends on each other and any change in API or class method could affect several of those repositories there is a need to use build-chain tool to handle cross repository builds and be sure that we always use latest version of the code for each repository.

build-chain tool is a build tool which can be used on command line locally or in Github Actions workflow(s), in case you need to change multiple repositories and send multiple dependent pull requests related with a change you can easily reproduce the same build by executing it on Github hosted environment or locally in your development environment. See local execution details to get more information about it.

How to retest this PR or trigger a specific build:

for pull request checks
Please add comment: Jenkins retest this
for a specific pull request check
Please add comment: Jenkins (re)run [kogito-examples] tests
for quarkus branch checks
Run checks against Quarkus current used branch
Please add comment: Jenkins run quarkus-branch
for a quarkus branch specific check
Run checks against Quarkus current used branch
Please add comment: Jenkins (re)run [kogito-examples] quarkus-branch
for quarkus main checks
Run checks against Quarkus main branch
Please add comment: Jenkins run quarkus-main
for a specific quarkus main check
Run checks against Quarkus main branch
Please add comment: Jenkins (re)run [kogito-examples] quarkus-main
for quarkus lts checks
Run checks against Quarkus lts branch
Please add comment: Jenkins run quarkus-lts
for a specific quarkus lts check
Run checks against Quarkus lts branch
Please add comment: Jenkins (re)run [kogito-examples] quarkus-lts
for native checks
Run native checks
Please add comment: Jenkins run native
for a specific native check
Run native checks
Please add comment: Jenkins (re)run [kogito-examples] native
for native lts checks
Run native checks against quarkus lts branch
Please add comment: Jenkins run native-lts
for a specific native lts check
Run native checks against quarkus lts branch
Please add comment: Jenkins (re)run [kogito-examples] native-lts

How to backport a pull request to a different branch?

In order to automatically create a backporting pull request please add one or more labels having the following format backport-<branch-name>, where <branch-name> is the name of the branch where the pull request must be backported to (e.g., backport-7.67.x to backport the original PR to the 7.67.x branch).

NOTE: backporting is an action aiming to move a change (usually a commit) from a branch (usually the main one) to another one, which is generally referring to a still maintained release branch. Keeping it simple: it is about to move a specific change or a set of them from one branch to another.

Once the original pull request is successfully merged, the automated action will create one backporting pull request per each label (with the previous format) that has been added.

If something goes wrong, the author will be notified and at this point a manual backporting is needed.

NOTE: this automated backporting is triggered whenever a pull request on main branch is labeled or closed, but both conditions must be satisfied to get the new PR created.

Rikkola · 2023-11-16T09:51:54Z

Jenkins retest this

Rikkola · 2023-11-16T11:34:03Z

Bad license merge. Fixing.

domhanak · 2023-11-16T14:52:09Z

@Rikkola is there an issue that will cover fixing them? if so please tag me there, thanks

Rikkola · 2023-11-16T14:59:47Z

@domhanak I think that is something that we need, but I am not sure where to add these. I assume there will be several tests like these and each one would need a ticket. Should I make one for each into incubator-kie-issues?

Problem is, I doubt there will be a rush to fix these. So they will stay there open for a while. Then many people look at the tickets to see how healthy the project is, and in this case the issues will make that list bigger. If there are no plans to fix, then maybe it would be better to delete these?

ricardozanini · 2023-11-16T15:32:16Z

@domhanak @jstastny-cz @Rikkola please open issues in this repo and a fup mail to the devlist, so the community is aware.

Rikkola · 2023-11-16T15:33:49Z

@ricardozanini ok a ticket for this repo makes sense

domhanak · 2023-11-16T16:11:04Z

@Rikkola @ricardozanini 👍 the issue would be something like Reevaluate these tests with the goal to fix the flaky-ness or cover the hole we have in coverage because of its removal.

ricardozanini · 2023-11-16T17:11:54Z

@Rikkola @ricardozanini 👍 the issue would be something like Reevaluate these tests with the goal to fix the flaky-ness or cover the hole we have in coverage because of its removal.

Makes sense to me.

domhanak

Link the issue here and we can merge. @Rikkola

jstastny-cz · 2023-11-20T12:21:05Z

Issue is tracked against the target repo: #1831

Though one thing - I think ignoring the test does not put them into the results as skipped, as is done for other tests currently being disabled - e.g. those OutboxIT in https://ci-builds.apache.org/blue/organizations/jenkins/KIE%2Fkogito%2Fmain%2Fpullrequest_jobs%2Fkogito-examples-pr/detail/PR-1830/6/tests

Rikkola · 2023-11-20T14:19:28Z

@jstastny-cz true it does not. And that explains why the other tests were not just Ignored. I will update the PR with that approach.

Rikkola · 2023-11-21T16:26:59Z

@jstastny-cz @domhanak Is this fine to be merged?

jstastny-cz · 2023-11-21T16:35:12Z

I thought you wanted to have the test shown as skipped in test results, based on previous discussion.

Rikkola · 2023-11-22T08:22:46Z

I do want to, but I likely did not get that result with what I did. Let me recheck.

Rikkola · 2023-11-22T08:50:04Z

Right should use disabled, I still went with ignored.

Disable flaky tests in incubator-kie-kogito-examples

Rikkola · 2023-11-29T08:07:02Z

@jstastny-cz Is there a way to make sure the build skipped the tests, not ignored them?

kie-ci3 · 2023-11-29T17:13:52Z

PR job #10 was: UNSTABLE
Possible explanation: This should be test failures

Reproducer

build-chain build full_downstream -f 'https://raw.githubusercontent.com/${AUTHOR:apache}/incubator-kie-kogito-pipelines/${BRANCH:main}/.ci/buildchain-config-pr-cdb.yaml' -o 'bc' -p apache/incubator-kie-kogito-examples -u #1830 --skipParallelCheckout

NOTE: To install the build-chain tool, please refer to https://github.com/kiegroup/github-action-build-chain#local-execution

Please look here: https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/pullrequest_jobs/job/kogito-examples-pr/job/PR-1830/10/display/redirect

Test results:

PASSED: 336
FAILED: 35

Those are the test failures:

org.kie.kogito.flexible.example.quarkus.ServiceDeskProcessTest.testSupportCaseExample