Random failing ParallelBuildChainTest.testIndividualProjectBuilds_WithManyProjects_ProjectRelaxedRule

[HeikoKlare]

ParallelBuildChainTest.testIndividualProjectBuilds_WithManyProjects_ProjectRelaxedRule randomly fails on Linux I-Builds, e.g., https://download.eclipse.org/eclipse/downloads/drops4/I20231105-1800/testresults/html/org.eclipse.core.tests.resources_ep430I-unit-cen64-gtk3-java21_linux.gtk.x86_64_21.html

It is random because execution succeeds in some builds: https://download.eclipse.org/eclipse/downloads/drops4/I20231103-0420/testResults.php

The test fails after around 20 seconds, which is the timeout value in ParallelBuildChainTest, so maybe the test is only performing too bad. The test case has been introduced with #661.

Exemplary log:

not all build jobs have started in time Expected: (a collection containing <P/longRunningBuildProject0> and a collection containing <P/longRunningBuildProject1> and a collection containing <P/longRunningBuildProject10> and a collection containing <P/longRunningBuildProject11> and a collection containing <P/longRunningBuildProject12> and a collection containing <P/longRunningBuildProject13> and a collection containing <P/longRunningBuildProject14> and a collection containing <P/longRunningBuildProject15> and a collection containing <P/longRunningBuildProject16> and a collection containing <P/longRunningBuildProject17> and a collection containing <P/longRunningBuildProject18> and a collection containing <P/longRunningBuildProject19> and a collection containing <P/longRunningBuildProject2> and a collection containing <P/longRunningBuildProject20> and a collection containing <P/longRunningBuildProject21> and a collection containing <P/longRunningBuildProject22> and a collection containing <P/longRunningBuildProject23> and a collection containing <P/longRunningBuildProject24> and a collection containing <P/longRunningBuildProject25> and a collection containing <P/longRunningBuildProject26> and a collection containing <P/longRunningBuildProject27> and a collection containing <P/longRunningBuildProject28> and a collection containing <P/longRunningBuildProject29> and a collection containing <P/longRunningBuildProject3> and a collection containing <P/longRunningBuildProject30> and a collection containing <P/longRunningBuildProject31> and a collection containing <P/longRunningBuildProject32> and a collection containing <P/longRunningBuildProject33> and a collection containing <P/longRunningBuildProject34> and a collection containing <P/longRunningBuildProject35> and a collection containing <P/longRunningBuildProject36> and a collection containing <P/longRunningBuildProject37> and a collection containing <P/longRunningBuildProject38> and a collection containing <P/longRunningBuildProject39> and a collection containing <P/longRunningBuildProject4> and a collection containing <P/longRunningBuildProject40> and a collection containing <P/longRunningBuildProject41> and a collection containing <P/longRunningBuildProject42> and a collection containing <P/longRunningBuildProject43> and a collection containing <P/longRunningBuildProject44> and a collection containing <P/longRunningBuildProject45> and a collection containing <P/longRunningBuildProject46> and a collection containing <P/longRunningBuildProject47> and a collection containing <P/longRunningBuildProject48> and a collection containing <P/longRunningBuildProject49> and a collection containing <P/longRunningBuildProject5> and a collection containing <P/longRunningBuildProject50> and a collection containing <P/longRunningBuildProject51> and a collection containing <P/longRunningBuildProject52> and a collection containing <P/longRunningBuildProject53> and a collection containing <P/longRunningBuildProject54> and a collection containing <P/longRunningBuildProject55> and a collection containing <P/longRunningBuildProject56> and a collection containing <P/longRunningBuildProject57> and a collection containing <P/longRunningBuildProject58> and a collection containing <P/longRunningBuildProject59> and a collection containing <P/longRunningBuildProject6> and a collection containing <P/longRunningBuildProject7> and a collection containing <P/longRunningBuildProject8> and a collection containing <P/longRunningBuildProject9>) but: a collection containing <P/longRunningBuildProject10> mismatches were: [was <P/longRunningBuildProject0>, was <P/longRunningBuildProject12>, was <P/longRunningBuildProject22>, was <P/longRunningBuildProject34>, was <P/longRunningBuildProject40>, was <P/longRunningBuildProject50>, was <P/longRunningBuildProject37>, was <P/longRunningBuildProject59>, was <P/longRunningBuildProject16>, was <P/longRunningBuildProject57>, was <P/longRunningBuildProject39>, was <P/longRunningBuildProject47>, was <P/longRunningBuildProject7>, was <P/longRunningBuildProject13>, was <P/longRunningBuildProject24>, was <P/longRunningBuildProject2>, was <P/longRunningBuildProject19>, was <P/longRunningBuildProject8>, was <P/longRunningBuildProject55>, was <P/longRunningBuildProject6>, was <P/longRunningBuildProject18>, was <P/longRunningBuildProject42>, was <P/longRunningBuildProject58>, was <P/longRunningBuildProject52>, was <P/longRunningBuildProject33>, was <P/longRunningBuildProject54>, was <P/longRunningBuildProject17>, was <P/longRunningBuildProject30>, was <P/longRunningBuildProject38>, was <P/longRunningBuildProject56>, was <P/longRunningBuildProject23>, was <P/longRunningBuildProject41>, was <P/longRunningBuildProject25>, was <P/longRunningBuildProject31>, was <P/longRunningBuildProject45>, was <P/longRunningBuildProject14>, was <P/longRunningBuildProject15>, was <P/longRunningBuildProject35>, was <P/longRunningBuildProject32>, was <P/longRunningBuildProject1>, was <P/longRunningBuildProject48>, was <P/longRunningBuildProject3>, was <P/longRunningBuildProject27>, was <P/longRunningBuildProject53>, was <P/longRunningBuildProject44>, was <P/longRunningBuildProject49>, was <P/longRunningBuildProject29>, was <P/longRunningBuildProject4>, was <P/longRunningBuildProject28>, was <P/longRunningBuildProject11>, was <P/longRunningBuildProject20>]

java.lang.AssertionError: not all build jobs have started in time
Expected: (a collection containing <P/longRunningBuildProject0> and a collection containing <P/longRunningBuildProject1> and a collection containing <P/longRunningBuildProject10> and a collection containing <P/longRunningBuildProject11> and a collection containing <P/longRunningBuildProject12> and a collection containing <P/longRunningBuildProject13> and a collection containing <P/longRunningBuildProject14> and a collection containing <P/longRunningBuildProject15> and a collection containing <P/longRunningBuildProject16> and a collection containing <P/longRunningBuildProject17> and a collection containing <P/longRunningBuildProject18> and a collection containing <P/longRunningBuildProject19> and a collection containing <P/longRunningBuildProject2> and a collection containing <P/longRunningBuildProject20> and a collection containing <P/longRunningBuildProject21> and a collection containing <P/longRunningBuildProject22> and a collection containing <P/longRunningBuildProject23> and a collection containing <P/longRunningBuildProject24> and a collection containing <P/longRunningBuildProject25> and a collection containing <P/longRunningBuildProject26> and a collection containing <P/longRunningBuildProject27> and a collection containing <P/longRunningBuildProject28> and a collection containing <P/longRunningBuildProject29> and a collection containing <P/longRunningBuildProject3> and a collection containing <P/longRunningBuildProject30> and a collection containing <P/longRunningBuildProject31> and a collection containing <P/longRunningBuildProject32> and a collection containing <P/longRunningBuildProject33> and a collection containing <P/longRunningBuildProject34> and a collection containing <P/longRunningBuildProject35> and a collection containing <P/longRunningBuildProject36> and a collection containing <P/longRunningBuildProject37> and a collection containing <P/longRunningBuildProject38> and a collection containing <P/longRunningBuildProject39> and a collection containing <P/longRunningBuildProject4> and a collection containing <P/longRunningBuildProject40> and a collection containing <P/longRunningBuildProject41> and a collection containing <P/longRunningBuildProject42> and a collection containing <P/longRunningBuildProject43> and a collection containing <P/longRunningBuildProject44> and a collection containing <P/longRunningBuildProject45> and a collection containing <P/longRunningBuildProject46> and a collection containing <P/longRunningBuildProject47> and a collection containing <P/longRunningBuildProject48> and a collection containing <P/longRunningBuildProject49> and a collection containing <P/longRunningBuildProject5> and a collection containing <P/longRunningBuildProject50> and a collection containing <P/longRunningBuildProject51> and a collection containing <P/longRunningBuildProject52> and a collection containing <P/longRunningBuildProject53> and a collection containing <P/longRunningBuildProject54> and a collection containing <P/longRunningBuildProject55> and a collection containing <P/longRunningBuildProject56> and a collection containing <P/longRunningBuildProject57> and a collection containing <P/longRunningBuildProject58> and a collection containing <P/longRunningBuildProject59> and a collection containing <P/longRunningBuildProject6> and a collection containing <P/longRunningBuildProject7> and a collection containing <P/longRunningBuildProject8> and a collection containing <P/longRunningBuildProject9>)
but: a collection containing <P/longRunningBuildProject10> mismatches were: [was <P/longRunningBuildProject0>, was <P/longRunningBuildProject12>, was <P/longRunningBuildProject22>, was <P/longRunningBuildProject34>, was <P/longRunningBuildProject40>, was <P/longRunningBuildProject50>, was <P/longRunningBuildProject37>, was <P/longRunningBuildProject59>, was <P/longRunningBuildProject16>, was <P/longRunningBuildProject57>, was <P/longRunningBuildProject39>, was <P/longRunningBuildProject47>, was <P/longRunningBuildProject7>, was <P/longRunningBuildProject13>, was <P/longRunningBuildProject24>, was <P/longRunningBuildProject2>, was <P/longRunningBuildProject19>, was <P/longRunningBuildProject8>, was <P/longRunningBuildProject55>, was <P/longRunningBuildProject6>, was <P/longRunningBuildProject18>, was <P/longRunningBuildProject42>, was <P/longRunningBuildProject58>, was <P/longRunningBuildProject52>, was <P/longRunningBuildProject33>, was <P/longRunningBuildProject54>, was <P/longRunningBuildProject17>, was <P/longRunningBuildProject30>, was <P/longRunningBuildProject38>, was <P/longRunningBuildProject56>, was <P/longRunningBuildProject23>, was <P/longRunningBuildProject41>, was <P/longRunningBuildProject25>, was <P/longRunningBuildProject31>, was <P/longRunningBuildProject45>, was <P/longRunningBuildProject14>, was <P/longRunningBuildProject15>, was <P/longRunningBuildProject35>, was <P/longRunningBuildProject32>, was <P/longRunningBuildProject1>, was <P/longRunningBuildProject48>, was <P/longRunningBuildProject3>, was <P/longRunningBuildProject27>, was <P/longRunningBuildProject53>, was <P/longRunningBuildProject44>, was <P/longRunningBuildProject49>, was <P/longRunningBuildProject29>, was <P/longRunningBuildProject4>, was <P/longRunningBuildProject28>, was <P/longRunningBuildProject11>, was <P/longRunningBuildProject20>]
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.eclipse.core.tests.internal.builders.ParallelBuildChainTest.assertBuildsToStart(ParallelBuildChainTest.java:483)
at org.eclipse.core.tests.internal.builders.ParallelBuildChainTest.lambda$2(ParallelBuildChainTest.java:150)
at org.eclipse.core.tests.internal.builders.ParallelBuildChainTest.executeIndividualFullProjectBuilds(ParallelBuildChainTest.java:455)
at org.eclipse.core.tests.internal.builders.ParallelBuildChainTest.testIndividualProjectBuilds_WithManyProjects_ProjectRelaxedRule(ParallelBuildChainTest.java:149)

[iloveeclipse]

Yes, I assume our build infrastructure is very unstable / slow on Linux.
So if the test can increase timeout, it would be better as to disable this test completely.

[HeikoKlare]

I increased timeout in #832. Let's see whether that helps. If it does not solve the problem, I will have a deeper look and probably adapt the test, as a test running for more than 30 seconds does not really make sense (except when being a performance test).

[HeikoKlare]

Still randomly fails with timeout increase from 20 seconds to 30 seconds: https://download.eclipse.org/eclipse/downloads/drops4/I20231107-1800/testresults/html/org.eclipse.core.tests.resources_ep430I-unit-cen64-gtk3-java21_linux.gtk.x86_64_21.html

I will have a look whether that's really only an issue of performance or whether something is functionally broken.

[BeckerWdf]

failed again for #1068

[HeikoKlare]

In I-Builds the test now fails not because the because not all builds are started but eben because not all jobs can be spawned, see https://ci.eclipse.org/platform/job/eclipse.platform/job/PR-1386/2/testReport/

Timeout after 10000ms waiting for status to change from WAIT_FOR_START to RUNNING
 [ThreadDump taken from thread 'main' at 2024-05-21 10:15:32.062:
   Thread "main" #1 prio=5 RUNNABLE
     at java.base@17.0.2/java.lang.Thread.dumpThreads(Native Method)
     at java.base@17.0.2/java.lang.Thread.getAllStackTraces(Thread.java:1662)
     at org.eclipse.core.tests.harness.TestBarrier2.getThreadDump(TestBarrier2.java:109)
     at org.eclipse.core.tests.harness.TestBarrier2.doWaitForStatus(TestBarrier2.java:93)
     at org.eclipse.core.tests.harness.TestBarrier2.waitForStatus(TestBarrier2.java:160)
     at org.eclipse.core.tests.harness.TestBarrier2.waitForStatus(TestBarrier2.java:198)
     at org.eclipse.core.tests.internal.builders.ParallelBuildChainTest.executeIndividualFullProjectBuilds(ParallelBuildChainTest.java:453)
     at org.eclipse.core.tests.internal.builders.ParallelBuildChainTest.testIndividualProjectBuilds_WithManyProjects_ProjectRelaxedRule(ParallelBuildChainTest.java:150)

[jukzi]

still randomly failing:

Timeout after 10030ms waiting for status to change from WAIT_FOR_START to RUNNING
 [ThreadDump taken from thread 'main' at 2024-06-06 09:02:07.932:
   Thread "main" #1 prio=5 RUNNABLE
     at java.base@17.0.2/java.lang.Thread.dumpThreads(Native Method)
     at java.base@17.0.2/java.lang.Thread.getAllStackTraces(Thread.java:1662)
     at org.eclipse.core.tests.harness.TestBarrier2.getThreadDump(TestBarrier2.java:109)
     at org.eclipse.core.tests.harness.TestBarrier2.doWaitForStatus(TestBarrier2.java:93)
     at org.eclipse.core.tests.harness.TestBarrier2.waitForStatus(TestBarrier2.java:160)
     at org.eclipse.core.tests.harness.TestBarrier2.waitForStatus(TestBarrier2.java:198)
     at org.eclipse.core.tests.internal.builders.ParallelBuildChainTest.executeIndividualFullProjectBuilds(ParallelBuildChainTest.java:453)
     at org.eclipse.core.tests.internal.builders.ParallelBuildChainTest.testIndividualProjectBuilds_WithManyProjects_ProjectRelaxedRule(ParallelBuildChainTest.java:150)

https://ci.eclipse.org/platform/job/eclipse.platform/job/PR-1399/3/testReport/junit/(root)/AutomatedResourceTests%20AllBuilderTests%20ParallelBuildChainTest/testIndividualProjectBuilds_WithManyProjects_ProjectRelaxedRule/

[jukzi]

happend again:
https://download.eclipse.org/eclipse/downloads/drops4/I20241211-1800/testresults/html/org.eclipse.core.tests.resources_ep435I-unit-linux-x86_64-java23_linux.gtk.x86_64_23.html