Unify Y-build and I-build configurations and their tests

[HannesWell]

Recently I have reworked the definition of I-build tests to reduce duplication and to make them more unified and simpler.
This has the effect that the definition of Y-build tests has now significantly diverged from the I-build tests and I intended to apply the same simplifications for Y-build tests as well.

And while looking into all the definitions of I-builds and Y-builds I noticed that they have a lot in common and wonder respectively think we should try to unify their definitions and try to define both pipelines from one source, while considering the points where they are different.
But this leads to the questions what exactly the differences between I-builds and Y-builds are? What I have found so far is

• Y-builds built another branch than the master, at least for jdt.core the development branch of ECJ supporting the next Java release
• Y-build tests have one linux setup that uses the latest EA JDK instead of the latest release
• The results of Y-builds are published to another website than I-builds
• Y-builds publish their build p2-repo e.g. to
  https://download.eclipse.org/eclipse/updates/4.35-Y-builds/
  instead of https://download.eclipse.org/eclipse/updates/4.35-I-builds/
• Y-builds don't publish to a Maven snapshot repository (I assume)

Together with #1950 this should to further simplify the overall releng.
Of course a unification would have to happen in multiple steps.

An alternative solution would be to just develop ECJ's support for upcoming java versions on the master branch and stop using another branch. AFAICT this has already been discussed respectively is currently discussed but will not be implemented soon.

The questions I have to the JDT-team:

• Are you fine with the proposed general goal?
• Who of you is interested to discuss the details that come along the path of making this happen?

@jarthana, @stephan-herrmann or @iloveeclipse are you interested in discussing this or can you tell who is interested?

Community

• I understand suggesting an enhancement doesn't mandate anyone to implement it. Other contributors may consider this suggestion, or not, at their own convenience. The most efficient way to get it fixed is that I implement it myself and contribute it back as a good quality patch to the project.

[stephan-herrmann]

I am interested in seeing successful Y-builds with useful test result, ideally with no big gaps around the times of release, or infra changes. If unification helps to this end, then surely I fully agree with the goals, and your description looks correct to me.

I don't, however, have much insight into the inner workings of I-builds nor Y-builds, so I don't know in which detail discussion I could help. I'm already looking into P-builds, which is about all I can contribute in terms of releng builds.

And while it's correct that the matter of master vs BETA_JAVA24 (all of jdt, not just jdt.core) is being discussed widely and for a long time already, it's pretty clear that the mere fact of branching will stay for the foreseeable future, for more than one reason.

[HannesWell]

I am interested in seeing successful Y-builds with useful test result, ideally with no big gaps around the times of release, or infra changes. If unification helps to this end, then surely I fully agree with the goals, and your description looks correct to me.

It will simplify the maintenance of the Y-build and will make it quicker to apply changes and it will also mean that the basic configurations that are meant to be equal, stay equal. So I think this could be a contribution to that overall goal.

I don't, however, have much insight into the inner workings of I-builds nor Y-builds, so I don't know in which detail discussion I could help. I'm already looking into P-builds, which is about all I can contribute in terms of releng builds.

It's mainly when I encounter differences to the I-build and I have to find out if they are really necessary or not. Then I hope to get an answer from you or somebody else from the JDT-team. But the technical Jenkins details I can hopefully handle by myself.

And while it's correct that the matter of master vs BETA_JAVA24 (all of jdt, not just jdt.core) is being discussed widely and for a long time already, it's pretty clear that the mere fact of branching will stay for the foreseeable future, for more than one reason.

Yes, I just wanted to mention that option, knowing that it's at the moment not feasible. :)

[HannesWell]

For your information I have now done the following further step to unify Y-build and I-build tests:

• [Build] Unify Y-build test job names to schema os-arch-javaVersion #2633
• [Y-Build-Tests] Define Mac and Windows jobs as Jenkins-pipeline #2634
• [Y-Build-Tests] Simplify and unify job definitions #2635

The tasks executed respectively the behavior should be the same, but please let me know if you encounter any problem upon the next execution. Respectively if you don't plan to run another Y-build soon, do you mind if I start one?

Does the JDT team mind if obsolete Y-build test jobs, i.e. those that are no longer in use, are removed from https://ci.eclipse.org/releng/job/YPBuilds/ or do you need them for some time for reference?

[HannesWell]

For your information I have now done the following further step to unify Y-build and I-build tests:

To me it looks like the first execution after these changes were ok, without regressions due to infrastructure changes.

@stephan-herrmann or @jarthana I have two questions about the Y-build setup:

1. Is there a reason why Y-build tests don't run on windows? Is it just to save precious computation time on the windows machine?

2. Would it be possible to use the jdk installations provided by default for Eclipse Jenkins instances in Y-build tests on Linux?
   At the moment specific open-jdks are download for each build:
   

   eclipse.platform.releng.aggregator/JenkinsJobs/YBuilds/Y_unit_linux.groovy

   Lines 5 to 7 in 768f042

   	[javaVersion: 17, javaHome: '''installJDK('https://download.java.net/java/GA/jdk17.0.2/dfd4a8d0985749f896bed50d7138ee7f/8/GPL/openjdk-17.0.2_linux-x64_bin.tar.gz')''' ],
   	[javaVersion: 21, javaHome: '''installJDK('https://download.java.net/java/GA/jdk21/fd2272bbf8e04c3dbaee13770090416c/35/GPL/openjdk-21_linux-x64_bin.tar.gz')''' ],
   	[javaVersion: 24, javaHome: '''installJDK('https://download.java.net/java/early_access/jdk24/18/GPL/openjdk-24-ea+18_linux-x64_bin.tar.gz')''' ]

   
   But EF Jenkins provides all these JDKs already, even EA builds for not yet released versions:
   See 'openjdk-ea-latest' at https://github.com/eclipse-cbi/jiro/wiki/Tools-(JDK,-Maven,-Ant)#openjdk
   @fredg02 can you tell how often the EA builds are updated for all Jenkins instances?

To the JDT-devs:
Would you be fine with changing to the provided 'openjdk-ea-latest' for the Y-build
Java 24 tests and switch to temurin for all others?
Just like all other builds:
https://github.com/eclipse-cbi/jiro/wiki/Tools-(JDK,-Maven,-Ant)#eclipse-temurin

[stephan-herrmann]

Does the JDT team mind if obsolete Y-build test jobs, i.e. those that are no longer in use, are removed from https://ci.eclipse.org/releng/job/YPBuilds/ or do you need them for some time for reference?

IMHO old jobs can be removed once the current set of jobs (build & test) is functional.

Let's perhaps just wait for one more +1 from @jarthana or @mpalat

[stephan-herrmann]

1. Is there a reason why Y-build tests don't run on windows? Is it just to save precious computation time on the windows machine?

I'll have to pass this one, as I don't know the reason.

2. Would it be possible to use the jdk installations provided by default for Eclipse Jenkins instances in Y-build tests on Linux?
   At the moment specific open-jdks are download for each build:
   

   eclipse.platform.releng.aggregator/JenkinsJobs/YBuilds/Y_unit_linux.groovy

   Lines 5 to 7 in 768f042

   	[javaVersion: 17, javaHome: '''installJDK('https://download.java.net/java/GA/jdk17.0.2/dfd4a8d0985749f896bed50d7138ee7f/8/GPL/openjdk-17.0.2_linux-x64_bin.tar.gz')''' ],
   	[javaVersion: 21, javaHome: '''installJDK('https://download.java.net/java/GA/jdk21/fd2272bbf8e04c3dbaee13770090416c/35/GPL/openjdk-21_linux-x64_bin.tar.gz')''' ],
   	[javaVersion: 24, javaHome: '''installJDK('https://download.java.net/java/early_access/jdk24/18/GPL/openjdk-24-ea+18_linux-x64_bin.tar.gz')''' ]

But EF Jenkins provides all these JDKs already, even EA builds for not yet released versions:
See 'openjdk-ea-latest' at https://github.com/eclipse-cbi/jiro/wiki/Tools-(JDK,-Maven,-Ant)#openjdk
@fredg02 can you tell how often the EA builds are updated for all Jenkins instances?

IMHO, the specific JDK version is not relevant here, as long as we have some "good" EA build of the current stream.

To the JDT-devs: Would you be fine with changing to the provided 'openjdk-ea-latest' for the Y-build Java 24 tests and switch to temurin for all others? Just like all other builds: https://github.com/eclipse-cbi/jiro/wiki/Tools-(JDK,-Maven,-Ant)#eclipse-temurin

No objections from my side.

[jarthana]

Does the JDT team mind if obsolete Y-build test jobs, i.e. those that are no longer in use, are removed from https://ci.eclipse.org/releng/job/YPBuilds/ or do you need them for some time for reference?

IMHO old jobs can be removed once the current set of jobs (build & test) is functional.

Let's perhaps just wait for one more +1 from @jarthana or @mpalat

No objections from me either.

1. Is there a reason why Y-build tests don't run on windows? Is it just to save precious computation time on the windows machine?

Just checked with @MohananRahul who says that it's configured but not configured to run automatically.

To the JDT-devs: Would you be fine with changing to the provided 'openjdk-ea-latest' for the Y-build Java 24 tests and switch to temurin for all others? Just like all other builds: https://github.com/eclipse-cbi/jiro/wiki/Tools-(JDK,-Maven,-Ant)#eclipse-temurin

I agree with Stephan. No problem for me.

[jarthana]

1. Is there a reason why Y-build tests don't run on windows? Is it just to save precious computation time on the windows machine?

Just checked with @MohananRahul who says that it's configured but not configured to run automatically.

Rahul just dug this up for me, thanks Rahul!
#457
Still doesn't explain why, though.

[fredg02]

@fredg02 can you tell how often the EA builds are updated for all Jenkins instances?

They are mostly updated on demand.

[HannesWell]

Thanks for your agreement, I have unified this with:

• [Build] Use pre-installed Temurin JDKs in I-/Y-build tests on Linux #2690

Rahul just dug this up for me, thanks Rahul!
#457

Thanks. Then just leave it as it is for now. The load is already high enough.

FYI after the I-build and all it's tests have been moved to Java-21 I applied the same change for the Y-build:

• [Y-Build] Move Y-build and its tests on Java-21 and above #2691

[stephan-herrmann]

Test job https://ci.eclipse.org/releng/job/YPBuilds/job/ep435Y-unit-linux-x86_64-java24/ aborts since 2024-12-31.

Error is:

java.lang.OutOfMemoryError: Java heap space
Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: a030fa77-332d-4e01-8dec-f10f51ace8d9
Caused: java.io.IOException: Remote call on JNLP4-connect connection from 10.40.71.91/10.40.71.91:47894 failed
	at hudson.remoting.Channel.call(Channel.java:1116)
	at hudson.FilePath.act(FilePath.java:1228)
	at hudson.FilePath.act(FilePath.java:1217)
	at PluginClassLoader for junit//hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:146)
	at PluginClassLoader for junit//hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:177)
	at PluginClassLoader for junit//hudson.tasks.junit.JUnitResultArchiver.parseAndSummarize(JUnitResultArchiver.java:282)
	at PluginClassLoader for junit//hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:62)
	at PluginClassLoader for junit//hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:27)
	at PluginClassLoader for workflow-step-api//org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:857)
Finished: FAILURE

Could this be connected to changes done here?

[HannesWell]

Test job https://ci.eclipse.org/releng/job/YPBuilds/job/ep435Y-unit-linux-x86_64-java24/ aborts since 2024-12-31.

Error is:

java.lang.OutOfMemoryError: Java heap space

Could this be connected to changes done here?

I saw that as well, but since the jobs are more and more unified (which is the goal of this issue) and Java-23 respectively I-builds don't show this I was hoping that the cause is an increased memory consumption (or maybe even a memory leak?) with java-24. Is that possible?

To help us in finding the cause I replayed the latest build, with -XX:+HeapDumpOnOutOfMemoryError enabled (as described in https://www.baeldung.com/java-heap-dump-capture#automatically):
https://ci.eclipse.org/releng/job/YPBuilds/job/ep435Y-unit-linux-x86_64-java24/15/

Maybe that's something we should enable in general because it has no cost if no OOM error happens and helps to track down memory issues (also for the builds not only the tests)?

[stephan-herrmann]

I was hoping that the cause is an increased memory consumption (or maybe even a memory leak?)

As the error happens while parsing test results, the leak might simply be a flood of failures that blows up memory. But due to the error we cannot see those test failures :-/

I wonder if the memory limit is defined by the executor or as a build parameter?

[HannesWell]

I was hoping that the cause is an increased memory consumption (or maybe even a memory leak?)

As the error happens while parsing test results, the leak might simply be a flood of failures that blows up memory. But due to the error we cannot see those test failures :-/

Good point. I didn't look at the logs at that detail 😅

I wonder if the memory limit is defined by the executor or as a build parameter?

For Linux the tests run in a kubernetes pod and we can indeed use a pod with increased resource limits by replacing

agent {
  label 'ubuntu-2404'
}

with

agent {
  kubernetes {
    inheritFrom 'ubuntu-2404'
    yaml """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: "jnlp"
    resources:
      limits:
        memory: "8Gi"
        cpu: "4000m"
      requests:
        memory: "4Gi"
        cpu: "2000m"
"""
  }
}

I have started another replay where I applied this:

• https://ci.eclipse.org/releng/job/YPBuilds/job/ep435Y-unit-linux-x86_64-java24/16

You can see the exact diff in the job configuration at (you probably have to be logged in into Jenkins):

• https://ci.eclipse.org/releng/job/YPBuilds/job/ep435Y-unit-linux-x86_64-java24/16/replay/diff

[HannesWell]

It failed again :/
But you could have a look at the test result files directly:
https://ci.eclipse.org/releng/job/YPBuilds/job/ep435Y-unit-linux-x86_64-java24/16/artifact/workarea/Y20250104-1000/eclipse-testing/results/xml/

Some of them have multiple MiB, but I don't see an error or failure. And although the number of tests didn't change, the size of org.eclipse.jdt.core.tests.compiler suite almost doubled.
Maybe the configuration changed? E.g. I see 22 hits for
<testcase classname="org.eclipse.jdt.core.tests.compiler.regression.TestAll" name="test067 - 16" (randomly chosen)

Maybe this also leads to an endless loops in the test-result reading step?

[stephan-herrmann]

It failed again :/ But you could have a look at the test result files directly: https://ci.eclipse.org/releng/job/YPBuilds/job/ep435Y-unit-linux-x86_64-java24/16/artifact/workarea/Y20250104-1000/eclipse-testing/results/xml/

Some of them have multiple MiB, but I don't see an error or failure. And altought the number of tests didn't change, the size of org.eclipse.jdt.core.tests.compiler suite almost doubled. Maybe the configuraiton changed? E.g. I see (randomly choosen) 22 hits for <testcase classname="org.eclipse.jdt.core.tests.compiler.regression.TestAll" name="test067 - 16"

Maybe this also leads to a endless in the test-result reading step?

Thanks! This led to a candidate culprit:
The recent merge from master to BETA_JAVA24 removed -Dcompliance.jre.24=1.8,11,17,21,24 from the test invocation resulting in all tests being executed at 16 distinct compliance levels rather than the intended 5. This didn't surface in PR builds, because the bogus change was in test.xml, which is not used in maven builds, but apparently this is what still drives tests during production builds.

Over to eclipse-jdt/eclipse.jdt.core#3517

[merks]

Nice teamwork. 👍

[HannesWell]

I have just pushed another unification of the result website via:

• [I/Y-Build] Unify index.template.php site for I and Y builds #2754

The result shouldn't be really different as before.

@stephan-herrmann I'm also reading your thread on the jdt-dev mailing list about how to compile ECJ on the Y-build not with the latest I-build but with the latest Y-build (if I'm not mistaken that's the actual intention isn't it?).
I have an idea for that, but have to verify it first before implementing it. I'll try to propose a patch in the next days.

[stephan-herrmann]

@stephan-herrmann I'm also reading your thread on the jdt-dev mailing list about how to compile ECJ on the Y-build not with the latest I-build but with the latest Y-build (if I'm not mistaken that's the actual intention isn't it?).

I think you got me right. There's just two things I'm not 100% sure about:

• at least in PR builds we use the compiler built in the very same build, I assume this is not happening in any production builds?
• you say "to compile ECJ", but isn't it actually about all compilation during that build, not just ECJ self-compilation?

At the bottom line, the same dog fooding from beta branches, as I-builds do for the master branch would be great (and for the rare use case discussed recently, this would prevent a strange hiccup).