[RelEng] Let Tycho build a self-contained p2-repo with minimal content #1222

HannesWell · 2023-01-29T10:55:20Z

In the first build I just want to test tycho-p2-repository-plugin#includeAllSources, which IIRC hat some issues a while ago.

If that works the next step is to use includeAllDependencies together with filterProvided, introduced in eclipse-tycho/tycho#2028.

@laeubi FYI

github-actions · 2023-01-29T11:17:53Z

Test Results

107 files ±0 107 suites ±0 10m 29s ⏱️ - 2m 59s
661 tests ±0 651 ✔️ ±0 10 💤 ±0 0 ❌ ±0
661 runs ±0 650 ✔️ ±0 11 💤 ±0 0 ❌ ±0

Results for commit 53bdd9f. ± Comparison against base commit 8dd14db.

♻️ This comment has been updated with latest results.

HannesWell · 2023-01-29T12:41:16Z

Stage one word well, now lets try stage two.

mickaelistria · 2023-01-29T19:12:51Z

Note that this can also later allow to get rid of source features.

HannesWell · 2023-01-30T07:39:38Z

@laeubi currently this fails with:

[ERROR] Failed to execute goal org.eclipse.tycho:tycho-p2-repository-plugin:4.0.0-SNAPSHOT:assemble-repository (default-assemble-repository) on project org.eclipse.m2e.repository: Execution default-assemble-repository of goal org.eclipse.tycho:tycho-p2-repository-plugin:4.0.0-SNAPSHOT:assemble-repository failed. UnsupportedOperationException -> [Help 1]
 org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.eclipse.tycho:tycho-p2-repository-plugin:4.0.0-SNAPSHOT:assemble-repository (default-assemble-repository) on project org.eclipse.m2e.repository: Execution default-assemble-repository of goal org.eclipse.tycho:tycho-p2-repository-plugin:4.0.0-SNAPSHOT:assemble-repository failed.
     at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:375)
     at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
     at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
     at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
     at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:294)
     at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
     at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
     at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
     at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
     at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
     at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
     at java.lang.reflect.Method.invoke (Method.java:568)
     at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
     at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
     at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
     at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
 Caused by: org.apache.maven.plugin.PluginExecutionException: Execution default-assemble-repository of goal org.eclipse.tycho:tycho-p2-repository-plugin:4.0.0-SNAPSHOT:assemble-repository failed.
     at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:148)
     at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370)
     at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
     at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
     at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
     at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:294)
     at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
     at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
     at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
     at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
     at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
     at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
     at java.lang.reflect.Method.invoke (Method.java:568)
     at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
     at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
     at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
     at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
 Caused by: java.lang.UnsupportedOperationException
     at java.util.Collections$UnmodifiableCollection.removeIf (Collections.java:1096)
     at org.eclipse.tycho.p2tools.TychoMirrorApplication.collectUnits (TychoMirrorApplication.java:217)
     at org.eclipse.tycho.p2tools.copiedfromp2.MirrorApplication.run (MirrorApplication.java:222)
     at org.eclipse.tycho.p2tools.MirrorApplicationServiceImpl.mirrorReactor (MirrorApplicationServiceImpl.java:183)
     at org.eclipse.tycho.plugins.p2.repository.AssembleRepositoryMojo.execute (AssembleRepositoryMojo.java:234)
     at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
     at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370)
     at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
     at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
     at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
     at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:294)
     at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
     at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
     at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
     at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
     at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
     at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
     at java.lang.reflect.Method.invoke (Method.java:568)
     at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
     at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
     at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
     at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)

laeubi · 2023-01-30T07:53:18Z

I created:

Fix UnsupportedOperationException because of unmodifable set eclipse-tycho/tycho#2036

HannesWell · 2023-02-15T23:07:35Z

The build succeeds, but the created p2-repo only contains the slf4j.api.source_1.7.36.jar but not the main artifact.
Besides that the created repo looks good. I just noticed that there are many org.eclipse.wst.* plugins in in it, which reminds me of #1248.

HannesWell · 2023-02-15T23:12:48Z

Besides that org.yaml.snakeyaml_1.27.0.v20221112-0806.jar and its source bundle made it into the location of the https://download.eclipse.org/wildwebdeveloper/releases/1.0.1/ repo, which is also referenced.
But looking at its plugin folder I cannot find there, so maybe it is referenced there as well?

laeubi · 2023-02-16T04:13:39Z

The build succeeds, but the created p2-repo only contains the slf4j.api.source_1.7.36.jar but not the main artifact.

As we are most likely not building slf4j.api there is no main artifact, the jar itself might be already included in some referenced repo then.

HannesWell · 2023-03-01T19:09:37Z

The build succeeds, but the created p2-repo only contains the slf4j.api.source_1.7.36.jar but not the main artifact.

As we are most likely not building slf4j.api there is no main artifact, the jar itself might be already included in some referenced repo then.

AFAICT the referenced repos only contain the slf4j-api bundle in an older version and even with another BSN.
IMHO an artifact should only be left out if the referenced repo contains that artifact with the exact same BSN and version. Otherwise a p2-repo cannot contain newer artifacts e.g. fetched from Maven-Central.

laeubi · 2023-03-01T19:23:16Z

If you think there is a bug please try providing a reproducer :-)
The BSN actually don't matter as P2 do not know anything about that, P2 only knows IU ID + Version ... so if any repository contains the API bundle with the same IU ID+Version it is removed.

HannesWell · 2023-06-25T15:44:03Z

Using the new possibilities of Tycho's p2-repository-plugin look very promising and the result does very good so far.

Two things that I noticed:
For com.google.guava.source_32.0.1.jre and jakarta.servlet-api.source_5.0.0 only the source jars are included.
I doubt that the main jar is included in any of the referenced p2 repos since guava 32 is quite new.
m2e.maven.runtime is also happy with older guava versions, which are provided by other p2-repos, I guess that this could be the reason?
My first assumption about how this would work was that Tycho first computes the set of artifacts like when building a self-contained repo and then remove those artifacts that are contained in a p2-repo that is available for reference (in exactly the version that would have been in a physically self-contained one).

laeubi · 2023-06-25T16:51:42Z

You can look at org.eclipse.tycho.p2tools.TychoMirrorApplication.collectUnits(IQueryable<IInstallableUnit>, IProgressMonitor) if you are interested in the details, there are two steps:

A slice is created, in this phase all items are collected and probably sources are included
Then the units are collected, while collecting removes all units that are present in any referenced repository using IInstallableUnit#equals where the default implementations uses id and version equals.

currently there is no debug output but you can put a breakpoint in the remove method triggered on true to find out what repository contains the units already.

Additionally include all sources in the M2E-repo automatically. See news: https://github.com/eclipse-tycho/tycho/blob/master/RELEASE_NOTES.md#new-parameter-for-tycho-p2-repository-pluginassemble-repository https://github.com/eclipse-tycho/tycho/blob/master/RELEASE_NOTES.md#new-option-to-filter-added-repository-references-when-assembling-a-p2-repository

HannesWell · 2023-10-05T21:12:17Z

Now that Tycho 4.0.3 is available we can finally use this.
I checked the result and everything is expected, we have the fully dependency closure available in the m2e-repo but keep the repository at a minimal size since references to all p2-repos in the TP are added automatically and artifacts provided there are removed and at the same time unnecessary references are not added. The Orbit repo is explicitly excluded from being added as reference.

HannesWell force-pushed the simplifyM2ERepo branch from 7f8c3dd to c7a48ef Compare January 29, 2023 12:40

HannesWell force-pushed the simplifyM2ERepo branch from c7a48ef to abf3d67 Compare February 15, 2023 21:39

HannesWell force-pushed the simplifyM2ERepo branch from abf3d67 to ad35765 Compare February 15, 2023 23:08

akurtakov force-pushed the simplifyM2ERepo branch from ad35765 to 235e3aa Compare March 1, 2023 09:29

HannesWell force-pushed the simplifyM2ERepo branch from 235e3aa to d29d00f Compare June 25, 2023 11:33

HannesWell changed the title ~~[Build] Let Tycho include all sources in the M2E-repo automatically~~ [RelEng] Let Tycho build a self-contained p2-repo with minimal content Jun 25, 2023

HannesWell force-pushed the simplifyM2ERepo branch from d29d00f to 2b7b2d9 Compare June 25, 2023 15:35

HannesWell mentioned this pull request Aug 19, 2023

Switch to use Language Configurations and gramars from TM4E Language … eclipse-wildwebdeveloper/wildwebdeveloper#1252

Closed

HannesWell force-pushed the simplifyM2ERepo branch from 2b7b2d9 to 0acaaf3 Compare August 22, 2023 23:45

HannesWell force-pushed the simplifyM2ERepo branch from 0acaaf3 to 6a6bcd5 Compare September 1, 2023 22:41

HannesWell mentioned this pull request Sep 1, 2023

Add an option to prevent repo-references from being added to a p2-repo eclipse-tycho/tycho#2742

Merged

HannesWell force-pushed the simplifyM2ERepo branch from 6a6bcd5 to 503d5b5 Compare September 28, 2023 23:22

HannesWell mentioned this pull request Sep 28, 2023

Suppress addition of unnecessary repo-refereces when assembling p2-repos eclipse-tycho/tycho#2744

Merged

HannesWell force-pushed the simplifyM2ERepo branch 2 times, most recently from e7a16f7 to ce41a2b Compare September 28, 2023 23:50

HannesWell added 2 commits October 4, 2023 23:26

Build with Tycho 4.0.3

3977182

HannesWell force-pushed the simplifyM2ERepo branch from ce41a2b to 53bdd9f Compare October 4, 2023 21:32

HannesWell marked this pull request as ready for review October 5, 2023 21:08

HannesWell merged commit dc4aa02 into eclipse-m2e:master Oct 5, 2023
4 of 5 checks passed

HannesWell deleted the simplifyM2ERepo branch October 5, 2023 21:13

HannesWell mentioned this pull request Oct 31, 2023

master build issue with signatures #1590

Closed

HannesWell added this to the 2.5.0 milestone Nov 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RelEng] Let Tycho build a self-contained p2-repo with minimal content #1222

[RelEng] Let Tycho build a self-contained p2-repo with minimal content #1222

HannesWell commented Jan 29, 2023 •

edited

Loading

github-actions bot commented Jan 29, 2023 •

edited

Loading

HannesWell commented Jan 29, 2023

mickaelistria commented Jan 29, 2023

HannesWell commented Jan 30, 2023

laeubi commented Jan 30, 2023

HannesWell commented Feb 15, 2023

HannesWell commented Feb 15, 2023

laeubi commented Feb 16, 2023

HannesWell commented Mar 1, 2023

laeubi commented Mar 1, 2023

HannesWell commented Jun 25, 2023

laeubi commented Jun 25, 2023

HannesWell commented Oct 5, 2023

[RelEng] Let Tycho build a self-contained p2-repo with minimal content #1222

[RelEng] Let Tycho build a self-contained p2-repo with minimal content #1222

Conversation

HannesWell commented Jan 29, 2023 • edited Loading

github-actions bot commented Jan 29, 2023 • edited Loading

Test Results

HannesWell commented Jan 29, 2023

mickaelistria commented Jan 29, 2023

HannesWell commented Jan 30, 2023

laeubi commented Jan 30, 2023

HannesWell commented Feb 15, 2023

HannesWell commented Feb 15, 2023

laeubi commented Feb 16, 2023

HannesWell commented Mar 1, 2023

laeubi commented Mar 1, 2023

HannesWell commented Jun 25, 2023

laeubi commented Jun 25, 2023

HannesWell commented Oct 5, 2023

HannesWell commented Jan 29, 2023 •

edited

Loading

github-actions bot commented Jan 29, 2023 •

edited

Loading