Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bom generations takes a very long time for multi-module projects #259

Closed
EvgeniaPatsoni opened this issue Jan 11, 2023 · 8 comments
Closed

Bom generations takes a very long time for multi-module projects #259

EvgeniaPatsoni opened this issue Jan 11, 2023 · 8 comments

Comments

@EvgeniaPatsoni
Copy link

Hello guys.

I have a multi-module project (~60 sub-modules), for which I need to generate a bom.xml file, in order to upload it to a Dependency Track instance.

Using the default configuration, made me realize that cyclonedx was generating a bom file for each and every submodule, as well as a centralized one at the parent level. This behavior, caused the bom generation process to take about 15 minutes to complete, while the actual project built takes half that time.

After some research, I came across the inherited property which I then disabled, in order for cyclonedx to only generate a centralized bom file, and not several, seperate ones. However, this reduced the bom generation time only a bit: it dropped from 15 to 10 minutes.

My current configuration (inside the parent pom.xml), is as-follows:

    <profile>
      <id>cyclonedx-bom</id>
      <activation>
        <property>
          <name>cyclonedx-bom</name>
        </property>
      </activation>
      <build>
        <plugins>
          <plugin>
            <groupId>org.cyclonedx</groupId>
            <artifactId>cyclonedx-maven-plugin</artifactId>
            <inherited>false</inherited>
          </plugin>
        </plugins>
        <pluginManagement>
          <plugins>
            <plugin>
              <groupId>org.cyclonedx</groupId>
              <artifactId>cyclonedx-maven-plugin</artifactId>
              <version>2.7.3</version>
              <executions>
                <execution>
                  <phase>package</phase>
                  <goals>
                    <goal>makeAggregateBom</goal>
                  </goals>
                </execution>
              </executions>
              <configuration>
                <schemaVersion>1.4</schemaVersion>
                <includeBomSerialNumber>true</includeBomSerialNumber>
                <includeCompileScope>true</includeCompileScope>
                <includeProvidedScope>true</includeProvidedScope>
                <includeRuntimeScope>true</includeRuntimeScope>
                <includeSystemScope>true</includeSystemScope>
                <includeTestScope>false</includeTestScope>
                <includeLicenseText>false</includeLicenseText>
                <outputReactorProjects>false</outputReactorProjects>
                <outputFormat>xml</outputFormat>
                <outputName>bom</outputName>
              </configuration>
            </plugin>
          </plugins>
        </pluginManagement>
      </build>
    </profile>

Is there anything else I could do in order to further reduce the bom generation times?
I haven't been able to find any other solutions so far.

Thank you very much in advance,
Jenny
@stevespringett
Copy link
Member

CycloneDX Maven Plugin v2.7.4 solves the performance issue. I'd highly recommend upgrading to that version, which also includes many other improvements.

@EvgeniaPatsoni
Copy link
Author

Hello @stevespringett

I tried upgrading to v2.7.4 as you suggested, but that caused the project to crash with the following errors:

[INFO] --- cyclonedx-maven-plugin:2.7.4:makeAggregateBom (default) @ server ---
[INFO] CycloneDX: Parameters
[INFO] ------------------------------------------------------------------------
[INFO] schemaVersion          : 1.4
[INFO] includeBomSerialNumber : true
[INFO] includeCompileScope    : true
[INFO] includeProvidedScope   : true
[INFO] includeRuntimeScope    : true
[INFO] includeTestScope       : false
[INFO] includeSystemScope     : true
[INFO] includeLicenseText     : false
[INFO] outputFormat           : xml
[INFO] outputName             : bom
[INFO] outputReactorProjects  : false
[INFO] ------------------------------------------------------------------------

Downloading from maven-default-http-blocker: http://0.0.0.0/org/sonatype/plugins/nexus-staging-maven-plugin/maven-metadata.xml

[WARNING] Could not transfer metadata org.sonatype.plugins:nexus-staging-maven-plugin/maven-metadata.xml from/to maven-default-http-blocker (http://0.0.0.0/): transfer failed for http://0.0.0.0/org/sonatype/plugins/nexus-staging-maven-plugin/maven-metadata.xml

[WARNING] An unexpected issue occurred attempting to resolve the effective pom for  org.xerial.snappy:snappy-java:1.1.8.4
org.apache.maven.project.ProjectBuildingException: Some problems were encountered while processing the POMs:

[ERROR] Unknown packaging: bundle @ line 6, column 16
    at org.apache.maven.project.DefaultProjectBuilder.build (DefaultProjectBuilder.java:207)
    at org.apache.maven.project.DefaultProjectBuilder.build (DefaultProjectBuilder.java:342)
    at org.apache.maven.project.DefaultProjectBuilder.build (DefaultProjectBuilder.java:299)
    at org.cyclonedx.maven.BaseCycloneDxMojo.getEffectiveMavenProject (BaseCycloneDxMojo.java:449)
    at org.cyclonedx.maven.BaseCycloneDxMojo.convert (BaseCycloneDxMojo.java:406)
    at org.cyclonedx.maven.CycloneDxAggregateMojo.analyze (CycloneDxAggregateMojo.java:144)
    at org.cyclonedx.maven.BaseCycloneDxMojo.execute (BaseCycloneDxMojo.java:609)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:301)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:211)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:165)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:157)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:121)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:127)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:294)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)

Caused by: org.apache.maven.model.building.ModelBuildingException: 1 problem was encountered while building the effective model for org.xerial.snappy:snappy-java:1.1.8.4

@quentingosset
Copy link

I have exactly the same error with org.xerial.snappy if anyone has a solution I'm interested.

@hboutemy
Copy link
Contributor

given the error message, it's a protection by Maven 3.8+ against downloading dependencies from HTTP repositories (instead of secure HTTPS): see https://maven.apache.org/docs/3.8.1/release-notes.html for details

Can you try building with Maven 3.6.3 and confirm it works?

Notice I'm surprised that with the same Maven version, you can run older cyclonedx-maven-plugin version but not 2.7.4: AFAIK, it should do the same work at that level...

If you can share a sample project to reproduce, this would help a lot investigating

@EvgeniaPatsoni
Copy link
Author

Greetings @hboutemy and thank you for your reply.

Unfortunately I cannot downgrade to Maven 3.6.3, as the project uses jdk17.
However I tinkered the settings.xml file in order to be able to download from http repositories. See here.

Still receiving errors regarding xerial.snappy:

[INFO] --- cyclonedx-maven-plugin:2.7.4:makeAggregateBom (default) @ server ---
[INFO] CycloneDX: Parameters
[INFO] ------------------------------------------------------------------------
[INFO] schemaVersion          : 1.4
[INFO] includeBomSerialNumber : true
[INFO] includeCompileScope    : true
[INFO] includeProvidedScope   : true
[INFO] includeRuntimeScope    : true
[INFO] includeTestScope       : false
[INFO] includeSystemScope     : true
[INFO] includeLicenseText     : false
[INFO] outputFormat           : xml
[INFO] outputName             : bom
[INFO] outputReactorProjects  : false
[INFO] ------------------------------------------------------------------------
Downloading from microprofile.repo.eclipse.org: https://repo.eclipse.org/content/groups/microprofile/org/eclipse/microprofile/maven/microprofile-maven-build-extension/maven-metadata.xml
Downloading from repo.eclipse.org: https://repo.eclipse.org/content/groups/cbi/org/eclipse/microprofile/maven/microprofile-maven-build-extension/maven-metadata.xml
Progress (1): 365 B                  
Downloaded from microprofile.repo.eclipse.org: https://repo.eclipse.org/content/groups/microprofile/org/eclipse/microprofile/maven/microprofile-maven-build-extension/maven-metadata.xml (365 B at 187 B/s)
Downloading from jboss-public-repository-group: http://repository.jboss.org/nexus/content/groups/public/org/sonatype/plugins/nexus-staging-maven-plugin/maven-metadata.xml

[WARNING] An unexpected issue occurred attempting to resolve the effective pom for  org.xerial.snappy:snappy-java:1.1.8.4
org.apache.maven.project.ProjectBuildingException: Some problems were encountered while processing the POMs:
[ERROR] Unknown packaging: bundle @ line 6, column 16
    at org.apache.maven.project.DefaultProjectBuilder.build (DefaultProjectBuilder.java:207)
    at org.apache.maven.project.DefaultProjectBuilder.build (DefaultProjectBuilder.java:342)
    at org.apache.maven.project.DefaultProjectBuilder.build (DefaultProjectBuilder.java:299)
    at org.cyclonedx.maven.BaseCycloneDxMojo.getEffectiveMavenProject (BaseCycloneDxMojo.java:449)
    at org.cyclonedx.maven.BaseCycloneDxMojo.convert (BaseCycloneDxMojo.java:406)
    at org.cyclonedx.maven.CycloneDxAggregateMojo.analyze (CycloneDxAggregateMojo.java:144)
    at org.cyclonedx.maven.BaseCycloneDxMojo.execute (BaseCycloneDxMojo.java:609)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:301)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:211)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:165)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:157)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:121)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:127)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:294)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.apache.maven.model.building.ModelBuildingException: 1 problem was encountered while building the effective model for org.xerial.snappy:snappy-java:1.1.8.4
[ERROR] Unknown packaging: bundle @ line 6, column 16
    at org.apache.maven.model.building.DefaultModelProblemCollector.newModelBuildingException (DefaultModelProblemCollector.java:197)
    at org.apache.maven.model.building.DefaultModelBuilder.build (DefaultModelBuilder.java:568)
    at org.apache.maven.model.building.DefaultModelBuilder.build (DefaultModelBuilder.java:454)
    at org.apache.maven.model.building.DefaultModelBuilder.build (DefaultModelBuilder.java:267)
    at org.apache.maven.project.DefaultProjectBuilder.build (DefaultProjectBuilder.java:173)
    at org.apache.maven.project.DefaultProjectBuilder.build (DefaultProjectBuilder.java:342)
    at org.apache.maven.project.DefaultProjectBuilder.build (DefaultProjectBuilder.java:299)
    at org.cyclonedx.maven.BaseCycloneDxMojo.getEffectiveMavenProject (BaseCycloneDxMojo.java:449)
    at org.cyclonedx.maven.BaseCycloneDxMojo.convert (BaseCycloneDxMojo.java:406)
    at org.cyclonedx.maven.CycloneDxAggregateMojo.analyze (CycloneDxAggregateMojo.java:144)
    at org.cyclonedx.maven.BaseCycloneDxMojo.execute (BaseCycloneDxMojo.java:609)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:301)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:211)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:165)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:157)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:121)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:127)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:294)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)

I am afraid I cannot share a sample as well, since it's a private project.

I also do not understand why this is connected with the cyclonedx version in any way

Thanks again

hboutemy added a commit that referenced this issue Feb 4, 2023
@hboutemy
add dependency that has bundle packaging #259
a67bee2 
Signed-off-by: Hervé Boutemy <hboutemy@apache.org>
@hboutemy
Copy link
Contributor

hboutemy commented Feb 4, 2023 •
edited
Loading

ok, reading

[WARNING] An unexpected issue occurred attempting to resolve the effective pom for  org.xerial.snappy:snappy-java:1.1.8.4
org.apache.maven.project.ProjectBuildingException: Some problems were encountered while processing the POMs:
[ERROR] Unknown packaging: bundle @ line 6, column 16

I just tried adding this org.xerial.snappy:snappy-java:1.1.8.4 dependency to a project and was able to reproduce this Unknown packaging: bundle error message: see a67bee2

I now understand that it is a warning in the build output, it does not fail the build nor makes an invalid BOM content, even if that trace is frightening (was probably hidden exception in previous releases)

we should probably open a separate issue with an adapted description, instead of hijacking the current "Bom generations takes a very long time for multi-module projects"

@hboutemy hboutemy mentioned this issue Feb 4, 2023
Closed
@hboutemy
Copy link
Contributor

hboutemy commented Feb 4, 2023

closing this issue as performance issue was fixed in 2.7.4
for the other issue found in 2.7.4, followup will happen in #272

@hboutemy hboutemy closed this as completed Feb 4, 2023
@knrc
Copy link
Contributor

knrc commented Mar 16, 2023

FYI #306 drastically improves the performance of the aggregate bom generation, it took one of our complex builds down from ~30 mins to around 2 mins.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@hboutemy @knrc @stevespringett @quentingosset @EvgeniaPatsoni