[Performance] Full build takes more time since 2024-09

[fedejeanne]

Hi,
I've been looking into some performance issues that started with 2024-09, specifically the time it takes to perform a full build. Interestingly enough, the time it takes to do a clean all also increased in 2024-09 but the biggest increase happened in the build phase.

• In 2024-06 it takes ~12 minutes to do a full clean and build.
• In 2024-09 it takes ~30 minutes to do a full build (sampled after the clean all)

samples.zip

The part that caught my eye was the increase in amount of calls (Hits) and therefore the Self Time (CPU) for the method(s) org.eclipse.jdt.internal.compiler.lookup.TypeBinding.findSuperTypeOriginatingFrom(...) (there are 2 flavors of this method).

2024-06

• 431 hits
• ~20s spent in these methods
  

2024-09

• 6972 hits
• ~9m spent in these methods
  

Question 1

• Why are there suddenly so many more calls to these methods ?

Question 2

• Would it be OK to simply build a cache to speed up some subsequent repeated calls or would this introduce any undesired effects (e.g. inconsistencies)?

I ask because I noticed that many of the calls are "repeated" i.e. the input parameters are the same so a cache would be a trivial optimization.

[iloveeclipse]

@fedejeanne : could you please share the project that reproduces the performance degradation, or provide something where one can see the regression?

If you can't share the code, could you please bisect the regression down to the commit? Or could you please check whether this is also reproducible on Linux (I see Windows specific platform code in the profiler).

@jukzi : since you work on performance and especially on Windows, did you noticed any degradation like described here?

[fedejeanne]

I can't share the code since it's our tool, but I can tell you it has ~2000 projects (plugins, fragments and features).

I can't easily test in Linux, but I can tell you the the code is inside folders that are excluded from the virus scanner, so that shouldn't be an issue. I will try to provide more details regarding the Windows side and come back to Linux if necessary.

[jukzi]

o that shouldn't be an issue

Unfortunately there may be a difference between "should" not and "is" not. The virus scanner may be informed a file is accessed and then the scanner may take a while to decide what todo. And some do even inject code in the running java application :-(

[jukzi]

Thanks for notifying me. Its interesting for us. I am not aware of such regression(s).
Please split it into two separate issues:

1. clean
2. findSuperTypeOriginatingFrom

• Please add also timings with current i-build (2024-12).
• When sampling with VisualVm make sure to not have java.** excluded, to see the filesystem hotspots that are inside the JDK. I use this config:
  
• Make sure to add the *.nps result for download.
• Give a public available reproducer.

[fedejeanne]

I will look into it an try to provide the samples/profiling.

What would you recommend for me to profile for both scenarios (clean + build)?

In any case, I am creating samples for both scenarios using 2024-06 and the latest I-BUILD. I can include those too if you want.

[jukzi]

What would you recommend for me to profile for both scenarios (clean + build)?

i almost never profile but always sample:

Profile costs too much overhead, the timings are inaccurate, instrumenting the code takes too long. Only makes sense if one needs a call count for a certain class. But normally one does not know in advance what one should search for.

[fedejeanne]

Sorry, my wires got crossed and I read "profile" 😅

Good, I'm sampling with this configuration (I increased to a frequency of 20ms to avoid degrading the performance of the application):

I got these results:

• 2024-06: 2nd Clean all (after full build) = ~40 s
• 2024-06: Build all (clean workspace): = ~10 m
• I-BUILD 20241118-1800: 2nd Clean all (after full build) = ~36 s
• I-BUILD 20241118-1800: Build all (clean workspace): = ~33 m

Files: Samples.zip

Notice that I sampled the clean operation after the full build in order to take advantage of the "2nd access" that you mentioned here. I don't think this plays a role because the code is always inside a folder that is excluded from the virus scanner but I did it anyway.

[iloveeclipse]

I've created #3327.
This looks really bad. Let close this discussion and continue with the bug.

[jukzi]

And please also note, that file access /(during clean) heavyly depends on the filesystem, cache and virus scanner(s), so that typically the first access takes far longer then a second access. So be sure to measure the same scenario (either a 2nd round or a first round after creating all the files from scratch)

[iloveeclipse]

Ideally you could test on Linux too, because your will have no antivirus scanner overhead and we could also sort out whether this is Windows specific IO issue or JDT core regression.

[jukzi]

The allSuperPairsWithCommonGenericType shown in sampling screenshots has been modified in that time by cc0e66e

[iloveeclipse]

@stephan-herrmann : FYI

[iloveeclipse]

Let continue on #3327.