Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the option to use an alternative compiler for compiling classes in JavaBuilder #2497

Merged
merged 15 commits into from
Jun 19, 2024

Conversation

testforstephen
Copy link
Contributor

@testforstephen testforstephen commented May 27, 2024

This is a follow up to #2490

What does in this PR:

  • Introduce a system property (e.g., -DAbstractImageBuilder.compilerFactory=x.y.z.MyCompilerFactory) to specify the compiler factory that produces the compiler for compiling classes in AbstractImageBuilder. Refer to MockCompiler in the test case org.eclipse.jdt.core.tests.builder.BasicBuildTests.MockCompiler for an example of a custom compiler implementation, and see JavacCompiler for a real Javac implementation.
  • Aggregate compiler settings (such as source path, classpath, output, annotation processors, and other compiler options) into a CompilerConfiguration model for reuse by the custom compiler.
    • Regarding the annotation processing workflow in Eclipse, project importers like m2e convert the annotation processing configuration from the build file into a .factorypath file, which the APT plugin uses for annotation processing. When switching to an alternative compiler like Javac, which can handle annotation processing directly, we don't need to invoke the APT plugin during compilation but still need it to resolve the factorypath configuration.

@@ -5,6 +5,7 @@ Bundle-SymbolicName: org.eclipse.jdt.core.tests.builder; singleton:=true
Bundle-Version: 3.12.400.qualifier
Bundle-Vendor: %providerName
Bundle-Localization: plugin
Fragment-Host: org.eclipse.jdt.core
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This ensures the tests shares the same classloader as jdt.core plugin, and the AbstractImageBuilder in jdt.core can create an instance of the MockCompiler in the tests.

@testforstephen
Copy link
Contributor Author

thanks to @mickaelistria for the original implementation of this new option.

Copy link
Contributor

@jukzi jukzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In principle the change looks simple enough and purely technical. Please also add a Junit test for the case where there is a problem with a custom supplied builder. Currently that seems to be possibly inconsistent (fall back vs hardcoded change in annotation processing)

* @param project the project to participate in
* @param isTest whether the annotation processor path is for test code
* @return the annotation processor paths
* @since 3.38
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be better if such API changes could be avoided, given that is more like an experiment

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Annotation processing configurations, such as the processor path and output directory, are part of the standard compiler arguments. It is better to explicitly provide these configurations within the host builder (for instance, AbstractImageBuilder) when invoking the custom compiler.

Currently, jdt.core utilizes the CompilationParticipant mechanism to interact with the APT plugin for annotation processing tasks. Extending the CompilationParticipant class appears to be a natural choice for obtaining APT configurations. Any other ideas we could consider for accessing APT information?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any other ideas we could consider for accessing APT information?

Does your client code need to call or override that API or can jdt just typecast to its internal implementation?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does your client code need to call or override that API

My client code does not use this API. It's AbstractImageBuilder in jdt.core that calls the API to get the related information and pass the result to the client code.

can jdt just typecast to its internal implementation?

jdt.core doesn't depend on the apt plugin directly. It uses extension point to discover AptCompilationParticipant instance. Not sure how to use typecast.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let a new intermediate jdt internal class
AbstractAptCompilationParticipant extends CompilationParticipant, let
AptCompilationParticipant extends AbstractAptCompilationParticipant (jdt apt should be a friend of jdt),
downcast CompilationParticipant to AbstractAptCompilationParticipant if the participiant is an instance of that.
That's a little detour but avoids otherwise unneeded public API.
-- Or mark the new API as Provisional so that it could be removed again without further warning.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. thanks for suggestion.

Comment on lines 156 to 157
* Provisional API for experimental support for an alternative compiler
* and can be removed without notice.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks like a case for @noreference

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice. @noreference tag added. thanks.

/**
* @since 3.38
*/
public class CompilerConfiguration {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about this? Are we exposing this to the outside world too? If not, we need @noreference here as well. Otherwise, this could use some documentation.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As this seems on;y used by AbstractImageBuilder, what about just moving it in the same (internal) package?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not only used by internal AbstractImageBuilder. It will be also used by clients for retrieving the compiler options.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But aren't all the expected clients basically implementing AbstractImageBuilder (so already consuming this internal package) ?
While I think all this will need to be API at some point, as it's early stage, I think we'd rather keep as much as possible in "internal" packages, until we feel comfortable enough with the design to turn it API.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The clients don't implement AbstractImageBuilder, but the ECJ internal Compiler class. Yes, we can put the new CompilerConfiguration within internal package since it remains accessible from other plugin.

Just curious what's guideline to split the public package and internal package in the JDT codebase?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes, my bad!
The public package become API that we can then hardly modify and that we have to maintain as such. internal packages are basically not supported externally and can be changed more freely. So in general, unless there is a good reason for something to become API, we try to keep things internal as it makes further maintenance significantly easier.
But here, it can be a case where a new API class makes sense. I think it's actually good to have an actual object formalizing portable standard compiler configuration that could be used in multiple clients, rather than passing some options and (potentially unknown) participants.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exactly. Currently, project compiler-related settings are scattered across multiple places. Introducing a centralized class to formalize them would make them easily portable for other clients to reuse.

@mickaelistria
Copy link
Contributor

@testforstephen can this one be rebased and updated and turned from draft to PR already?

@testforstephen testforstephen marked this pull request as ready for review June 5, 2024 01:19
@testforstephen
Copy link
Contributor Author

@testforstephen can this one be rebased and updated and turned from draft to PR already?

Done.

Here’s the CI failure message. This CI error also occurs for other pull requests such as #2519. It seems to be a general environment issue when we start a new version (e.g., 4.33).

10:44:43  [ERROR] Failed to execute goal org.eclipse.tycho.extras:tycho-p2-extras-plugin:4.0.8:compare-version-with-baselines (compare-attached-artifacts-with-release) on project org.eclipse.jdt.core.compiler.batch: Baseline and reactor have the same fully qualified version, but different content
10:44:43  [ERROR] different
10:44:43  [ERROR]    META-INF/ECLIPSE_.RSA: not present in baseline
10:44:43  [ERROR]    META-INF/ECLIPSE_.SF: not present in baseline

Copy link
Contributor

@jukzi jukzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall it looks like its going in the right direction. Please fix the faiiing tests

@testforstephen
Copy link
Contributor Author

See detailed comments, of which I consider the following as mandatory:

  • A statement about whether/how this will be advertised

This is currently for experimental Javac support and not intended for a general audience. That's why we retain the internal class references and minimize intrusion into the existing mechanism.

  • A statement about supporting incremental builds

Already replied in the related comment.

  • A warning free workspace in Eclipse

The warning for require-bundle is fixed.

in Eclipse I see many "discouraged" warnings against org.eclipse.jdt.core.tests.builder.mockcompiler.

No idea for this. I don't see the warnings for them in the Eclipse IDE.

  • Error-free execution of RunBuilderTests.

Fixed.

@stephan-herrmann
Copy link
Contributor

in Eclipse I see many "discouraged" warnings against org.eclipse.jdt.core.tests.builder.mockcompiler.

No idea for this. I don't see the warnings for them in the Eclipse IDE.

I could still see it, perhaps related to what target platform one had configured in Eclipse, but once I updated to 1627800 the warnings disappeared. Theory: with the redundant Require-Bundle perhaps we saw classes from o.e.jdt.core via the wrong dependency-path: when seen as a regular dependency, access is discouraged, when seen as owned by the fragment host, all is fine.

Copy link
Contributor

@stephan-herrmann stephan-herrmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once we have a green build this is good to go - as the experimental option it is, perhaps to be revisited before the next release.

Copy link
Contributor

@mickaelistria mickaelistria left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree the current proposal is a (very) good enough iteration.
I think everything @stephan-herrmann suggested or remarked is relevant though, so if there is a chance to cover some of them easily right now, then I think it's better to do it before merging, so that's fewer reasons to come back to this topic later.
But if implemented the suggested change seems too much effort at the moment, I'm all for merging as is now.

@testforstephen testforstephen merged commit 52b659f into eclipse-jdt:master Jun 19, 2024
9 checks passed
@testforstephen testforstephen deleted the jinbo_compiler branch June 19, 2024 02:32
@testforstephen
Copy link
Contributor Author

@jukzi @stephan-herrmann @jarthana @mickaelistria @fbricon thanks everyone for the review. I appreciate all your feedback.

@mpalat
Copy link
Contributor

mpalat commented Jun 19, 2024

Hey @testforstephen [Jinbo Wang] - After pulling this change, Am getting compilation errors
[8 of them - putting the most relevant below:
Neither an available bundle nor the associated JRE export package 'org.eclipse.jdt.core.tests.builder.mockcompiler'
shown in Manifest]. Is anyone else facing an issue?

@jukzi
Copy link
Contributor

jukzi commented Jun 19, 2024

me2
image

@jarthana
Copy link
Member

Hey @testforstephen [Jinbo Wang] - After pulling this change, Am getting compilation errors [8 of them - putting the most relevant below: Neither an available bundle nor the associated JRE export package 'org.eclipse.jdt.core.tests.builder.mockcompiler' shown in Manifest]. Is anyone else facing an issue?

For the IDE, I guess you will have to import the new projects from the repo after pulling.

@akurtakov
Copy link
Contributor

Do you have org.eclipse.jdt.core.tests.builder.mockcompiler imported in your workspace?

@mpalat
Copy link
Contributor

mpalat commented Jun 19, 2024

Hey thanks all - yes, got it resolved. Didn't realize that a new project was added - a rare occurrence in jdt.core world - [and how stupid of me for not taking that possibility into consideration :( ] - thanks!

@jukzi
Copy link
Contributor

jukzi commented Jun 19, 2024

I am typical using OOmphs "Perform Setup Task" but in this case it does not work. Does the new project need to be registered somewhere for a OOmph setup? @merks

@merks
Copy link
Contributor

merks commented Jun 19, 2024

That depends. The ones with * requirement will find all projects, the ones with requirements listed will transitively include all those requirements:

image

So if there is something new somewhere in the clones that do not use * and are not required in any feature already required, then it may need to be added. Just to avoid reading the whole long thread, which specific project is of concern here?

@merks
Copy link
Contributor

merks commented Jun 19, 2024

Doing a pull and resolving the modular targlet resulted in:

Importing project org.eclipse.jdt.core.tests.builder.mockcompiler

So that appears to be working okay.

@jukzi
Copy link
Contributor

jukzi commented Jun 19, 2024

Just to avoid reading the whole long thread, which specific project is of concern here?

eclipse.jdt.core\org.eclipse.jdt.core.tests.builder.mockcompiler.project

@merks
Copy link
Contributor

merks commented Jun 19, 2024

FYI, the generally running just these two Help -> Perform Setup Tasks will install the latest and update the target platform to the latest, importing new projects as a side effect:

image

@jukzi
Copy link
Contributor

jukzi commented Jun 19, 2024

my complaint is that this:
image
did not import the new project.
I just tried it again and it failed with

Collected 4 artifacts for https://download.eclipse.org/eclipse/updates/4.33-I-builds in 0.768s
Preparing to commit the provisioning operation.
Committing the provisioning operation.
ERROR: org.eclipse.oomph.targlets.core code=0 Resolution problems
  at org.eclipse.oomph.targlets.internal.core.TargletContainer.forceUpdate(TargletContainer.java:919)
  at org.eclipse.oomph.setup.targlets.impl.TargletTaskImpl$4.run(TargletTaskImpl.java:1232)
  at org.eclipse.oomph.util.pde.TargetPlatformUtil.runWithTargetPlatformService(TargetPlatformUtil.java:120)
  at org.eclipse.oomph.setup.targlets.impl.TargletTaskImpl.perform(TargletTaskImpl.java:1092)
  at org.eclipse.oomph.setup.internal.core.SetupTaskPerformer.doPerformNeededSetupTasks(SetupTaskPerformer.java:3864)
  at org.eclipse.oomph.setup.internal.core.SetupTaskPerformer$WorkspaceUtil$1.run(SetupTaskPerformer.java:5200)
  at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2452)
  at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2477)
  at org.eclipse.oomph.setup.internal.core.SetupTaskPerformer$WorkspaceUtil.performNeededSetupTasks(SetupTaskPerformer.java:5193)
  at org.eclipse.oomph.setup.internal.core.SetupTaskPerformer.performNeededSetupTasks(SetupTaskPerformer.java:3798)
  at org.eclipse.oomph.setup.internal.core.SetupTaskPerformer.performTriggeredSetupTasks(SetupTaskPerformer.java:3773)
  at org.eclipse.oomph.setup.internal.core.SetupTaskPerformer.perform(SetupTaskPerformer.java:3651)
  at org.eclipse.oomph.setup.ui.wizards.ProgressPage$9.run(ProgressPage.java:592)
  at org.eclipse.oomph.setup.ui.wizards.ProgressPage$11$1.run(ProgressPage.java:721)
  at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)
  ERROR: org.eclipse.oomph.targlets.core code=0 Problems resolving composed target 'eclipse-sdk-prereqs'
    ERROR: org.eclipse.pde.core code=0 Problems occurred while resolving the target contents
      ERROR: org.eclipse.pde.core code=0 Problems loading repositories
        ERROR: org.eclipse.pde.core code=0 Unable to locate installable unit org.apache.lucene.core 9.11.0.v20240607-1000
        ERROR: org.eclipse.pde.core code=0 Unable to locate installable unit org.apache.lucene.analysis-smartcn 9.11.0.v20240607-1000
        ERROR: org.eclipse.pde.core code=0 Unable to locate installable unit org.apache.lucene.analysis-common 9.11.0.v20240607-1000
        ERROR: org.eclipse.pde.core code=0 Unable to locate installable unit org.apache.lucene.core.source 9.11.0.v20240607-1000
        ERROR: org.eclipse.pde.core code=0 Unable to locate installable unit org.apache.lucene.analysis-smartcn.source 9.11.0.v20240607-1000
        ERROR: org.eclipse.pde.core code=0 Unable to locate installable unit org.apache.lucene.analysis-common.source 9.11.0.v20240607-1000
      ERROR: org.eclipse.pde.core code=0 Problems loading repositories
        ERROR: org.eclipse.pde.core code=0 Unable to locate installable unit org.eclipse.emf.common.feature.group 2.32.0.v20240604-0832
        ERROR: org.eclipse.pde.core code=0 Unable to locate installable unit org.eclipse.emf.common.source.feature.group 2.32.0.v20240604-0832
        ERROR: org.eclipse.pde.core code=0 Unable to locate installable unit org.eclipse.emf.ecore.feature.group 2.38.0.v20240604-0832
        ERROR: org.eclipse.pde.core code=0 Unable to locate installable unit org.eclipse.emf.ecore.source.feature.group 2.38.0.v20240604-0832
        ERROR: org.eclipse.pde.core code=0 Unable to locate installable unit org.eclipse.emf.edit.feature.group 2.23.0.v20240604-0832
        ERROR: org.eclipse.pde.core code=0 Unable to locate installable unit org.eclipse.emf.edit.source.feature.group 2.23.0.v20240604-0832
        ERROR: org.eclipse.pde.core code=0 Unable to locate installable unit org.eclipse.emf.databinding.feature.group 1.12.0.v20240604-0832
        ERROR: org.eclipse.pde.core code=0 Unable to locate installable unit org.eclipse.emf.databinding.source.feature.group 1.12.0.v20240604-0832
        ERROR: org.eclipse.pde.core code=0 Unable to locate installable unit org.eclipse.emf.databinding.edit.feature.group 1.12.0.v20240604-0832
        ERROR: org.eclipse.pde.core code=0 Unable to locate installable unit org.eclipse.emf.databinding.edit.source.feature.group 1.12.0.v20240604-0832
Took 46 seconds.
There are failed tasks.
Press Back to choose different settings or Cancel to abort.

might be an unrelated error but prevented to import the new project?

@merks
Copy link
Contributor

merks commented Jun 19, 2024

Ever since we added resolving the actual *.target, this happens once in a while. If it still fails after a restart, open that *.target in the PDE Editor:

image

and wait for it to resolve properly.

I think you can also do there here in the preferences:

image

It's an annoying problem. 😢

@iloveeclipse
Copy link
Member

Back to original code now.

While thinking about comments on #2606 I've noticed that we use a system property to select compiler.

Sorry I haven't participated in the review here, is there any reason that the alternative compiler is supplied as a system property and not compiler option here?

String compilerFactoryClassName = System.getProperty(COMPILER_FACTORY_KEY);

Wouldn't be natural to read which compiler is to be used few lines above from project options in

Map projectOptions = this.javaBuilder.javaProject.getOptions(true);

If I read the code right, this would be possible as the compiler is created per project builder and that also would allow have two compilers running in same IDE? Also this would allow any set of tests running in parallel with both compilers etc, avoid mocking of system properties.

Of course one could still have that property for projects without settings.

@mickaelistria
Copy link
Contributor

is there any reason that the alternative compiler is supplied as a system property and not compiler option here?

Currently, the only reason is convenience: a system property allows to dynamically switch compiler in the IDE. They allow to repeat the same operation with a simple flag that can be easily controlled in a debugger to compare results, they also allow easy "external" configuration (eg JDT-LS can just add a -vmArg to force usage of a compiler or the other).
That said, your proposal might be better; but the idea to use something else than a system property hadn't emerged so far, maybe because it's not useful yet.

that also would allow have two compilers running in same IDE?

Yes, that would be possible with your proposal to have distinct projects using distinct compilers in parallel. Currently, the system property only allows to switch the compiler globally for all projects (but dynamically).

@iloveeclipse
Copy link
Member

@mickaelistria : so would you or @testforstephen follow up on that proposal?

@mickaelistria
Copy link
Contributor

@mickaelistria : so would you or @testforstephen follow up on that proposal?

This particular proposal is not so interesting for our current goals regarding Javac integration. However, if you are certain that this can be soon profitable for the JDT Core project, and since it's not too much code to change, I think we can invest the small effort to unleash extra value to the ecosystem. So the (genuine) question is now for you: do you really think if we develop it now, it's going to be useful soon?

@stephan-herrmann
Copy link
Contributor

Sorry I haven't participated in the review here, is there any reason that the alternative compiler is supplied as a system property and not compiler option here?

I see the system property as an implementation of the statement that this is an experimental flag not to be exposed to any wider audience. Right now we wouldn't want the option to (automatically?) appear in .settings/org.eclipse.jdt.core.prefs, for people to look at and start searching for documentation :)

Creating a compiler option can IMHO wait until we discuss making this an official feature.

robstryker pushed a commit to robstryker/eclipse.jdt.core that referenced this pull request Jul 18, 2024
…n JavaBuilder (eclipse-jdt#2497)

* Add the option to use an alternative compiler for compiling classes in JavaBuilder
gayanper pushed a commit to gayanper/eclipse.jdt.core that referenced this pull request Sep 7, 2024
…n JavaBuilder (eclipse-jdt#2497)

* Add the option to use an alternative compiler for compiling classes in JavaBuilder
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants