RFC: Infrastructure to execute code with a project #25

coolya · 2018-07-30T22:43:27Z

This PR is not meant to be merged yet. I opened it to get feedback as early as possible and to discuss potential further work.

Infrastructure to execute code with a project

As we are more and more often required to execute certain tasks during the build the IdeaEnvironment becomes the go to solution. We also have some further plans to add new functionality to this plugin that requires access to a loaded MPS plugin (#21, #22). This PR aims to provide a common infrastructure to make these functionality as easy to use as possible. To do so the common bits of the execute-generators subproject were extracted.

Java API

The Java API is a single function executeWithProject it has been written in Kotlin but is perfectly callable from normal java code as a static function but is more idiomatic Kotlin and java. It might feel strange to java programmers to work with such API especially as it's more a functional style than normal OO.

The function takes all arguments required to setup the IdeaEnvironment plus a function that gets the project once it's loaded. The function can produce a value which is returned to the caller. The executeWithProject function also takes care of the complete lifecycle of the project and the IdeaEnvironment.

Discussion

Is the idiomatic Kollin API good enough for java developers or do we need a more OO like facet?
Are the use cases were caching the the IdeaEnvironment is essential over several project opening?
Error logging/handling is currently mostly missing

Command line API

In addition to the actual Java API I tried to standardise a command line API. As most of our current use cases spawn a new JVM rather then loading all of MPS into the same JVM as gradle we often need to pass information around via command line arguments. To avoid reinventing the wheel and having multiple incompatible implementations floating around I created a extensible command line argument definition. It takes all required parameters and allows passing them to a overload of the executeWithProject directly.

Application specific arguments can be added by extending the Args class.

Discussion

Do we want to standardise this interface?
The reuse is limited to Kotlin as the argument parser isn't really usable from oder JVM languages.

Reuse

The project-loader is reusable beyond the scope of this project. We could point others who want to do something with a headless MPS to this plugin and it's command line interface. That's why I would suggest publishing the project-loader as a separate maven publication. This comes with the downside that we will have to be more carefully of introducing breaking changes as we will not have all of the users under our control.

Open tasks:

update GenerateMpsProjectPlugin to use new command line API.

Moved all the logic required to start up a idea environment into the `project-loader` subproject. The subproject contains all code required to load plugins and open a project. The loader also takes care of the complete lifecycle of the idea application and the project. Users of The loader pass a lambda/function that takes the project as an argument. In case of any errors during the execution of the lambda or when the lambda is executed successfully the loader tries to gracefully shut down the idea environment. In addition to that the loader specifies a basic command line interface for potential users of the loader. The `execute-generators` subproject is the first user of the new `project-loader`. Since it required changes to the command line interface this is a breaking change.

coolya · 2018-09-07T08:31:22Z

I would appreciate any feedback on this PR.

sergej-koscejev · 2018-09-07T14:56:55Z

I'll review this on Monday.

sergej-koscejev

The biggest problem is the macros vs macro typo :) I leave the rest up to you to decide.

sergej-koscejev · 2018-09-10T09:41:24Z

project-loader/src/main/kotlin/Args.kt

+    { toPlugin(this) }
+
+    val macros by parser.adding("--macros",
+            help = "macro to define. The format is --macro=<name>:<value>")


so is this --macros or --macro?

sergej-koscejev · 2018-09-10T09:46:41Z

project-loader/src/main/kotlin/Loader.kt

+ * @param buildNumber optional build number that is used to determine if the plugins are compatible. Only guaranteed to
+ * work with MPS 2018.2+ but might work in earlier versions as well.
+ */
+fun <T> executeWithProject(project: File,


How about bundling the environment options into a class? I'm on the fence about this myself but maybe the class will be a natural place to add methods to in the future.

Ah ok, I see below that the Args class is playing that role. It's probably enough then.

sergej-koscejev · 2018-09-10T09:47:11Z

project-loader/src/main/kotlin/Loader.kt

+
+    if (pluginLocation != null) {
+        logger.info("overriding plugin location with: ${pluginLocation.absolutePath}")
+        System.setProperty(PROPERTY_PLUGINS_PATH, pluginLocation.absolutePath)


Should we set the property to the old value when we're finished? To clean up after ourselves.

sergej-koscejev · 2018-09-10T09:47:30Z

project-loader/src/main/kotlin/Loader.kt

+
+    if (buildNumber != null) {
+        logger.info("setting build number to \"$buildNumber\"")
+        System.setProperty(PROPERTY_PLUGINS_COMPATIBLE_BUILD, buildNumber)


sergej-koscejev · 2018-09-25T14:41:57Z

I would like to have this functionality accessible from Maven so I'm looking into this now.

A few years ago I wrote mps-maven-plugin where I used a similar approach but I used the GeneratorWorker and Script classes from mps-tool.jar to start the generator.

This would allow us to save some code and it seems to be a more "official" entry point since GeneratorWorker/Script are used by the generate Ant task. What do you think about that approach?

coolya · 2018-10-22T09:32:11Z

@sergej-koscejev Sorry for the very late reply. If I recall the details correctly, I think the major reason was that the MPS-generate task does require you to give it a calculated make script with all the libraries and also to calculate the chunks manually. Which the LangaugeWorkbenchMakeService does for you.

…ting

When using the tests we need to know the version ot the plugin we are testing against statically, because there is no code can run before the plugin block of a build script is evaluated. I added parameter to force a snapshot version that we can then use in the tests.

coolya · 2018-10-23T14:23:14Z

While manually testing the build scripts I found a bug in the argument parser and fixed it. I wanted to setup automated tests for the gradle scripts but this turned out to be harder than expected.

I needed to update to gradle 4.10 to get support for snapshot plugin versions as a start.

The next problem is testing against the plugin version that was just build. I tried as suggested in the documentation by using a composite build in the test project that includes the plugin build, but I was unable to get gradle to find the plugin with this setup.

The only workaround I found is that I publish a snapshot version of the plugin to maven local and the use maven local as a plugin repository in the tests. This seems to work but will required us to do some changes in the CI to have the tests execute correctly. I thought about using gradles testkit to implement the tests but then getting the required mps projects into the build location wasn't really straight forward. So I opted for adding independent gradle projects to the repo that use maven local to get the plugin.

To test this manually you need to do the following:

In the root project execute ./gradlew publishToMavenLocal this will put the gradle plugin (which is mostly a pom file) and the other artefacts into maven local
in test/generate-* run ./gradlew generate this will pickup the plugin from maven local eventually succeed generating the project.

I added two smoke tests, one that generates a simple java class and a second one that generate a build.xml file from the mps build project.

coolya · 2018-11-04T12:49:57Z

Form my perspective this is ready to be merged. See my above comments to see what I changed from the last review. I would also suggest that we now bump the version number to 1.1 which we forgot to do last time.

If anyone used the command line generation with the current version these changes would break the command line interface to the spawned java subprocess. So just to be sure I bumped the version to 1.1.

coolya requested review from sergej-koscejev, joao-silveira and arimer July 30, 2018 22:43

Merge branch 'master' into project-loader

57b8623

coolya changed the title ~~RFC/WIP: Infrastructure to execute code with a project~~ WIP/RFC: Infrastructure to execute code with a project Jul 30, 2018

coolya added 2 commits July 31, 2018 00:46

parser: use direct constructor invocation

6dd17bf

build: replace deprecated compile dependencies with implementation

7f3c816

coolya changed the title ~~WIP/RFC: Infrastructure to execute code with a project~~ RFC: Infrastructure to execute code with a project Aug 4, 2018

sergej-koscejev requested changes Sep 10, 2018

View reviewed changes

Merge branch 'master' into project-loader

d728a9e

address feedback from code review

09fe861

sergej-koscejev approved these changes Oct 22, 2018

View reviewed changes

coolya added 7 commits October 23, 2018 15:32

fix passing the project path

651999d

update to gradle 4.10.2 to allow snapshot versions of plugins for tes…

515f71a

…ting

pass state back if generation was successful

4ec603f

add smoke test for generating a build solution in a mps project

2452295

add smoke test for generating a simple java class

d2f2428

ignore files related to new mps test projects

a14cd58

bump version to 1.1.*

5333cf5

coolya mentioned this pull request Nov 4, 2018

model checking support #31

Merged

Merge branch 'master' into project-loader

5a7371e

coolya merged commit d0ba845 into master Nov 22, 2018

coolya deleted the project-loader branch November 22, 2018 16:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: Infrastructure to execute code with a project #25

RFC: Infrastructure to execute code with a project #25

coolya commented Jul 30, 2018

coolya commented Sep 7, 2018

sergej-koscejev commented Sep 7, 2018

sergej-koscejev left a comment

sergej-koscejev Sep 10, 2018

sergej-koscejev Sep 10, 2018

sergej-koscejev Sep 10, 2018

sergej-koscejev Sep 10, 2018

sergej-koscejev Sep 10, 2018

sergej-koscejev commented Sep 25, 2018

coolya commented Oct 22, 2018

coolya commented Oct 23, 2018 •

edited

Loading

coolya commented Nov 4, 2018

RFC: Infrastructure to execute code with a project #25

RFC: Infrastructure to execute code with a project #25

Conversation

coolya commented Jul 30, 2018

Infrastructure to execute code with a project

Java API

Discussion

Command line API

Discussion

Reuse

coolya commented Sep 7, 2018

sergej-koscejev commented Sep 7, 2018

sergej-koscejev left a comment

Choose a reason for hiding this comment

sergej-koscejev Sep 10, 2018

Choose a reason for hiding this comment

sergej-koscejev Sep 10, 2018

Choose a reason for hiding this comment

sergej-koscejev Sep 10, 2018

Choose a reason for hiding this comment

sergej-koscejev Sep 10, 2018

Choose a reason for hiding this comment

sergej-koscejev Sep 10, 2018

Choose a reason for hiding this comment

sergej-koscejev commented Sep 25, 2018

coolya commented Oct 22, 2018

coolya commented Oct 23, 2018 • edited Loading

coolya commented Nov 4, 2018

coolya commented Oct 23, 2018 •

edited

Loading