Implement "headless plugins" in a new plugin host outside of frontend connections

[cdamus]

What it does

Implement a new kind of plugin that I call "headless plugin" that is hosted in a child Node process like the current backend plugins except that this host is not started on a per-frontend-connection basis but exists independently of all such connections.
Thus it is "headless" as it will never know a Theia frontend.

This implements the functionality requested in #12946 and, because it makes no sense not to, #12959.

There are a few points to note about this implementation:

• I am not wedded to the term "headless plugin." I welcome suggestions for a better or more descriptive name, but I would like to avoid the term "backend" because we already have backend plugins that are the backend of a frontend connection, whereas headless plugins have no frontend. And the resulting similarity of names in the code will just be too confusing
• several of the existing interfaces and classes in the plugin-ext package are refactored to pull up common functionality to reuse for headless plugins with the aim of promoting maintainability of both plugin stacks over the long term. I have endeavoured to maintain API compatibility for Theia extenders, but please do keep a sharp eye out for compatibility issues
• a @theia/headless-plugin API declaration is provided that is essentially a greatly stripped-down variant of @theia/plugin, providing only capabilities that are available outside of frontend connections, and probably even less than that (see below for some possible follow-up in that area)
• it is intended by this PR that the support for "headless plugins" be optional. A Theia application could be built without the headless-plugin and headless-plugin-ext packages and work exactly as it does today. If the Theia team would rather just fold support for headless plugins into the existing plugin-ext package (e.g. because that would eliminate the need for some of the refactorings done in this PR, actually simplifying the code and possibly making it more correct) then I would be happy to do that
• the layout of the source tree in the new headless-plugin-ext extension tries to match the organization of corresponding code in the plugin-ext package, to make it easier to understand the similarities and differences. However, as there is no frontend to headless plugins, of course there are no browser directories so some things (such as XyzMain API) are organized differently, usually in the node/ folders
• refactoring in the Terminal Service Ext API merged after this PR was initially implemented required some late, and quick, adaptation that may not be ideal, such as the MinimalTerminalServiceExt type used by the AbstractPluginManagerExt class to declare only what it actually needs to do its work

How to test

This PR includes new examples demonstrating the use case that motivated issue #12946 that can be used for testing. These are:

• examples/api-provider-sample/: a new example Theia extension that demonstrates how to provide a custom API object for plugins to use. This is important because the original use case motivating the headless plugins feature is one in which those plugins exist to plug into such application-specific APIs. The API provides a "Greeting of the Day" service with custom functions to call, classes to create, and call-backs for a plugin to register for events
• sample-plugins/sample-namespace/plugin-gotd: an example dual headless/backend plugin that uses the greeting-of-the-day API of the other new example and also illustrates simple usage of the API provided by Theia, itself, to headless plugins. It further demonstrates how Theia plugins may provide both backend and headless entry-points to run in both plugin hosts

To test the intended use case of this feature, simply copy the plugin-gotd-v1.44.0.tgz tarball from the sample plugin into your plugins/ folder and start up the Browser Backend. You should see the new headless plugin host start up, load this plugin, and output from it on the console using [GOTD] tags to help you locate them in the output. Then hit the frontend in your browser to see that the connection-scoped backend plugin host does its usual thing, and that the GOTD plugin's backend entry-point runs in this context, printing messages tagged with [GOTD-BE] to distinguish them from messages printed by the headless plugin.

Then, of course, test the same behaviour in Electron as much as makes sense.

Apart from that, please go as crazy as you like in regression testing the usual plugin capabilities of Theia.

Follow-ups

• this PR does not yet include updates to user guides and other public documentation about the Theia plugin architecture. I will be happy to make such updates in this PR once we have agreed (if we do agree) to merge the implementation. Or I will be equally happy to raise a follow-up issue to tackle the docs. (Developer Guide Documentation for Headless Plugins #13290 )
• some no attempt has been made to support a few any core Theia services in the headless plugin host that should be available generally, outside of frontend connections. I don't really know how to test most of that so there may be problems found by early adopters that we need to fix. (Consider what default API should be supported for Headless Plugins #13292)
• also there may be other core Theia services that should be exposed in the new @theia/headless-plugin API that currently are not, contribution points and activation events to support in the package manifest that currently are not implemented. I should like to leave it to adopters to identify real-world needs in these areas that we can add later (Consider what default API should be supported for Headless Plugins #13292)
• this PR does not provide any default API (analogous to the theia or vscode API namespace) to plugins but relies on extensions and other plugins to provide APIs, as demonstrated by the new example GotD plugin and the API that it uses provided by the new example extension. It is an area of future consideration whether to provide some default API namespace and what capabilities it should offer, as use cases are discovered (Consider what default API should be supported for Headless Plugins #13292)

Review checklist

• As an author, I have thoroughly tested my changes and carefully followed the review guidelines

Reminder for reviewers

• As a reviewer, I agree to behave in accordance with the review guidelines

---

Updated 5 Dec 2023

This description is amended to rebrand the new kind of plugin as "headless plugin" (was "base plugin") and to adapt the test instructions for the now dual nature of the example GotD plugin, which previously supported only headless mode but now also backend (of a frontend connection) mode.

Updated 11 Dec 2023

This description is amended to reflect the withdrawal of the default API namespace that was provided to headless plugins.

Updated 21 Jan 2024

This description is amended to link to the follow-up tickets raised for follow-ups in the relevant section of the template.

[jcortell68]

I am not wedded to the term "base plugin."

I agree. backend plugin would be terribly confusing. Just putting this out there after starting to read your comments. How about headless plugin? Something better might come to mind, but that's what I have on my first hour after ten days away from work.

[jcortell68]

If the Theia team would rather just fold support for base plugins into the existing plugin-ext package (e.g. because that would eliminate the need for some of the refactorings done in this PR, actually simplifying the code and possibly making it more correct) then I would be happy to do that

We definitely would not want that given how many Theia front-end things plugin-ext currently pulls in. Someone should be able to at some point develop a completely headless Theia app that supports these new types of plugins, and with minimal bloat. That just wouldn't be possible if both plugin stacks were in the same package.

[cdamus]

We definitely would not want that given how many Theia front-end things plugin-ext currently pulls in. Someone should be able to at some point develop a completely headless Theia app that supports these new types of plugins, and with minimal bloat. That just wouldn't be possible if both plugin stacks were in the same package.

Currently the @theia/base-plugin-ext package has a dependency on @theia/plugin-ext because it reuses a lot of the latter's code. This is on the theory that Theia's greatest value proposition is in the support for a UI frontend connected to that backend so there's little incentive to support an absolutely headless Theia application (why not just use Node.js in that case?). The "base plugins" should work, however, in a scenario like CLI invocation of a Theia application that then doesn't need to bring up any front-end for that brief interaction.

To factor out the code from plugin-ext that base-plugin-ext requires would be feasible, perhaps introducing yet another new extension @theia/common-plugin-ext but per the above I'm not really convinced that the additional complexity is worth it.

[jcortell68]

so there's little incentive to support an absolutely headless Theia application (why not just use Node.js in that case?).

That's a good point. It's unlikely someone intending to write a CLI application would come to Theia. With the introduction of this new plugins stack, I suppose it makes the scenario more likely, though. I.e., if I wanted to write a CLI application in TypeScript and support third parties extending the application using a robust plugin mechanism that firewalls the main application from the plugins...then maybe??? But it's a stretch, I'll admit. I don't think the likelihood warrants further work at this point. We can cross that bridge if and when we get there.

A (slightly?) more likely scenario is a Theia app that developer that wants to allow extensions of his app only via this new type of plugin and not allow customization at the GUI level. There again...let's cross the bridge when we need to.

[cdamus]

Commit b79be18 updates the PR to

• change "base plugin" terminology to "headless plugin"
• support dual-entry-point backend plugins for the usual backend-of-a-frontend-connection host and also the headless plugin host process. This lets a single Theia plugin package provide both headless and backend plugins

The PR description is updated accordingly.

[tsmaeder]

I don't think "headless" is a good term to describe this kind of plugin because that is not the relevant distinction: I would argue most plugins would hook into some kind of UI. The distinction is scope: the back end plugins are shared among all front-ends. And "back end" is the right term here, since they are scoped to what we call the "back end" (as opposed to the Electron main process, for example). As with dependency injection, there is really no conceptual reason to not consider runtime extensibility for all parts of our system, including the Electron main process.

[tsmaeder]

I've looked at the PR on a very high level, and here are a couple of suggestions.

1. Split the plugin-ext package into plugin-ext, containing the mechanism to contribute and run plugin namespaces and plugin-ext-theia, which contributes the theia namespace. There is way too much code serving different purposes in this packages and any cleanup and separation of concerns would be a boon. It would also solve the "we're getting too much stuff in the back-end plugins" problem that @jcortell68 mentioned.
2. While we're at it: why not use dependency injection from the start in the plugin host? It will facilitate sharing ext-services between different api namespaces: that's one roadblock to making vscode and theia namespaces contributed instead of built-in.
3. Let's not have a default namespace: all back-end plugin namespaces should be contributed
4. Lets not contribute any namespaces by default: we currently have no idea what users might want
5. If we do decide to contribute any API in the back end, it should not be derived in any way from theia.d.ts: for one, the theia namespace is tightly bound to the vscode namespace, so we're not free to evolve it. Secondly, the requirements for for back end plugins might be very different from what the vscode API (of which theia.d.ts is basically a copy) might need. Let's not restrict ourselves.

[jcortell68]

I don't think "headless" is a good term to describe this kind of plugin because that is not the relevant distinction: I would argue most plugins would hook into some kind of UI. The distinction is scope: the back end plugins are shared among all front-ends. And "back end" is the right term here, since they are scoped to what we call the "back end" (as opposed to the Electron main process, for example). As with dependency injection, there is really no conceptual reason to not consider runtime extensibility for all parts of our system, including the Electron main process.

I respectfully disagree. I think it is precisely the relevant distinction. There are currently two types of plugins in Theia (and VS Code). One runs in the backend, the other in the frontend, but both are wired to the frontend. They drive functionality in the UI (the "head"). The new type of plugin this PR introduces not only lives in the backend but interacts only with the backend (of Theia extensions). Thus "headless" actually seems like a very suitable term. As one of the existing types of plugins today runs in the backend, I think calling this new thing a "backend plugin" would be pretty confusing.

since they are scoped to what we call the "back end" (as opposed to the Electron main process, for example).

The Electron main process is really two things running side-by-side in the same process: Node.js and the Chromium browser. I.e., it is both the frontend and the backend. Thus I'm not really understanding your point here.

[tsmaeder]

The Electron main process is really two things running side-by-side in the same process: Node.js and the Chromium browser. I.e., it is both the frontend and the backend. Thus I'm not really understanding your point here.

The "back end" is a process running an express server that is separate from the Electron main process.

[jcortell68]

The "back end" is a process running an express server that is separate from the Electron main process.

Oh wow. Just when I thought I was understanding Theia/electron/etc pretty well... I don't know how I missed that process when I've looked in process explorer.

[cdamus]

Commit 99b5002 removes the @theia/headless-plugin extension defining a default "theia" API namespace for headless plugins. It therefore also removes a lot of code supporting the implementation, initialization, and module-loading of that API.

It did require refactoring a few dependencies on concrete Ext implementations in the common plugin manager code to use instead internal interfaces that could be stubbed with no-ops for the headless plugin manager.

The GotD example plugin is updated to not attempt to use the API that no longer is available. Otherwise, everything seems to run as before (headless and VS Code plugins).

[jcortell68]

I wonder if we shouldn't be using activate/deactivate for the headless plugin entry points instead of start/stop. Theia has the notion of "starting" plugins which is a predecessor to activating the plugin. In fact, the starting of a plugin always happens as long as it meets the criteria of being a plugin. But Theia doesn't actually run any code in the plugin until an activation criteria has been met. I know that currently headless plugins don't support activation events, but eventually they will (as per previous discussions). And so we want to be able to distinguish that initial loading (starting) activity from the plugin actually being activated. Thus I think activate/deactivate are better and more consistent entry point names, IMO.

[cdamus]

I wonder if we shouldn't be using activate/deactivate for the headless plugin entry points instead of start/stop.

For whatever reason, start and stop are the function names expected in Theia plugins using the theiaPlugin engine. As our new headless plugins have no analogue in VS Code, it seemed appropriate to use the Theia-aligned names without some very compelling reason to diverge from them. The fact that internally in the plugin infrastructure there's a "starting" of plugins (which honestly doesn't really start anything) is not exposed in any way to plugin authors, so I don't see that this should be a point of confusion.

[jcortell68]

is not exposed in any way to plugin authors

It is exposed in the electron app's stdout.

It's yet another unfortunate discrepancy that is likely to cause confusion (another is the "extension" vs "plugin" naming thing). I believe most plugin development for Theia apps will be of the type VS Code extensions and headless plugins. I'm told people aren't really writing theiaPlugin types of plugins, so the start/stop choice there is mostly irrelevant. There is no need to try to stay consistent with a non-existent practice. I recommend we at least make the plugin mechanism support both. If we want to give precedent to start/stop in the implementation, that's fine. The documentation should point folks in the direction of activate/deactivate, though. IMO, it's more important to stay consistent with an active practice than an inactive one.

[cdamus]

Commit ccc1181 adds support for the "headless" entry point in a plugin that otherwise is an ordinary VS Code plugin. And because of this the expected start/stop lifecycle function names are changed to activate and deactivate respectively. This means that

• the VS Code plugin scanner is enhanced to pick up the theiaPlugin.headless entrypoint in the manifest if it happens to be specified
• the headless plugin scanner is updated to adapt Theia plugins by renaming their start/stop lifecycle function names
• the GotD example plugin is changed
  • to be a VS Code plugin, using the vscode API in its normal backend entrypoint script, now renamed extension.js as is tradition for VS Code plugins
  • in the headless.js script to rename its exported lifecycle functions

[jcortell68]

Commit a551b36 adds support for the "headless" entry point in a plugin that otherwise is an ordinary VS Code plugin...

Thank you!

[jcortell68]

@cdamus I'm still working with your initial commit, as I had to rebase your changes onto Theia 1.43.0 and that was a bit tricky. Bringing in any further changes might take a little time. So far, you haven't changed anything fundamental in the subsequent commits so I'll keep trucking as-is for now.

It appears that the headless plugin isn't activated unless I have the following in the plugin's package.json

"activationEvents": ["*"]

which is unexpected. My understanding is that there is no support yet for activation events, so I assume the plugin should be loaded unconditionally at this point.

[jcortell68]

Or maybe it's not that there is no support for activation events and that there just aren't any activation events to trigger off of. So you have to basically request to be loaded unconditionally with "*". If that's the case, then we'd for sure want to use a unique package.json element specifically for headless plugin--maybe headlessActivationEvents

[cdamus]

Or maybe it's not that there is no support for activation events and that there just aren't any activation events to trigger off of. So you have to basically request to be loaded unconditionally with "*". If that's the case, then we'd for sure want to use a unique package.json element specifically for headless plugin--maybe headlessActivationEvents

I had thought about this. There are no activation events specified by the framework, but I expect that applications will naturally find their own suite of events according to what their extension points are that headless plugins will supply. So I thought that applications could read the "activationsEvents" from the plugin manifests and manage activation themselves, which would mean that the "*" event still has meaning and in its absence a plugin should not be activated by the framework but that responsibility would be delegated to the application.

Does that make sense?

BTW, is there a problem with adding application-specific activation events that VS Code doesn't understand to the "activationEvents" object? I'd rather not reduplicate everything in the package manifest schema for headless plugins.

[jcortell68]

Does that make sense?

Yes. It does.

BTW, is there a problem with adding application-specific activation events that VS Code doesn't understand to the "activationEvents" object? I'd rather not reduplicate everything in the package manifest schema for headless plugins.

That's all fine and well when plugins specify actual activation events. We're sort of counting on there being no name collision on the activation events between the two types of plugins. Not ideal, but that will probably be fine. The real issue comes when one type of plugin wants to be activated by default ("*"). Under the current design/implementation, a "*" dictates the behavior for both types of plugins, which we really don't want. I might have a plugin package with a very lightweight headless plugin but a beast of a VS Code extension. If I need the headless plugin to be loaded by default, that means an unnecessary heavy penalty on the frontend.

One approach is to have a distinct string to represent "*" for headless. But that starts looking ugly. With the approach I recommended (headlessActivationEvents), we kill both of those birds with one stone. And I get it...that's not winning any beauty pageants either, but I think it is more future proof and arguably more intuitive.

[cdamus]

Commit dab5684 rebases the branch onto the latest master to resolve conflicts, including:

• new version numbers across the board
• intervening changes in the plugin host framework
• accounting for the new asynchronous execution of plugin deployment during the backend application start-up sequence

[cdamus]

How shall the DI container in the plugin host process be configured? Should the Theia app generator generate a new script to load ContainerModules referenced by a new package.json property (e.g. pluginHostExtensions) in similar fashion to the Container setup currently generated from the theiaExtensions property? And then presumably each object in the pluginHostExtensions array would have properties backend, frontend, and headless (to be renamed) to target injections to each of these plugin hosts.

Discussed off-line, it's clear that, in fact, we should be able to accomplish this using the plugin module loading already in place. As we need to load a module to access the start/stop functions and API factory of the plugin already, this can be expanded to access an Inversify container-module to configure our host's DI container.

[martin-fleck-at]

Hi Christian, thank you very, very much for that change! I had a look at the code and will do some runtime testing a little bit later. From a first glance everything looks good and I believe this feature will add a lot of value to Theia, especially with the introduction of Inversify on the extension host side. My comments are mostly regarding understanding and extensibility. As soon as everything is reviewed and agreed upon, we should consider whether the documentation properly explains the possibilities and pit falls. In any case, great work!

[cdamus]

Thanks @jfaltermeier and @martin-fleck-at for your reviews! I think I have addressed most, if not all, of your comments with commit 5e877b8. Overall I think this really helped to clean up and rationalize the code!

[jcortell68]

@cdamus Not sure if I'm confused, but at some point, this PR had a number of commits that reflected the evolving nature of the change. Today I see just one commit. Did you collapse all the commits into one and force push your branch?

[cdamus]

@cdamus Not sure if I'm confused, but at some point, this PR had a number of commits that reflected the evolving nature of the change. Today I see just one commit. Did you collapse all the commits into one and force push your branch?

Yes. There is a lot of concurrent activity in the keeping-up-with-VSCode-API sphere that requires occasional conflict resolution, and it's not feasible to maintain multiple commits because every commit requires manual conflict resolution every time. So I have to keep this squashed.

[martin-fleck-at]

Thank you very much for the detailed explanation and the updated PR. Everything looks good to me now from a code and runtime perspective. Great work! Since this is such a large PR, we'll ask another reviewer to have a look before merging it.

[msujew]

This looks alright to me. I can confirm that the feature works as expected and introduces no regressions in the plugins I've tested.

[cdamus]

Commit cffddc2 and successors are rebased to resolve the merge conflict by just deleting the changelog entry as it will be generated automatically.

[jcortell68]

Celebration and congratulations are in order!