Resolve dependencies by namespace, label or installation name

[carolynvs]

Let's fill in the gaps around dependency management and resolution. If you like where this is going, give it a 👍 and please leave any feedback or suggestions! I'll leave this open for a week and if there's consensus, then I will submit a PEP.

Use Cases

Here are some use cases we have collected so far:

Exclusive

I need my own instance of my dependency

When a bundle is run, new installations are created for its dependencies. For example, an application requires a dedicated redis cache per installation. This is the only supported use case by Porter today.

Global One-to-Many

I just need my dependency to be present

I am writing the Porter Operator to work with Operator Lifecycle Manager (OLM). When I install the Porter Operator, OLM should be installed on the cluster. It should not be installed more than once.

Scoped One-to-Many

My micro-services share a common dependency

My application has 4 micro-services and a shared postgres database. When I install the first service, a database should be created for it. When I install the other services, I want to share the database (and the connection string) between them.

User Provided

I already installed the dependency previously

I don't want Porter to install SQL Server for me, but I do want Porter to help connect the outputs from a named installation into another bundle. For example, a manually set up on-premise shared database server, the 64 core beast that you keep in the office closet. I want to automatically inject the connection string for the database into the bundles I install.

Proposal

@steder had a great idea for how we can support more scenarios for resolving dependencies. We can reduce all of these scenarios to a single decision "GetOrCreate". The bundle declares a set of match criteria that is used to identify an existing installation that satisfies the dependency. If one cannot be resolved, then a new installation is created.

Currently, a dependency can be resolved via the bundle name and version. We can add additional criteria to give bundle author and end-users more control over reusing an existing installation for a dependency:

• installation labels
• installation namespace
• installation name

Labels

Every installation can have a set of labels defined, KEY=VALUE. Keys and values are strings.

Namespaces

Installations can be scoped to a namespace, or the namespace can be left unset to indicate that it is a global installation. When dependencies are resolved, Porter resolves the dependency using only the bundles in the global namespace and the current namespace if set.

Installation Name Criteria

When a user installs a bundle, they can tell Porter to use a specific named installation to satisfy a dependency.

Now you can express a dependency such as:

Exclusive

Use a label that would not exist already so that a new installation is created just for this bundle in the current namespace.

• bundle-name: redis
• label: owner="current bundle's installation name"
• namespace: current

Global One-to-Many

Install OLM and do not set a namespace. The Porter Operator is installed into a namespace, and it reuses the global installation of OLM. When a match cannot be found, OLM is installed without a namespace set.

• bundle-name: olm
• namespace: global ("")

When more than one installation matches, installations in the current namespace are preferred over global installations.

Scoped One-to-Many

The micro-services that depend upon it use a namespace criteria for their dependency on postgres. The first micro-service won't find a match, and installs postgres in the current namespace. The subsequent micro-services reuse that installation.

• bundle-name: postgres
• namespace: current
• labels: app=my-application

The labels let us have multiple applications (which can be comprised of micro-services) installed in a namespace and only share a postgres installation between micro-services within the same application.

Unmanaged

SQL Server is either set up by a Porter bundle or manually because it's existing infrastructure. For existing infra, you define a bundle that just exposes the connection string via an output. The bundle itself doesn't modify any resources.

• installation-name: shared-dev-sql-server

Managed vs Unmanaged Dependencies

Currently when a Porter action is executed on a bundle, that action is automatically executed on its dependencies as well. In many of the use cases above, this isn't the correct behavior.

New configuration at the bundle level and at the point when a bundle action is run should allow people to control this behavior.

Managed Dependencies

When the bundle author defines the dependency, they can declare if the dependency should be managed together with the current bundle as a unit (current behavior). This is useful when an application is decomposed into multiple bundles, but it is intended to be managed together as a single unit, e.g. meta bundles.

Unmanaged Dependencies

Dependencies that are not declared as managed are not upgraded automatically when the current bundle is upgraded. This is useful in cases where the dependency is shared with unrelated bundles such as a shared SQL Server installation.

Overrides

All bundle actions (install, upgrade, invoke, uninstall) have a common set of flags that let the user override the dependency behavior set in the bundle. The user can decide to install a bundle without its dependencies, manually passing in data that would have come from the dependency as credentials. They can upgrade just a single bundle, without upgrading its dependencies. When it comes time to uninstall, they can choose whether or not to remove the dependencies when no other installations are using the dependency.

[carolynvs]

@MChorfa @MikeLarah I would really appreciate it if you are able to look this over and let me know if this seems workable or still has gaps.

[MikeEvansLarah]

I really like the flexibility of this approach. I think you have covered all the scenarios in the proposal.

The only other thing I can think of is the ability to conditionally include dependencies based on a bundle parameter. For example, you require a SQL Server instance but you want to specify whether that is your on-prem instance that already exists, or whether you want Porter to set up / manage that instance for you. Or is this scenario already covered through a combination of 'Global One-to-Many' and 'Unmanaged' use cases?

[carolynvs]

You're right that this doesn't cover conditionally requiring a dependency. The set of dependencies for a bundle is fixed at buildtime.

However, for the use case where your bundle requires SQL Server, the "unmanaged" scenario should cover that. You create a bundle that doesn't do anything other than output the connection string for your on-prem server. Then when you install bundles that need SQL Server, you can do something like this porter install --dep sql-server=our-shared-dev-server. That will let you wire up existing services (that may not have been installed by porter) as dependencies for your other bundles.

[MChorfa]

This is an awesome proposal 🚀. Thank you for this.
I would suggest to :

• Add an explain command that traverses the dependency graph of the managed and unmanaged dependencies
• Maintaining dependencies state for the managed and unmanaged use cases.
• Dependency operation completion state, where the other dependencies are aware of the result of a performed action and act of that
• Managed Dependencies: Ability to update one dependency whitout updating the other dependencies within the same bundle installation? not sure about this one

[carolynvs]

Yes, thanks for listing those out! I agree that all of these should be included in the PEP.

For the third one, are you thinking that each bundle should be given information about the larger action that is being performed that triggered the action up on a dependency? I want to make sure that I understand what state info the dependent bundles need access to.

[MChorfa]

My concern is if one dependency is in a faulty state and other dependencies would not be aware of it. I wouldn't say a Health check, but sort of, just to make sure to have a healthy state for the overall deployment. Probably this is not Porter responsibility, but the K8s one, make sure the desired state is maintained OR PorterOperator?

[carolynvs]

Oh I see. Porter will stop if the bundle reports a failure so I think this needs to be a responsibility of the bundle author to wait for the deployment to succeed or fail. For example, in this bundle that installs OLM it waits for the deployment before finishing install

https://github.com/getporter/operator/blob/c82ef7038d59c31d79d3af91cdbff4ef9699c2d7/installer-olm/helpers.sh#L36-L37

[carolynvs]

I just opened an issue in the CNAB spec proposing that we add support for namespaces and labels.

In the CLI, I'm thinking we can work with namespaces similar to kubectl.

~/.porter/config.toml

namespace = "demo"

You can change which config file you are using with PORTERCONFIG. To keep the config format simple (and avoid common problems with managing kubeconfig's) each file will contain only a single set of configuration options, and there won't be a context.

EXPORT PORTERCONFIG=porter-local.toml

You can change the namespace using either the config file, an environment variable PORTER_NAMESPACE or the --namespace flag.

EXPORT PORTER_NAMESPACE=demo
porter list --namespace demo

To make it easier to switch between namespaces, let's add a config command that helps you modify the config file:

porter config set namespace demo

It would be helpful to have additional config commands as well:

porter config create [PATH] # generate a commented out config.toml
porter config show # output the effective config taking into account environment variables
porter config path # print the path to the current config file

Specify the namespace in various ways

porter credentials list # Use the current namespace
porter credentials list azure --namespace demo
porter credentials list --namespace "" # Explicitly use the global namespace

In the example below, "azure" refers to the credential set defined in namespace demo. If a credential set is not defined in the demo namespace, porter will look for it in the global namespace

porter install porterops --namespace demo -c azure

[carolynvs]

PEP-003 has been assigned. Official proposal incoming 🚀