Proposal: feature discovery / version "negotiation" flag-file

[squeed]

The problem

We at the CNI project have long held administrators wholly responsible for understanding and creating their CNI configurations. In other words, there is expected to be a single party who totally manages the contents of the CNI configuration file.

In today's world, this is rarely the case. CNI configurations are often generated by third-party networking providers, and are expected to function in a wide variety of environments and software versions.

A CNI configuration manager might want to know

• What is the highest supported CNI version?
• Are CNI drop-ins supported? (Proposal: drop-in directories #928)

Specification?

CNI as a specification does not make any statement about configuration management and discovery. Thus whatever is proposed is, currently, outside of the body of the specification and falls in to the murky world of libcni convention and inertia. Perhaps that shouldn't be the case. As part of CNI 2.0, we may wish to codify this behavior as well (some kind of discovery is clearly in the cards).

That said, better is the enemy of done, and I'd like to solve this without getting too far in the weeds.

Proposed solution:

libcni writes a file in its configuration directory indicating which versions and, possibly, features are supported. Something like

{
    "apiVersion": "v1",
    "kind": "CNIRuntimeInfo",
    "supportedVersions":["1.5", "1.0"],
    "supportedFeatures": {
        "dropins": false
    }
}

The absence of such a file would imply libcni v1.1.

[squeed]

This came to head as part of the discussion with STATUS (#859). Specifically, plugins need a way to know if they can expect CNI v1.1 behavior (STATUS), or if they should use pre-v1.1 behavior and write their configuration only when ready.

So far, the proposed solutions are:

1. v1.1 runtimes read from /etc/cni/net.d/cniv1.1/ as well as /etc/cni/net.d
   Pros: solves backwards-compat; could flag other features like dropin directories
   Cons: complicates runtimes (crio, containerd) that use fsnotify to watch for changes
2. Plugins write an invalid v1.0 but valid v1.1 config file
   Pros: Doesn't introduce any new files
   Cons: Total hack, relies on specific behavior from runtimes to not ignore invalid files
3. Switch to an entirely new directory
   Pros: Easy way to enable new features... once
   Cons: Need to update the entire world
4. Add a new filename extension (e.g. .conflist.1) to signify a v1.1 plugin (in not-ready state)
   Pros: Easy way to roll out this feature, backwards compatible
   Cons: May break chaining scripts (e.g. Istio)

Given that I also want to try for drop-in directories, I think a feature file is also inevitable. Just need to find a way to handle the downgrade case.

[danwinship]

libcni writes a file in its configuration directory

Uh, whose libcni? What if there are multiple "CNI-plugin-invokers" on the system that support different features?

We at the CNI project have long held administrators wholly responsible for understanding and creating their CNI configurations. In other words, there is expected to be a single party who totally manages the contents of the CNI configuration file.

In today's world, this is rarely the case. CNI configurations are often generated by third-party networking providers, and are expected to function in a wide variety of environments and software versions.

I don't think "expecting the administrator to manage the configuration" is wrong. The administrator knows what they want. If systems have "CNI configurations ... generated by third-party networking providers", it is almost certainly because the administrators of those systems installed those third-party networking providers themselves, and are planning to use them.

The problem, rather, is that we require there to be a single CNI configuration file containing internal implementation details of every CNI plugin the administrator wants to use, mixed together with site-specific information known only to the administrator.

What we need to do is to separate out the concepts of plugin configurations and network configurations, so we can say that plugins ship with their own static plugin configurations, explaining how to invoke them, and what capabilities they have, and what CNI versions they support, etc.

And then, after installing all the plugins they need, the administrator creates a network configuration (maybe using a template provided by one of the plugin authors) referencing the plugin configurations it needs to use (but not including them inline). And the CNI-plugin-invoker can figure out the intersection of features and supported versions and such from there.

Are CNI drop-ins supported?

Drop-ins seem like the answer to the wrong problem. No plugin should be trying to automatically change the configured behavior of the network(s) on the system. Plugins just offer functionality. Only the administrator configures functionality.

plugins need a way to know if they can expect CNI v1.1 behavior (STATUS), or if they should use pre-v1.1 behavior and write their configuration only when ready.

So one other possible solution to this is that the administrator knows whether the CNI-plugin-invoker and the CNI plugin both support the v1.1 STATUS behavior and so configures them accordingly. (eg, putting writeCNIConfigAtStartup: false in the right config file)

[squeed]

libcni writes a file in its configuration directory

Uh, whose libcni? What if there are multiple "CNI-plugin-invokers" on the system that support different features?

That is the single biggest issue with this proposal, and the thing I find most distasteful. I completely agree.

The problem, rather, is that we require there to be a single CNI configuration file containing internal implementation details of every CNI plugin the administrator wants to use, mixed together with site-specific information known only to the administrator.

Even worse than that, some of the site-specific information comes from internal details of the CNI plugin.

What we need to do is to separate out the concepts of plugin configurations and network configurations
...
Plugins just offer functionality. Only the administrator configures functionality.

I agree this is the ideal form. Especially with respect to chaining, only the administrator understands the desired network configuration - not k8s, not the runtime, not the plugin vendor(s). That said, we don't make it easy. Neither the tooling nor practice is there to make managing CNI configuration (or files on disk, for that matter) easy in a k8s world. And, like you correctly observed, a CNI configuration file is a mix of administrative intent, internal plugin details, and boilerplate.

So, understandably, we have third-party vendors doing the work for the administrator. Which works most of the time, until we get in to this sort of situation.

So one other possible solution to this is that the administrator knows whether the CNI-plugin-invoker and the CNI plugin both support the v1.1 STATUS behavior and so configures them accordingly. (eg, putting writeCNIConfigAtStartup: false in the right config file)

Right, but inevitably vendors and administrators shy away from these sort of flags. "Just autodetect!" they all cry (they don't care, but let's pretend). So vendors add duct-tape to try and do some sort of sane defaulting.

plugins ship with their own static plugin configurations, explaining how to invoke them, and what capabilities they have, and what CNI versions they support, etc.

This could even be a CNI verb (i.e. negotiated) rather than a file on disk, but the principle is the same. Either way, the problem still remains: we need an extremely low-friction way to specify the ordered set of plugins and their network-specific configuration. I can dream up a couple options regarding auto-discovery, but they all scare me a bit from a security perspective.

Perhaps there is something we can do to make this simpler for the broader ecosystem.

[danwinship]

I can dream up a couple options regarding auto-discovery

I really think we want less auto-discovery. Administrators have a specific networking intent. They should just specify that intent, rather than having to juggle plugins in exactly the right way to ensure that the correct intent gets auto-detected.

[squeed]

At this point in time, general consensus in the meeting is that a flag file from libcni is probably not the right idea, especially in the case of multiple consumers.

For the case of plugins wanting to easily append themselves, we should instead work on a slightly longer-term project to make CNI configuration files as simple as possible. Right now, they have a mix of plugin-internal protocol details as well as administrator intent.