container-hotplug

Hot-plug (and unplug) devices into a container as they are (un)plugged.

Description

Docker provides the --device flag to give a container access to a device. However the devices specified this way must be present when the container is created.

For dynamically created devices Docker provides the --device-cgroup-rule. However this requires knowing the device's major and minor numbers, which are dynamically allocated by the kernel. The rule accepts a glob * to mean "any minor" or "any major". However this would still give the container access to all the devices handled by a particular driver.

This program tries to solve that problem by listening to udev events to detect when a device is (un)plugged. It then interfaces directly with the container's cgroup to grant it access to that specific device.

To limit the devices the container can access, a root device is specified. The container will receive access to any device descending from the root device. This is particularly useful if the root device is set to a USB hub. The hub can be specified directly, or it can be specified as "the parent of device X", e.g., we can giving a container access to all devices connected to the same hub as an Arduino board.

Another concern is providing a container with well known paths for the devices. On bare-metal systems this would usually be achieved with a SYMLINK directive in a udev rule. This program tries to provide a similar functionality for containers, allowing you to specify symlinks for certain devices.

Usage

This tool wraps runc with the additional hotplug feature, therefore it can be used as a drop in replace for many container managers/orchestrators such as Docker, Podman, and Kubernetes. You need to ensure runc is available in your PATH so container-hotplug can find it.

It supports two annotations, org.lowrisc.hotplug.device and org.lowrisc.hotplug.symlinks.

For Docker, you can specify an alternative runtime by changing /etc/docker/daemon.json:

{
  "runtimes": {
    "hotplug": {
      "path": "/path/to/container-hotplug/binary"
    }
  }
}

and use it with the --runtime hotplug flag and appropriate annotation, e.g.

sudo docker run --runtime hotplug -it --annotation org.lowrisc.hotplug.device=parent-of:usb:2b2e:c310 ubuntu:latest

For podman, you can specify the path directly, by:

sudo podman run --runtime /path/to/container-hotplug/binary -it --annotation org.lowrisc.hotplug.device=parent-of:usb:2b2e:c310 ubuntu:latest

For containerd (e.g. when using kubernetes), you can edit /etc/containerd/config.toml to add:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.hotplug]
  runtime_type = "io.containerd.runc.v2"
  pod_annotations = ["org.lowrisc.hotplug.*"]

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.hotplug.options]
  SystemdCgroup = true
  BinaryName = "/path/to/container-hotplug/binary"

this would allow you to use hotplug as handler in k8s, e.g. add a runtime class with

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: hotplug
handler: hotplug

and use it in a pod with

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
  annotations:
    org.lowrisc.hotplug.device: parent-of:usb:0bda:5634
spec:
  runtimeClassName: hotplug
  containers:
  - name: ubuntu
    image: ubuntu:latest
    stdin: true
    tty: true

If you want symlinks to the tty devices created by interfaces 1 and 3 of the CW310, add

--annotation org.lowrisc.hotplug.symlinks=usb:2b3e:c310:1=/dev/ttyACM_CW310_0,usb:2b3e:c310:3=/dev/ttyACM_CW310_1

to docker/podman command line or

org.lowrisc.hotplug.symlinks: usb:2b3e:c310:1=/dev/ttyACM_CW310_0,usb:2b3e:c310:3=/dev/ttyACM_CW310_1

to k8s config.