Custom log exporting options for in-VM services

[ginglis13]

What problem are you trying to solve

Today, the only option firecracker-containerd offers to expose logs from services running in-VMs is to enable forwarding of the VM journal to the serial console via command line option systemd.journald.forward_to_console=yes. The Firecracker team has seen issues with performance for enabling logging to the serial console and does not recommend its use in production:

Without proper handling, because the guest has access to the serial device, this can lead to unbound memory or storage usage on the host side. Firecracker does not offer users the option to limit serial data transfer, nor does it impose any restrictions on stdout handling.

Furthermore, we do not recommend that users enable the serial device in production.

As firecracker-containerd depends on Firecracker and users running in production, we can harden the security of the project by moving towards a solution that eliminates the need to write to the serial console for performance and security conscious use cases of firecracker-containerd. We can disable writing to the serial console and replace log forwarding to the console with a custom logging solution that ships logs over vsock.

Describe the solution you’d like

Firecracker supports vsock devices, which firecracker-containerd is already using to transfer data between guest and host. According to Firecracker, these devices are performant and safer compared to the serial console. The proposed solution will transfer logs over vsock devices from guest to host. This will require a few new components to be implemented in firecracker-containerd:

1. Guest Log Exporter
   1. process running in a VM that listens, accepts, and writes logs over a vsock connection.
   2. agnostic to logger (e.g. journald v.s. syslog)
   3. reference implementation will be provided by the firecracker-containerd project
2. Host Exporter Processes
   1. scraping processes, each of which initiates connection with a single VM for reading logs over vsock and writing to the host destination
3. Managing Process (firecracker-control)
   1. launches and reaps Host Log Reader processes as VMs are spun up and shut down

Describe alternatives you’ve considered

The main alternative considered so far is aggregating logs via the managing process, firecracker-control, and using batch writes to the configured logging destination rather than the proposed solution of each individual Host Log Exporter process writing to the host log destination.

Additional context

• Long boot time firecracker#792
• https://github.com/firecracker-microvm/firecracker/blob/main/docs/prod-host-setup.md#8250-serial-device
• Support optionally forwarding agent and systemd journal logs from within VM to the host for debugging #276
• Add send2vsock to send systemd journal from a guest to the host #352