[Q&A] Cannot override fluentd config via k8s ConfigMap

[williamtrelawny]

What is a problem?

Following the instructions here, I'm trying to apply the 3 lines of code so that fluentd knows to apply the CRI parser to my logs instead of the default Docker parser. It says to use a ConfigMap to override the /fluentd/etc/tail_container_parse.conf file within the container, which is pretty straightforward. I'm also using the provided DaemonSet configuration for Graylog as a starting point.

The Fluentd repo provided an article on how to use your own custom config via ConfigMap, which says your ConfigMap should be mounted in /fluentd/etc/[filename.conf], so here is my DaemonSet manifest using that technique:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: fluentd
  labels:
    k8s-app: fluentd-logging
    version: v1
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-logging
      version: v1
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      # Enable tolerations if you want to run daemonset on control-plane nodes.
      # Recommended to disable on managed k8s.
      #tolerations:
      #- key: node-role.kubernetes.io/control-plane
      #  effect: NoSchedule
      #- key: node-role.kubernetes.io/master
      #  effect: NoSchedule
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1-debian-graylog-arm64
        imagePullPolicy: IfNotPresent
        env:
          - name: K8S_NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name:  FLUENT_GRAYLOG_HOST
            value: "graylog01.lab"
          - name:  FLUENT_GRAYLOG_PORT
            value: "12202"
          - name:  FLUENT_GRAYLOG_PROTOCOL
            value: "udp"
          - name:  FLUENTD_SYSTEMD_CONF
            value: "disable"
          - name: FLUENT_CONTAINER_TAIL_EXCLUDE_PATH
            value: >
                [
                "/var/log/containers/fluent*"
                ]
          #- name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
          #  value: /^(?<time>.+) (?<stream>stdout|stderr)( (?<logtag>.))? (?<log>.*)$/
        resources:
          requests:
            cpu: 200m
            memory: 0.5Gi
          limits:
            # ===========
            # Less memory leads to child process problems.
            cpu: 1000m
            memory: 1Gi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        # When actual pod logs in /var/log/pods, the following lines should be used.
        - name: containerlogdirectory
          mountPath: /var/log/pods
          readOnly: true
        # ConfigMap:
        - name: override-1
          mountPath: /fluentd/etc/tail_container_parse.conf
          subPath: tail_container_parse.conf
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      # When actual pod logs in /var/log/pods, the following lines should be used.
      - name: containerlogdirectory
        hostPath:
          path: /var/log/pods
      - name: override-1
        configMap:
          name: cri-parser

However, this fails with the following error:

Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/snap/microk8s/common/var/lib/kubelet/pods/54829439-dec1-4586-a6a7-33fe625c9b0c/volume-subpaths/override-1/fluentd/2" to rootfs at "/fluentd/etc/tail_container_parse.conf": mount /var/snap/microk8s/common/var/lib/kubelet/pods/54829439-dec1-4586-a6a7-33fe625c9b0c/volume-subpaths/override-1/fluentd/2:/fluentd/etc/tail_container_parse.conf (via /proc/self/fd/6), flags: 0x5001: not a directory: unknown

StackExchange says to resolve this I need to mount it as a directory with the filename itself as the value in the subPath argument, so I tried that:

# ConfigMap:
- name: override-1
  mountPath: /fluentd/etc/
  subPath: tail_container_parse.conf

...and the Pod fails because there is no /fluent/etc/fluent.conf file anymore, indicating the entire /fluent/etc/ folder has been overwritten by my volume mount, which makes sense.

So how am I supposed to override the config files in /fluent/etc if I can't mount the file itself and I can't mount the folder??

Describe the configuration of Fluentd

No response

Describe the logs of Fluentd

No response

Environment

- Fluentd version: 1.17.1
- TD Agent version:
- Fluent Package version:
- Docker image (tag):
- Operating system: 
- Kernel version: Linux fluentd-tlbg4 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:17 UTC 2024 aarch64 GNU/Linux

[kenhys]

It might be cri-parser does not define tail_container_parse.conf.

[williamtrelawny]

Sorry, forgot to follow up on this... This issue was my fault- I thought ConfigMaps contained the contents of the files specified in their config, not the files themselves. So in my volume mount I was referencing a file named tail_container_parse.conf but my ConfigMap contained a file named cri-override.conf which was the name of the file containing the cri parsing config on my host.

Changing all the filenames used to tail_container_parse.conf worked because now the ConfigMap had the correct filename that the volume mount was looking for.