Kubernetes Logging with Fluent Bit v1.0.4

Fluent Bit is a lightweight and extensible Log Processor that comes with full support for Kubernetes:

• Read Kubernetes/Docker log files from the file system or through systemd Journal
• Enrich logs with Kubernetes metadata
• Deliver logs to third party storage services like Elasticsearch, InfluxDB, HTTP, etc.

This repository contains a set of Yaml files to deploy Fluent Bit which consider namespace, RBAC, Service Account, etc.

Getting started

Fluent Bit must be deployed as a DaemonSet so that it will be available on every node of your Kubernetes cluster. To get started run the following commands to create the namespace, service account and role setup:

$ kubectl create namespace logging
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-service-account.yaml
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role.yaml
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role-binding.yaml

Alternatively, you can use Kustomize, which is also built into newer versions of kubectl.

$ kustomize build github.com/fluent/fluent-bit-kubernetes-logging/output/kustomize | kubectl create -f -

Fluent Bit to Elasticsearch

The next step is to create a ConfigMap that will be used by our Fluent Bit DaemonSet:

$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-configmap.yaml

Fluent Bit DaemonSet ready to be used with Elasticsearch on a normal Kubernetes Cluster:

$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-ds.yaml

Or, if you are using kustomize:

$ kustomize build github.com/fluent/fluent-bit-kubernetes-logging/output/elasticsearch/ | kubectl create -f -

Fluent Bit to Elasticsearch on Minikube

If you are using Minikube for testing purposes, use the following alternative DaemonSet manifest:

$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-ds-minikube.yaml

Or, if you are using kustomize:

$ kustomize build github.com/fluent/fluent-bit-kubernetes-logging/output/elasticsearch-minikube | kubectl create -f -

Fluent Bit to Kafka

Create a ConfigMap that will be used by our Fluent Bit DaemonSet:

$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/kafka/fluent-bit-configmap.yaml

Fluent Bit DaemonSet ready to be used with Kafka on a normal Kubernetes Cluster:

$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/kafka/fluent-bit-ds.yaml

Or, if you are using kustomize:

$ kustomize build github.com/fluent/fluent-bit-kubernetes-logging/output/kafka | kubectl create -f -

Details

The default configuration of Fluent Bit makes sure of the following:

• Consume all containers logs from the running Node.
• The Tail input plugin will not append more than 5MB into the engine until they are flushed to the Elasticsearch backend. This limit aims to provide a workaround for backpressure scenarios.
• The Kubernetes filter will enrigh the logs with Kubernetes metadata, specifically labels and annotations. The filter only goes to the API Server when it cannot find the cached info, otherwise it uses the cache.
• The default backend in the configuration is Elasticsearch set by the Elasticsearch Output Plugin. It uses the Logstash format to ingest the logs. If you need a different Index and Type, please refer to the plugin option and do your own adjustments.
• There is an option called Retry_Limit set to False that means if Fluent Bit cannot flush the records to Elasticsearch it will re-try indefinitely until it succeeds.

Get in touch with us!

Your contribution to testing is highly appreciated. We aim to make logging cheaper for everybody so your feedback is fundamental. Please get in touch on:

• Mailing List / Google Group
• Slack Channel #fluent-bit