Run Filebeat on Kubernetes

Stack

You can use Filebeat Docker images on Kubernetes to retrieve and ship container logs.

You deploy Filebeat as a DaemonSet to ensure there’s a running instance on each node of the cluster.

The container logs host folder (/var/log/containers) is mounted on the Filebeat container. Filebeat starts an input for the files and begins harvesting them as soon as they appear in the folder.

Everything is deployed under the kube-system namespace by default. To change the namespace, modify the manifest file.

To download the manifest file, run:

curl -L -O https://raw.githubusercontent.com/elastic/beats/9.1/deploy/kubernetes/filebeat-kubernetes.yaml

By default, Filebeat sends events to an existing Elasticsearch deployment, if present. To specify a different destination, change the following parameters in the manifest file:

- name: ELASTICSEARCH_HOST
  value: elasticsearch
- name: ELASTICSEARCH_PORT
  value: "9200"
- name: ELASTICSEARCH_USERNAME
  value: elastic
- name: ELASTICSEARCH_PASSWORD
  value: changeme

Kubernetes control plane nodes can use taints to limit the workloads that can run on them. To run Filebeat on control plane nodes, you may need to update the Daemonset spec to include proper tolerations:

spec:
 tolerations:
 - key: node-role.kubernetes.io/control-plane
   effect: NoSchedule

If you are using Red Hat OpenShift, you need to specify additional settings in the manifest file and enable the container to run as privileged. Filebeat needs to run as a privileged container to mount logs written on the node (hostPath) and read them.

1. Modify the DaemonSet container spec in the manifest file:

   securityContext:
     runAsUser: 0
     privileged: true
   

2. Grant the filebeat service account access to the privileged SCC:

   oc adm policy add-scc-to-user privileged system:serviceaccount:kube-system:filebeat
   

   This command enables the container to be privileged as an administrator for OpenShift.

3. Override the default node selector for the kube-system namespace (or your custom namespace) to allow for scheduling on any node:

   oc patch namespace kube-system -p \
   '{"metadata": {"annotations": {"openshift.io/node-selector": ""}}}'
   

   This command sets the node selector for the project to an empty string. If you don’t run this command, the default node selector will skip control plane nodes.

In order to support runtime environments with Openshift (for example, CRI-O, containerd), you need to configure the following path:

filebeat.inputs:
- type: filestream
  id: container-${data.kubernetes.container.id}
  prospector.scanner.symlinks: true
  parsers:
    - container: ~
  paths:
    - /var/log/containers/*.log

1. All filestream inputs require a unique ID.

2. Container logs use symlinks, so they need to be enabled.

3. The same path needs to be configured in the autodiscover settings, if enabled:

   filebeat.autodiscover:
     providers:
       - type: kubernetes
         node: ${NODE_NAME}
         hints.enabled: true
         hints.default_config:
           type: filestream
           id: container-${data.kubernetes.container.id}
           prospector.scanner.symlinks: true
           parsers:
             - container: ~
           paths:
             - /var/log/containers/*.log
   

Filebeat comes packaged with various pre-built Kibana dashboards that you can use to visualize logs from your Kubernetes environment.

If these dashboards are not already loaded into Kibana, you must install Filebeat on any system that can connect to the Elastic Stack, and then run the setup command to load the dashboards. To learn how, see Load Kibana dashboards.

The setup command does not load the ingest pipelines used to parse log lines. By default, ingest pipelines are set up automatically the first time you run Filebeat and connect to Elasticsearch.

To deploy Filebeat to Kubernetes, run:

kubectl create -f filebeat-kubernetes.yaml

To check the status, run:

$ kubectl --namespace=kube-system get ds/filebeat

NAME       DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE-SELECTOR   AGE
filebeat   32        32        0         32           0           <none>          1m

Log events should start flowing to Elasticsearch. The events are annotated with metadata added by the add_kubernetes_metadata processor.

The application logs from workloads running on Kubernetes are usually in JSON format. In such cases, special handling can be applied to parse the JSON logs properly and decode them into fields.

You can configure Filebeat autodiscover to identify and parse JSON logs in two different ways:

• Using templates and ndjson parser options
• Using hints and annotations

We will illustrate this using an example of one pod with two containers where only the logs of one container are in JSON format.

Example log:

{"type":"log","@timestamp":"2020-11-16T14:30:13+00:00","tags":["warning","plugins","licensing"],"pid":7,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}

To use this method to parse the JSON logs in our example, configure autodiscover:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      templates:
        - condition:
            contains:
              kubernetes.container.name: "no-json-logging"
          config:
            - type: filestream
              id: container-${data.kubernetes.container.id}
              prospector.scanner.symlinks: true
              parsers:
                - container: ~
              paths:
                - /var/log/containers/*-${data.kubernetes.container.id}.log
        - condition:
            contains:
              kubernetes.container.name: "json-logging"
          config:
            - type: filestream
              id: container-${data.kubernetes.container.id}
              prospector.scanner.symlinks: true
              parsers:
                - container: ~
                - ndjson:
                    target: ""
                    add_error_key: true
                    message_key: message
              paths:
                - /var/log/containers/*-${data.kubernetes.container.id}.log

To configure autodiscover to parse JSON logs using this method, it is important to annotate the pod to only parse logs of the correct container as JSON logs. To achieve this, construct the annotations like this:

co.elastic.logs.<kubernetes_container_name>/json.keys_under_root: "true"
co.elastic.logs.<kubernetes_container_name>/json.add_error_key: "true"
co.elastic.logs.<kubernetes_container_name>/json.message_key: "message"

For the example we're using:

1. Configure autodiscover:

   filebeat.autodiscover:
     providers:
       - type: kubernetes
         node: ${NODE_NAME}
         hints.enabled: true
         hints.default_config:
           type: filestream
           id: container-${data.kubernetes.container.id}
           prospector.scanner.symlinks: true
           parsers:
             - container: ~
           paths:
             - /var/log/containers/*-${data.kubernetes.container.id}.log
   

2. Then annotate the pod:

   annotations:
     co.elastic.logs.json-logging/json.keys_under_root: "true"
     co.elastic.logs.json-logging/json.add_error_key: "true"
     co.elastic.logs.json-logging/json.message_key: "message"
   

Refer to the official Kubernetes documentation on log rotation.

Filebeat supports reading from rotating log files. However, some log rotation strategies can result in lost or duplicate events when using Filebeat to forward messages. For more information, refer to Log rotation results in lost or duplicate events.