Nowadays, it is quite common for systems to be made up of many subsystems. These include small programs that work together, full Operating Systems and proprietary applications.

This growing number of subsystems needs maintenance to assess their performance, detect potential vulnerabilities and correct errors. One of the main tools IT professionals have to keep that many subsystems operational are logs (i.e., big amounts of data presented in text traces that provide information on aspects that may be malfunctioning).

However, this recent increase in the number of subsystems, together with the introduction of microservice architecture in developments, makes it harder to work with logs. Since subsystems are interconnected, log traces often need to be cross-referenced to analyze or detect potential problems. In these scenarios, manually accessing the log files for each subsystem until a potential error is detected is unworkable.

The importance of “log centralization” has been growing for the past few years. It collects the logs of all subsystems and sends them to a server that runs a program where we can analyze them from a single viewpoint and in a much easier and optimal manner. This is known as “centralized log management”.

Centralized log management

When it comes to centralized log management, there are many alternatives (such as Datadog, Splunk Enterprise, Dynatrace, Graylog, etc.). However, all of them need the logs of each subsystem to be taken to the central management server. One of the best tools to carry out this task is Fluentd. Fluentd is an open-source log collector developed by Treasure Data to collect data from different sources. Having said this, the present blog is not going to talk in depth about Fluentd but will focus on its specific version for microservice architecture (Fluent Bit). This tool has also been developed by Treasure Data and its documentation can be found in the following link: https://fluentbit.io/

Fluent Bit allows us to collect logs from the different components that run in our microservice architecture (i.e., in any container orchestrator like Docker Swarm, Kubernetes, etc.). In addition to collecting the logs of each microservice we’re executing, it also collects the logs belonging to the host nodes found in the operating system and any other program running within it.

This blog will describe how to centralize logs in a Kubernetes infrastructure using Fluent Bit. This process is quite simple, since we can install Fluent Bit in our Kubernetes cluster as one more container (by downloading the image from the Docker Hub repository: https://hub.docker.com/r/fluent/fluent-bit/ 

We can then configure the logs we want to have collected through a configuration file or using a ConfigMap belonging to the Fluent Bit application within Kubernetes. This configuration file allows all stages the log collection processing flow is divided into to be configured. Further information on these stages can be found in the official documentation available at https://docs.fluentbit.io/manual/concepts/data-pipeline

Two ways in which to collect logs using Fluent Bit in Kubernetes

First, install Fluent Bit as a DaemonSet (i.e., deploy a replica in each host node in the cluster and set up a “hostpath” volume to link the Fluent Bit container to the file system in which logs are generated inside the host node). This way, by means of the configuration, we can specify the paths log files leave in each node (including container-generated logs). Fluent Bit will then collect them and send them to the log management server.

The second option is to run Fluent Bit as a Sidecar within an existing pod and set up a shared volume for the existing and Fluent Bit containers. If we configure the existing container so that it generates logs in the path where this volume is set up, we can collect those logs using Fluent Bit and send them to the centralized log management server. Despite having to individually configure the different Fluent Bit sidecars to collect the logs from that container (and that container alone), we have access to logs in plain text format and that can prove very useful when, for example, we have multi-line logs and we need to parse them in a customized manner.

We can simultaneously use both Fluent Bit log-collecting tools in Kubernetes, since they are complementary.

In Teldat we use Fluent Bit technology for our SDN / WAN products. Using Fluent Bit technology is one of many points that has enabled Teldat to develop a unique SDN / SD-WAN solution for our customers.