ELK Stack Kubernetes Logging: Effortlessly Set Up with Helm

John Abhilash

The ELK stack is a popular open-source logging and observability platform. It consists of three main components: Elasticsearch, Logstash, and Kibana.

- Elasticsearch is a distributed search and analytics engine that can be used to store and search large volumes of data.

- Logstash is a pipeline processor for ingesting, transforming, filtering, and storing logs.

- Kibana is a data visualization dashboard that allows users to explore and analyze Elasticsearch data.

Helm is a package manager for Kubernetes that makes it easy to install and manage complex applications.

This blog post will show you how to effortlessly set up the ELK stack on Kubernetes using Helm for logging.

Prerequisites

- A Kubernetes cluster

- Helm installed on your local machine

1.Installing the ELK stack with Helm

To install the ELK stack with Helm, you can use the following commands:

# Install Elasticsearch
helm install elasticsearch elastic/elasticsearch

# Install Kibana
helm install kibana elastic/kibana

# Install Logstash
helm install logstash elastic/logstash

# Install Filebeat
helm install filebeat elastic/filebeat

This will install the ELK stack with a default configuration. You can customize the configuration by passing a values.yaml file to the helm install command.

2.Configuring Filebeat to ship logs to Elasticsearch

Once you have installed all of the components of the ELK stack, you need to configure Filebeat to ship logs to Elasticsearch.

To do this, edit the Filebeat configuration file. The default location for this file is /etc/filebeat/filebeat.yml.

In the Filebeat configuration file, add the following lines:

output.elasticsearch:
  hosts: ["elasticsearch-master:9200"]

This will tell Filebeat to ship logs to the Elasticsearch cluster running on the elasticsearch-master service.

3.Starting the ELK stack

Once you have configured Filebeat, you can start the ELK stack by running the following commands:

# Start Elasticsearch
helm start elasticsearch

# Start Kibana
helm start kibana

# Start Logstash
helm start logstash

# Start Filebeat
helm start filebeat

4.Accessing Kibana

Once the ELK stack is running, you can access Kibana by visiting the following URL in your web browser:

http://<kibana-service-host>:<kibana-service-port>

The default port for the Kibana service is 5601.

5.Using Kibana to analyze logs

Once you have logged in to Kibana, you can start analyzing your logs. Kibana provides a variety of dashboards and visualizations that you can use to explore your logs.

To get started, you can use the Discover dashboard. This dashboard provides a basic overview of your logs, including the number of logs, the log sources, and the log types.

You can also use the Dashboard page to create your own custom dashboards. To do this, click the Create Dashboard button and select the types of visualizations that you want to add to your dashboard.

Once you have created a dashboard, you can use it to analyze your logs in more detail. For example, you can use the Line Chart visualization to see how the number of logs changes over time. You can also use the Pie Chart visualization to see the distribution of log types.

In this blog post, you have learned how to effortlessly set up the ELK stack on Kubernetes using Helm for logging. This will allow you to collect, store, and analyze logs from your Kubernetes applications.

6.Additional tips

- You can scale the ELK stack by increasing the number of Elasticsearch, Logstash, and Kibana replicas.

- You can use persistent volumes to store Elasticsearch and Logstash data.

- You can use Helm to upgrade the ELK stack to newer versions.

- You can use Logstash to filter and transform logs before they are indexed in Elasticsearch.

- You can use Kibana to create dashboards and visualizations to analyze your logs in more detail.

7.Troubleshooting

If you are having problems with the ELK stack, you can check the Elasticsearch, Logstash, and Kibana logs for more information. You can also check the Helm logs to see if there are any errors.If you are still having problems, you can ask for help on the Elasticsearch or Kubernetes mailing lists.

If you are looking for an easy way to manage and automate your cloud infrastructure, Sailor Cloud is a good option to consider. To learn more about Sailor Cloud, please visit the Sailor Cloud website: https://www.sailorcloud.io/