Prometheus is an open-source automated monitoring and alerting system. It has become a widely accepted tool for monitoring highly dynamic container environments such as Kubernetes and Docker Swarm. It can also monitor traditional non-container infrastructures such as monitoring Linux/windows servers, single applications, Apache Server, and Database services.
When running Prometheus to monitor Kubernetes, it scrapes metrics data from the Kubernetes Cluster for monitoring the system. These metrics are CPU status, memory/disk space storage, request duration, number of running processes, server resources, and node status.
Grafana is a multi-platform that gets data from a data source such as Prometheus and transforms it into visualizations charts. This article will teach you how to integrate Prometheus and Grafana on Kubernetes using Helm.
Let’s start by discussing how Prometheus and Grafana work.
How Promethetheus works
When you install Prometheus in Kubernetes, it sets up Prometheus Server and Alert Manager.
The Prometheus Server
This is the core of Prometheus. It does the actual monitoring work and has three components:
- Data Retrieval Worker.
- Time Series Database.
- HTTP Server.
Data Retrieval Worker
This component scrapes and pulls the metrics data from the application. It then transforms the metrics data into time series data.
Time Series Database
This component stores the metric data in time series format.
HTTP Server
This component accepts queries for the stored time series data and displays the data in a web UI or dashboard. It can use the inbuilt/default Prometheus Web UI or a third-party platform.
We will use Grafana as the third-party platform for displaying the time series data. We will talk about Grafana later.
Alert Manager
It provides automated alerting through email or slack to the system administrator. It enables the administrator to know any changes in the system that Prometheus is monitoring in real-time.
Why Prometheus Monitoring is Important in DevOps
Prometheus is essential in monitoring highly dynamic container development environments and microservices infrastructure. Modern DevOps is becoming more and more complex. DevOps engineers are building complex infrastructures that are becoming difficult to handle/maintain and monitor manually. It, therefore, needs more automation such as Prometheus monitoring.
When you use Prometheus to monitor a Kubernetes cluster, it automatically manages all the microservices. It can then easily detect any failures in the processes running the microservices. It also provides quicker debugging of the Kubernetes Cluster to identify errors and failures.
When Prometheus detects an error in one of the microservices, it uses the Alert Manager component to notify the system administrator. The system administrator will then take the necessary measures to resolve the error.
Let’s summarize the importance of Prometheus as follows:
- Constantly monitoring the microservices running in the Kubernetes Cluster.
- Automatically alert the administrator when a microservice crashes.
- It reports on the cluster’s health and performance.
- Display the metric data on the Prometheus Web UI or a third-party platform.
How Grafana Works
Grafana is an open-source multi-platform that gets data from a data source such as Prometheus and transforms it into visualizations charts. Grafana can create interactive charts, graphs, dashboards, and web UIs. You can see all the content on a web page once you connect your data source.
Grafana cannot create these visualization graphs, charts, and dashboards without a data source. It has to connect to a data source such as Prometheus.
Prometheus will send the metrics data it collects from the Kubernetes Cluster, and Grafana uses it to create the visualizations. It allows developers to create alerts, run queries, create visualizations, and understand their metrics no matter the data source.
How to Integrate Prometheus and Grafana on Kubernetes
The are two ways of integrating Prometheus and Grafana on Kubernetes:
1. Manually Deploy all Deployments, Services, ConfigMaps, and Secrets configurations files.
These configuration files are in YAML format. Kubernetes will read these files and deploy the application with all the Kubernetes resources.
This method of manually creating these files is efficient when you are dealing with a simple application. Prometheus and Grafana are complex applications that may take time to configure all the files. This is why most developers prefer the second method of using Helm. It makes their work easier.
2. Using Helm
Helm is popularly known as the package manager for Kubernetes. Helm helps to configure and install any application to Kubernetes. It bundles a Kubernetes application into a package known as a Helm Chart.
Getting Started with Helm and Charts
Helm makes it easy to deploy applications to Kubernetes. It is a collection of YAML files that defines all the Kubernetes resources such as Kubernetes Deployments, Services, ConfigMaps, and Secrets for the Kubernetes application. Helm interacts and communicates with Kubernetes to deploy the Helm Charts to the Kubernetes Cluster.
Helm Charts will help us to define, install, upgrade and integrate even the most complex application on Kubernetes using a few Helm commands. You can easily download and use existing Helm Charts.
We will use Helm to download the Helm Charts for Prometheus and Grafana. We will then deploy the downloaded Helm Charts to Kubernetes. It will save the time of building the entire application from scratch.
The Helm Community creates and maintains the Helm charts for various applications. Helm charts are reusable, easy to install, and up to date. Helm charts are distributed in public and private repositories such as ArtifactHub.
Installing Helm
Helm runs on a Kubernetes Cluster. Before you start installing Helm, ensure you have a Kubernetes cluster that is running. You will install Helm on that Kubernetes Cluster.
In this article, we will install Helm on Minikube. Minikube is a local Kubernetes Cluster that allows the Kubernetes Engine to run our machine using the memory and CPU of the computer. To install Helm on Minikube, follow these steps:
Step One: Tools
You must have the following tool already set up in your machine:
- Docker Engine: This is the virtualization platform that will enable us to run Minikube. Minikube requires Minikube to run locally on our computer.
Step Two: Start Minikube
Minikube comes together with the Docker Engine. After installing Docker, run this command to start the Minikube Kubernetes Cluster.
minikube start --driver=docker
The command will start your cluster as shown below:
Step Three: Installing Helm
Now that our cluster is running, I will show you how to install Helm.
Installing Helm on Windows
On the Windows terminal, run this command:
choco install kubernetes-helm
We will use the Chocolatey package manager for windows applications to install Helm.
Installing Helm on Linux
On the Linux terminal, run this command:
sudo apt-get install helm
We will use the Apt package manager for Ubuntu applications to install Helm.
Installing Helm on macOS
On the macOS terminal, run this command:
brew install helm
We will use the Homebrew package manager for macOS applications to install Helm. I am running Windows. The Helm installation process produces the following output:
Helm Chart Structure
The general structure for a Helm Chart is shown in the image below:
A Helm Chart consists of a Chart folder, Template folder, and chart.yaml file and Values.yaml file.
Chart Folder
It contains the chart dependencies for the downloaded Helm Chart.
Chart.yaml
This file contains the meta-information about the downloaded Helm Chart such as name, description version number, and list of dependencies.
Template Folder
This folder contains all the configuration YAML files you are deploying to the Kubernetes Cluster with the Helm Chart. These files are the Deployments, Services, ConfigMaps, and Secrets configurations YAML files.
Values.yaml
This file is where all the values for the configurations of YAML files are configured and set. The configuration YAML files read their values from this file.
Helm Commands
To check the most common Helm commands, run this command in your terminal:
helm
It displays the following information:
These are the four Helm Commands:
- helm search: We will use this command to search for our Prometheus and Grafana Helm Charts.
- helm pull: It will pull and download the Prometheus and Grafana Helm Charts from ArtifactHub public repository.
- helm install: We will use this command to upload or deploy the Prometheus and Grafana Helm Charts to Minikube Kubernetes Cluster.
- helm list: We will use this command to list the installed Helm Charts in Kubernetes.
Let’s start by searching for Prometheus and Grafana Helm Charts.
Searching for Prometheus Helm Chart
You can search Prometheus Helm from the command line as follows:
helm search hub prometheus
The command displays the following Helm Charts:
You can also search the Prometheus Helm Chart from the ArtifactHub UI as shown below:
We will use the official Prometheus Helm Chart from Prometheus. Let’s add the ‘prometheus-community’ repository for our Prometheus Helm Chart as follows:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
To update the added repository, run this command:
helm repo update
It gives the following output:
Installing the Prometheus Helm Chart
Installing the Prometheus Helm Chart will upload or deploy the Prometheus Helm Chart to Minikube Kubernetes Cluster. To install the Prometheus Helm Chart, use this ‘helm install’ command:
helm install prometheus prometheus-community/prometheus
Installing the Prometheus Helm Chart gives the following output:
When you run the ‘helm install’ command:
- Kubernetes reads the configuration YAML files in the Helm Chart.
- It then sends the Deployments, Services, ConfigMaps, and Secrets configuration YAML files to the Kubernetes API server.
- Kubernetes takes these files and creates the Prometheus application inside the Minikube Kubernetes Cluster.
- It will install a Prometheus Server, an Alert Manager, and monitoring components.
To get all the deployed Kubernetes resources for the Prometheus Kubernetes application, run this ‘kubectl’ command:
kubectl get all
The command will display all the created Kubernetes objects/components in the Kubernetes cluster as shown below:
We have created pods, Kubernetes deployments, and services. Helm simplifies creating all these Kubernetes objects only using a single command.
You can read more about Helm here, on Sweetcode.
Get All the Created Kubernetes Services
To get all the created Kubernetes Services, run this command:
kubectl get service
It displays the following Kubernetes Services:
The Prometheus Helm Chart has created six Kubernetes services. We will access the Prometheus application using the ‘prometheus-server’ Kubernetes service.
The ‘prometheus-server’ service has a ‘ClusterIP’ Kubernetes Service type. It allows you to access the Prometheus application only inside the Minikube Kubernetes Cluster.
Accessing the Prometheus Kubernetes Application
To access the Prometheus Kubernetes application outside the Minikube Kubernetes Cluster, you will run the following command:
kubectl expose service prometheus-server --type=NodePort --target-port=9090 --name=prometheus-server-ext
The command will convert the ‘prometheus-server’ Kubernetes Service from the ‘ClusterIP’ type to the ‘NodePort’ type.
It will allow us to access the Prometheus application outside the Minikube cluster on port ‘9090’.
The command outputs the following:
service/prometheus-server-ext exposed
To check the newly created ‘prometheus-server-ext’ Kubernetes Service, run this command:
kubectl get service
The command outputs the following:
You should be able to access the ‘prometheus-server-ext’ Kubernetes Service outside the Minikube Cluster using an URL.
To generate the URL for accessing this service, run this command:
minikube service prometheus-server-ext
The command outputs the following URL:
To access the Prometheus application, input the URL in your browser as shown below:
From the image above, we have installed the Prometheus application inside the Minikube Kubernetes cluster. This is the Prometheus web UI.
Exploring the Prometheus UI
Let’s explore this Prometheus UI as follows:
- Prometheus Targets
- Prometheus Alerts
- TSDB Status
- Runtime Information
These are some of the components on the Prometheus web UI. The next step is to install Grafana on the Minikube Kubernetes cluster. We will then integrate it to use Prometheus as the data source. Grafana will create dashboards and charts from the data source.
Searching for Grafana Helm Chart
We will search the Grafana Helm Chart from the Artifact Hub as shown below:
We will use the official Grafana Helm from Grafana. Let’s add the `grafana` repository for our Grafana Helm Chart as follows:
helm repo add grafana https://grafana.github.io/helm-charts
To update the added Grafana repository, run this command:
helm repo update
It gives the following output:
Installing the Grafana Helm Chart
Installing the Grafana Helm Chart will upload or deploy the Grafana Helm Chart to Minikube Kubernetes Cluster. To install the Grafana Helm Chart, use this ‘helm install’ command:
helm install grafana grafana/grafana
Installing the Grafana Helm Chart gives the following output:
When you run the ‘helm install’ command:
- Kubernetes reads the configuration YAML files in the Helm Chart.
- It then sends the Deployments, Services, ConfigMaps, and Secrets configuration YAML files to the Kubernetes API server.
- Kubernetes takes these files and creates the Grafana application inside the Minikube Kubernetes Cluster.
Listing the Installed Helm Charts
To list the two installed Helm charts, run this command:
helm list
The command gives the following output:
We have ‘grafana’ and ‘prometheus’ deployed to the Minikube Kubernetes Cluster. To get the Kubernetes objects created after deploying the Grafana Helm Chart, run this command:
kubectl get all
The command gives the following output:
We have created pods, Kubernetes deployments, and services for the Grafana Kubernetes application.
Get All the Created Kubernetes Services for the Grafana Kubernetes Application
To get all the created Kubernetes Services, run this command:
kubectl get service
It displays the following Kubernetes Services:
The Grafana Helm Chart has created a ‘grafana’ Kubernetes Service. We will access the Grafana application using the ‘grafana’ Kubernetes service.
The ‘grafana’ service has a ‘ClusterIP’ Kubernetes Service type. It allows you to access the Grafana application only inside the Minikube Kubernetes Cluster.
Accessing the Grafana Kubernetes Application
To access the Grafana Kubernetes application outside the Minikube Kubernetes Cluster, you will run the following command:
kubectl expose service grafana --type=NodePort --target-port=3000 --name=grafana-ext
The command will also convert the ‘Grafana-server’ Kubernetes Service from the ‘ClusterIP’ type to the ‘NodePort’ type. This will allow us to access the Grafana application outside the Minikube cluster on port ‘3000’.
The command outputs the following:
service/grafana-ext exposed
To check the newly created ‘grafana-ext’ Kubernetes Service, run this command:
kubectl get service
The command outputs the following:
You should be able to access the ‘grafana-ext’ Kubernetes Service outside the Minikube Cluster using an URL. To generate the URL for accessing this service, run this command:
minikube service grafana-ext
The command outputs the following URL:
To access the Grafana application, input the URL in your browser as shown below:
From the image above, we have installed the Grafana Helm Chart application inside the Minikube Kubernetes cluster. The next thing is to login into the Grafana application.
Login Into Grafana
To login into Grafana, we need to get the password for the `admin` user using the following command:
kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
On the Grafana login page, input the username as ‘admin’ and your decoded password to access the Grafana application. After logging in, you will be redirected to the Grafana homepage as shown below:
Let’s integrate Grafana with Prometheus. Grafana will start using Prometheus as the data source.
Integrating Grafana and Prometheus
To integrate Grafana and Prometheus, add Prometheus as the data source. Prometheus is running on http://192.168.49.2:31011. The steps are shown in the images below:
Once you have connected with Prometheus, let’s create a visualization dashboard. We will use the visualization dashboard to monitor the Minikube cluster. It will show the statistics of the cluster usage and running applications in the cluster.
Creating a Visualization Dashboard
To create a visualization dashboard, follow the steps below:
1. Click ‘Create your first Dashboard’
2. Browse to find a Grafana Dashboard
3. Select Prometheus as the Data Source
After selecting Prometheus as the data source, Grafana creates a Dashboard shown in the images below:
From the image above, we have set Prometheus as the data source. Grafana will use the Kubernetes cluster metrics to monitor its performance.
These Grafana dashboards show the ‘Cluster Pod Usage’, ‘Cluster CPU Usage’, ‘Cluster Memory Usage’, and ‘Cluster Disk Usage’. It also shows the status of the ‘Nodes’, ‘Pods’, and ‘Containers’.
It will enable easy management and monitoring of the cluster. We have successfully integrated Prometheus and Grafana on Kubernetes using Helm.
Conclusion
In this article, you have learned how to integrate Prometheus and Grafana on Kubernetes using Helm. We started by discussing how Prometheus and Grafana work. Next, we discussed the three components of Prometheus and the importance of Prometheus monitoring in DevOps. We also covered how to install Helm in your machine and how to create Helm Charts.
Then we installed Prometheus and Grafana Helm Charts on the Minikube Kubernetes Cluster. We used Kubernetes Services to access these two applications. Our two applications were running inside the Minikube Kubernetes Cluster. After logging into Grafana, we integrated Grafana and Prometheus. We then created a visualization dashboard using the Prometheus data source. We used the Grafana dashboard to display Kubernetes cluster usage and monitor its performance.
This article is just the beginning of getting started with Prometheus and Grafana. Feel free to explore more on this topic to gain more exposure and experience. Thank you all for reading this article and all the best in your DevOps journey.
