Monitoring Kubernetes cluster using Prometheus and Grafana

Prometheus is really trending. It is the second largest project in CNCF. It’s much familiar because of its wide use cases and the simplicity in operations. We have one sub category for the Prometheus, if you guys are interested please have a look here: https://www.crybit.com/category/devops/prom/

Prometheus has a lot of advantages over other monitoring platforms. The main and important advantages of Prometheus are:

  • Prometheus is TSDB
  • Prometheus is Pull based tool
  • Centralised control
  • In built Alerting facility
  • Easy for monitoring teams
  • Data visualisation
  • Service discovery (sd)
  • Scalability
  • PromQL

You can see a detailed explanations about all these points here Advantages of Prometheus

Kubernetes is one of the very matured cluster orchestration tools available now. Almost all big companies now moved their containerised workloads to Kubernetes (different flavours of Kubernetes) because of its ease in use and scalability. Monitoring the Kubernetes cluster is also important and this can be easily done by the help of Prometheus.

In current Kubernetes world everything can be deployed by the use of operators. Most of the advanced and well matured Kuberenetes providers (like OpenShift) includes Prometheus operators by default as part of cluster operators or it’s away from a single click. In this article, I will explain how we can simply deploy a Prometheus setup along with Grafana for monitoring your cluster resources and Kubernetes workloads.

As we all know, monitoring is crucial. Lets go with Prometheus.

Prerequisites

  • A running Kuberenetes cluster
  • kubectl
  • helm 3

Why Helm?

To deploy Prometheus easily. Helm is the standard package manager for Kubernetes workloads. It’s easy to deploy, maintain and manage the deployments using helm charts. Read more about helm from the official site: https://helm.sh/docs/topics/charts/

Deploy Prometheus and Grafana on Kubernetes

You can test this setup on your minicube setup as well. Follow the below steps to deploy Prometheus along with Grafana for your Kubernetes cluster.

Prometheus

Create a namespace for Prometheus

It’s a best practice to deploy Prometheus or your monitoring components to a different namespace. You can allow access to your production or staging namespaces by the help of Networking Policies later, if you face issues or in case you need to add more restrictions. It also help the cluster admin to limit resources for your prometheus components by applying limits in namespace level as well.

$ kubectl create namespace prometheus

You can check the namespace by using get call.

$ kubectl get namespaces | grep prometheus
prometheus Active 35m

Now we are good to deploy prometheus and it’s dependent components using helm. Hope you are familiar with helm and its use cases.

Make sure you have prometheus repo added in your helm repo

To see currently available repos you can use helm repo list from your CLI.

$ helm repo list
NAME URL
stable https://charts.helm.sh/stable
dandydev https://dandydeveloper.github.io/charts
bitnami https://charts.bitnami.com/bitnami

Add Prometheus community helm repo

Future Note: This article is published on July 2021. Please make sure any changes in repo/chart and add the repo accordingly.

The stable/prometheus-operator chart is deprecated and they moved it to https://github.com/prometheus-community/helm-charts

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts/

If you see SSL error, please make sure that your client (kubectl) configurations are properly configured. Otherwise pass –insecure-skip-tls-verify parameter

Error: looks like "https://prometheus-community.github.io/helm-charts/" is not a valid chart repository or cannot be reached: Get "https://prometheus-community.github.io/helm-charts/index.yaml": x509: certificate signed by unknown authority

Fix

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts/ --insecure-skip-tls-verify

Now you have added the Prometheus community repo to your helm.

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts/ --insecure-skip-tls-verify
"prometheus-community" has been added to your repositories
$ helm repo list
NAME URL
stable https://charts.helm.sh/stable
dandydev https://dandydeveloper.github.io/charts
bitnami https://charts.bitnami.com/bitnami
prometheus-community https://prometheus-community.github.io/helm-charts/

Now we are good to install / deploy Prometheus components to your Kubernetes cluster.

Install Prometheus using helm

$ helm repo update

Hang tight while we grab the latest from your chart repositories…
…Successfully got an update from the "dandydev" chart repository
…Successfully got an update from the "prometheus-community" chart repository
…Successfully got an update from the "stable" chart repository
…Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈

Install by running

$ helm install prometheus prometheus-community/kube-prometheus-stack --namespace prometheus

That’s it. You can see all pods by running the get call.

$ kubectl --namespace prometheus get pods

Sample output

$ kctl --namespace prometheus get pods -l "release=prometheus"

 NAME                                                   READY   STATUS    RESTARTS   AGE
 prometheus-kube-prometheus-operator-59c5dcf5bc-g4ctl   1/1     Running   0          34s
 prometheus-prometheus-node-exporter-4bvmw              1/1     Running   0          34s
 prometheus-prometheus-node-exporter-ddw5g              1/1     Running   0          34s
 prometheus-prometheus-node-exporter-rgn5q              1/1     Running   0          34s

$ kubectl --namespace prometheus get pods
 NAME                                                     READY   STATUS    RESTARTS   AGE
 alertmanager-prometheus-kube-prometheus-alertmanager-0   2/2     Running   0          46s
 prometheus-grafana-6576787696-qrkqq                      2/2     Running   0          71s
 prometheus-kube-prometheus-operator-59c5dcf5bc-g4ctl     1/1     Running   0          71s
 prometheus-kube-state-metrics-fcd578d77-bhxs5            1/1     Running   0          71s
 prometheus-prometheus-kube-prometheus-prometheus-0       2/2     Running   1          46s
 prometheus-prometheus-node-exporter-4bvmw                1/1     Running   0          71s
 prometheus-prometheus-node-exporter-ddw5g                1/1     Running   0          71s
 prometheus-prometheus-node-exporter-rgn5q                1/1     Running   0          71s

To see all available charts

$ helm search repo prometheus-community

Now you can port forward your Prometheus service or direct pod by using the Kubernetes port forwarding technique and test your prometheus installation. Read more about port forwarding from official documentation.

$ kubectl port-forward svc/prometheus-kube-prometheus-prometheus 9090 -n prometheus
Forwarding from 127.0.0.1:44829 -> 9090
Forwarding from [::1]:44829 -> 9090

It should be accessible from your local browser after connecting to your cluster using SSH port forwarding. Example command is added below:

$ ssh -t -L 9090:localhost:9090 [email protected] -i "kub_key.pem"

Now you can access prometheus using localhost:44829/graph It should looks like

Now time to explore Prometheus. Refer: https://prometheus.io/

Grafana

Similar way you can access Grafana as well. Identify svc or pod for Grafana and expose & access it. As the Prometheus installation is managed by the operator, it install Grafana as well.

$ kubectl port-forward svc/prometheus-grafana 80 -n prometheus
Forwarding from 127.0.0.1:44829 -> 3000
Forwarding from [::1]:44829 -> 3000

Now access it http://localhost:3000/login after forwarding local port.

Where is Grafana credentials?

This can be found from the secrets.

$ kubectl get secrets prometheus-grafana -n prometheus
NAME TYPE DATA AGE
prometheus-grafana Opaque 3 57m
$ kubectl get secrets prometheus-grafana -n prometheus -o yaml
apiVersion: v1
data:
admin-password: cHJvbS1vcGVyYXRvcg==
admin-user: YWRtaW4=
ldap-toml: ""
kind: Secret

Secret data encoded with base64 you can decode it by the following command:

  • For getting username
$ openssl base64 -d
YWRtaW4=
admin
  • For getting password
$ openssl base64 -d
cHJvbS1vcGVyYXRvcg==
prom-operator

Now you are in to Grafana

Explore Grafana. It comes with a lot of default dashboards which helps to monitor your Kubernetes cluster.

Will explain the details about each dashboards and the way to troubleshoot your cluster in an efficient way will include in upcoming blog posts. Stay tuned.

We just exposed the Prometheus and Grafana using port forwarding. In case of real production cluster you can create ingress for accessing the endpoints. This also will

Please let me know if you have any questions or suggestions. I will try to reach out to you.

Thanks!!

Post navigation

Arunlal Ashok

DevOps Engineer II at Zeta Suite. Linux lover. Traveller. Father of two princesses.

Always happy for an open discussion! Write to arun ((@)) crybit ((dot)) com.
LinkedIn: https://www.linkedin.com/in/arunlal-a-18037773/

Leave a Reply

Your email address will not be published. Required fields are marked *