Homelab Cluster Monitoring with Prometheus and Grafana

/ Homelab, Programming / By

Introduction

Monitoring and observability are crucial aspects of managing any Kubernetes cluster, whether in a production environment or a homelab. Prometheus and Grafana are two of the most popular open-source tools for collecting, storing, and visualizing metrics. In this blog post, we’ll guide you through the process of setting up Prometheus and Grafana on your homelab Kubernetes cluster using Helm, the package manager for Kubernetes.

Why Use Prometheus and Grafana?

  • Prometheus: A powerful monitoring system that collects metrics from your applications and infrastructure, stores them in a time-series database, and enables you to query these metrics using its flexible query language.
  • Grafana: A visualization tool that allows you to create interactive and beautiful dashboards using data from various sources, including Prometheus.

Together, Prometheus and Grafana provide a comprehensive solution for monitoring your Kubernetes cluster, helping you to track performance, detect issues, and gain insights into your system’s behavior.

Prerequisites

Before we begin, ensure you have the following:

  • Kubernetes Cluster: A running Kubernetes cluster. This can be a homelab cluster set up with MicroK8s, K3s, or any other Kubernetes distribution.
  • Helm: Helm installed and configured on your local machine. If Helm isn’t installed, follow the official Helm installation guide.

Step 1: Add the Prometheus and Grafana Helm Repositories

Helm charts for Prometheus and Grafana are available in popular Helm repositories. First, we need to add these repositories to Helm.

  1. Add the Prometheus Helm repository:bashCopy codehelm repo add prometheus-community https://prometheus-community.github.io/helm-charts
  2. Add the Grafana Helm repository:bashCopy codehelm repo add grafana https://grafana.github.io/helm-charts
  3. Update Helm repositories:bashCopy codehelm repo update

Step 2: Install Prometheus using Helm

Next, we’ll deploy Prometheus on your Kubernetes cluster using the Helm chart provided by the Prometheus community.

  1. Create a namespace for monitoring:bashCopy codekubectl create namespace monitoring
  2. Install Prometheus:bashCopy codehelm install prometheus prometheus-community/prometheus -n monitoring
  3. Verify the installation:
    • Check the pods to ensure Prometheus is running:bashCopy codekubectl get pods -n monitoring
    • You should see several Prometheus-related pods in a Running state.

Step 3: Install Grafana using Helm

With Prometheus set up, we’ll now install Grafana, which will be used to visualize the metrics collected by Prometheus.

  1. Install Grafana:bashCopy codehelm install grafana grafana/grafana -n monitoring
  2. Verify the installation:
    • Check the pods to ensure Grafana is running:bashCopy codekubectl get pods -n monitoring
    • Look for a Grafana pod in the Running state.
  3. Accessing Grafana:
    • By default, Grafana is accessible within the cluster. To access it externally, you can use port forwarding:bashCopy codekubectl port-forward svc/grafana 3000:80 -n monitoring
    • Open your browser and go to http://localhost:3000. The default login credentials are admin for both username and password.

Step 4: Configure Grafana to Use Prometheus as a Data Source

Once Grafana is up and running, you need to configure it to use Prometheus as a data source.

  1. Log in to Grafana:
    • Navigate to http://localhost:3000 and log in with the default credentials.
  2. Add Prometheus as a Data Source:
    • Click on the gear icon (⚙️) in the left sidebar and select “Data Sources.”
    • Click “Add data source” and select “Prometheus” from the list.
    • In the HTTP URL field, enter the following if using default settings:arduinoCopy codehttp://prometheus-server.monitoring.svc.cluster.local:80
    • Click “Save & Test” to verify the connection.
  3. Import a Dashboard:
    • Grafana provides several pre-built dashboards for Kubernetes monitoring. To import one:
      • Click on the “+” icon in the left sidebar and select “Import.”
      • Enter the dashboard ID from Grafana’s dashboard repository (e.g., 315 for a popular Kubernetes cluster monitoring dashboard).
      • Configure the Prometheus data source to be used by the dashboard and click “Import.”

Step 5: Explore and Visualize Metrics

Now that Prometheus and Grafana are set up and connected, you can start exploring the data and creating custom dashboards.

  1. Explore Prometheus Metrics:
    • Use the “Explore” feature in Grafana to query Prometheus metrics directly and visualize them on-the-fly.
  2. Create Custom Dashboards:
    • You can build custom dashboards tailored to your specific needs. For example, you might create a dashboard that monitors CPU usage, memory usage, pod status, and network traffic across your Kubernetes cluster.
  3. Set Up Alerts:
    • Grafana allows you to set up alerts based on specific conditions. For instance, you can configure an alert to notify you if CPU usage exceeds a certain threshold, ensuring that you’re aware of potential issues in real-time.

Conclusion

Setting up Prometheus and Grafana on your homelab Kubernetes cluster provides you with a powerful monitoring and visualization solution. With Helm, the deployment process is straightforward and flexible, allowing you to scale and customize your monitoring setup as needed. Whether you’re monitoring a simple application or a complex multi-node cluster, Prometheus and Grafana give you the insights and tools you need to ensure your systems run smoothly.

Have you set up Prometheus and Grafana on your Kubernetes cluster? What challenges did you face, and how did you overcome them? Share your experiences in the comments below!