Prometheus Basics Training

1.1 Tasks: Setup

Mon, 01 Jan 0001 00:00:00 +0000

Executing oc commands

Note

As you will be executing some oc commands in the following labs, make sure you have the tool installed and are logged in to your OpenShift Cluster.

You can copy the login Command from the OpenShift UI:

Browse to http://LOCALHOST_OPENSHIFT
Click on your name in the top right
Copy login command
Replace 6443 with 443

Task 1.1.1: Identify your monitoring repository

Before we get started, take the time to familiarize yourself with the your config repository. In the training you will not be working with the config repository of your team to prevent the training resources from getting in the way of the day to day business. You can find more information on how to deploy the Baloise monitoring stack for your team at Deploying the Baloise Monitoring Stack .

1.1 Tasks: Setup

Mon, 01 Jan 0001 00:00:00 +0000

Task 1.1.1: Getting Started - Web IDE

The first thing we’re going to do is to explore our lab environment and get in touch with the different components.

The namespace with the name corresponding to your username is going to be used for all the hands-on labs. And you will be using the folling tools during the lab

Gitea Git Server: https://gitea.training.cluster.acend.ch - Login with <user> and the provided password
Argo CD Server: https://argocd.training.cluster.acend.ch - Login via Gitea
git - Login with <user> and the provided password
kubectl - already logged in

Note

The URL and Credentials to the Web IDE will provided by the teacher. Use Chrome for the best experience.

Once you’re successfully logged into the web IDE open a new Terminal by hitting CTRL + SHIFT + ¨ or clicking the Menu button –> Terminal –> new Terminal and check the installed kubectlversion by executing the following command:

2.1 Tasks: File-Based Service Discovery

Mon, 01 Jan 0001 00:00:00 +0000

In this lab you are going to configure Prometheus to scrape OpenShift-external targets by using file-based service discovery.

Task 2.1.1: Create static targets

We are going to use the file-based service discovery mechanism that has been deployed on OpenShift (documented in Confluence ). As file input you will create a ConfigMap defining the static targets.

In the monitoring folder within your repository, create a YAML file training_target.yaml defining a ConfigMap and add the file to your repository. Use the following example:

2.1 Tasks: Thanos Querier

Mon, 01 Jan 0001 00:00:00 +0000

To have centralized access to the metrics of both the shared infrastructure Prometheus instance as well as your own userwokload Prometheus instance, we will install the Thanos Querier in this lab. This will give us a central point of view for the following prometheus instances:

Infrastructure Prometheus (K8S Metrics, Node Metrics, …) in the global monitoring namespace
Userworkload Prometheus (the stack we can use for our application metrics) in the <user>-monitoring namespace

Task 2.1.1: Install Thanos Querier

To install the Thanos Querier, change query.enabled to true in the values.yaml of your user-monitoring Helm release. ArgoCD will automagically install all needed components in your namespace.

2.2 Tasks: Explore metrics

Mon, 01 Jan 0001 00:00:00 +0000

In this lab you are going to explore various metrics, which your Prometheus server is collecting.

Task 2.2.1: Prometheus/Thanos web UI

As we have now attached the Thanos Querier to your own Prometheus as well as the infrastructure Prometheus, we will use its UI to have a global view.

Get a feel for how to use the Thanos Querier web UI. Open the web UI and navigate to the Graph menu (right on top in the grey navigation bar next to Stores).

2.3 Tasks: PromQL

Mon, 01 Jan 0001 00:00:00 +0000

In this lab you are going to learn a bit more about PromQL (Prometheus Query Language) .

PromQL is the query language that allows you to select, aggregate and filter the time series data collected by prometheus in real time.

Note

Execute the queries in the Thanos Querier UI .

PromQL can seem overwhelming. It may take a little time to get used to it. There may be different approaches to solve the tasks. Our solution is just one possibility.

3.1 Tasks: Grafana intro

Mon, 01 Jan 0001 00:00:00 +0000

Task 3.1.1 Have a look at the default team dashboard

The Monitoring Stack provides a dashboard that shows you common aggregated metrics of your applications. You should see the dashboard as soon as you log in.

Have a look at the provided dashboard .

Select either the cluster caast01 or caasp01 to get metrics from applications running on these clusters

Task 3.1.2 Have a look at the default team namespace dashboard

The Monitoring Stack provides a dashboard that shows you metrics of your applications.

3.1 Tasks: Grafana intro

Mon, 01 Jan 0001 00:00:00 +0000

Task 3.1.1: Install Grafana

Similar to the basic setup, we are just going to update our configuration of the ArgoCD application to install and create our Grafana instance. Update your monitoring application (charts/user-monitoring/values.yaml) and update the grafana.enabled flag to true:

charts/user-monitoring/values.yaml:

user: <user> # Replace me
# prometheus
prometheus:
 enabled: true
# thanos-query
query:
 enabled: true
# grafana
grafana:
 enabled: true
# blackboxexporter
blackboxexporter:
 enabled: false
# pushgateway
pushgateway:
 enabled: false
# alertmanager
alertmanager:
 enabled: false
# thanos-ruler
ruler:
 enabled: false

Verify the installation and sync process in the ArgoCD UI . When the application is synchronized successfully navigate to your freshly created Grafana instance: https://<user>-grafana.training.cluster.acend.ch . Use the admin account to login in. Skip the password change.

3.2 Tasks: Grafana dashboards

Mon, 01 Jan 0001 00:00:00 +0000

Task 3.2.1: Import a dashboard

As a first Task we import an existing dashboard to our grafana. There is a huge collection of predefined dashboards available on https://grafana.com/grafana/dashboards .

In this task we learn how to import existing dashboards, which we can also use as reference to write our own queries and dashboards.

Choose one or more of the preexisting dashboards from https://grafana.com/grafana/dashboards and import them into your grafana. Use the daterange, interval picker on the top right to change the timerange (between 5 minutes and 10 minutes) of the displayed mertics.

4.1 Tasks: Blackbox exporter

Mon, 01 Jan 0001 00:00:00 +0000

Task 4.1.1: Add a blackbox target

We will use the blackbox exporter to create a new probe which accepts a 2xx return code as a valid http return code. This will return the probe_success metric from the blackbox exporter with the value 1, if the http status code is 2xx.

Task description:

Create a probe in the monitoring directory which uses the HTTP prober and expects a 2xx return code as a valid status code
Define https://bitbucket.balgroupit.com/status as a single static target, which the blackbox should probe
Use the following documentation as reference 06 - HTTP and TCP endpoint monitoring

Hints

To configure the blackbox exporter you have to add the following file training_blackbox_target.yaml to your monitoring directory, commit and push the changes:

4.1 Tasks: Blackbox exporter

Mon, 01 Jan 0001 00:00:00 +0000

Task 4.1.1: Install Blackbox exporter

Similar to the basic setup, we are just going to update our configuration of the ArgoCD application to install the Blackbox exporter. Update your monitoring application (charts/user-monitoring/values.yaml) and update the blackboxexporter.enabled flag to true:

charts/user-monitoring/values.yaml:

user: <user> # Replace me
# prometheus
prometheus:
 enabled: true
# thanos-query
query:
 enabled: true
# grafana
grafana:
 enabled: true
 datasources:
 - name: prometheus
 access: proxy
 editable: false
 type: prometheus
 url: http://prometheus-operated:9090
# blackboxexporter
blackboxexporter:
 enabled: true
# pushgateway
pushgateway:
 enabled: false
# alertmanager
alertmanager:
 enabled: false
# thanos-ruler
ruler:
 enabled: false

Commit and push the changes.

4.2 Tasks: Pushgateway

Mon, 01 Jan 0001 00:00:00 +0000

Task 4.2.1 - Install and configure Pushgateway

Update your monitoring application (charts/user-monitoring/values.yaml) and update the blackboxexporter.enabled flag to true:

charts/user-monitoring/values.yaml:

user: <user> # Replace me
# prometheus
prometheus:
 enabled: true
# thanos-query
query:
 enabled: true
# grafana
grafana:
 enabled: true
 datasources:
 - name: prometheus
 access: proxy
 editable: false
 type: prometheus
 url: http://prometheus-operated:9090
# blackboxexporter
blackboxexporter:
 enabled: true
# pushgateway
pushgateway:
 enabled: true
# alertmanager
alertmanager:
 enabled: false
# thanos-ruler
ruler:
 enabled: false

Commit and push the changes and verify whether the deployment worked correctly.

6.1 Tasks: Instrumenting

Mon, 01 Jan 0001 00:00:00 +0000

Task 6.1.1: Get the source code

git clone https://github.com/acend/prometheus-training-go-instrumentation.git

Change into the freshly cloned git repository

cd prometheus-training-go-instrumentation

TODO: create Dockerfile This application creates a simple webserver which listens at http://localhost:8083 and returns the string Prometheus Training after a random interval between 0 and 1000 milliseconds. You can test the app by building it as a docker image using docker build -t prometheus-training-go-instrumentation:local . and then running the built image using docker run --rm -p 8083:8083 prometheus-training-go-instrumentation:local. You can then (in a separate terminal) issue a request using the command curl http://localhost:8083.

6.1 Tasks: Instrumenting

Mon, 01 Jan 0001 00:00:00 +0000

Task 6.1.1: Spring Boot Example Instrumentation

Using the micrometer metrics facade in Spring Boot Applications lets us collect all sort of metrics within a Spring Boot application. Those metrics can be exported for Prometheus to scrape by a few additional dependencies and configuration.

Let’s have a deeper look at how the instrumentation of a Spring Boot application works. For that we can use the prometheus-training-spring-boot-example application located at https://github.com/acend/prometheus-training-spring-boot-example . To make the application collect metrics and provide a Prometheus endpoint we now need to simply add the following two dependencies in the pom.xml file, where it says :

6.1 Tasks: Instrumenting

Mon, 01 Jan 0001 00:00:00 +0000

Task 6.1.1: Spring Boot Example Instrumentation

7.1 Tasks: Enable and configure Alertmanager

Mon, 01 Jan 0001 00:00:00 +0000

Task 7.1.1: Install Alertmanager and Thanosruler

Update your monitoring application (charts/user-monitoring/values.yaml) and update the alertmanager.enabled and ruler.enabled flag to true:

charts/user-monitoring/values.yaml:

user: <user> # Replace me
# prometheus
prometheus:
 enabled: true
# thanos-query
query:
 enabled: true
# grafana
grafana:
 enabled: true
 datasources:
 - name: prometheus
 access: proxy
 editable: false
 type: prometheus
 url: http://prometheus-operated:9090
# blackboxexporter
blackboxexporter:
 enabled: true
# pushgateway
pushgateway:
 enabled: true
# alertmanager
alertmanager:
 enabled: true
# thanos-ruler
ruler:
 enabled: true

Commit and push the changes.

Verify the installation and sync process in the ArgoCD UI . Or execute the following command:

7.1 Tasks: Setup custom alerting rules

Mon, 01 Jan 0001 00:00:00 +0000

Task 7.1.1: Add alerting rules

Note

Alertmanager will automatically send mails to the defined responsible email address in the teams root configuration when you set the label severity=critical in your PrometheusRule. To change this behaviour and/or add Alerting to MS Teams, check the documentation 03 - Setup custom alerting rules in Confluence.

The Prometheus Operator allows you to configure Alerting Rules (PrometheusRules). This enables OpenShift users to configure and maintain alerting rules for their projects. Furthermore it is possible to treat Alerting Rules like any other Kubernetes resource and lets you manage them in Helm or Kustomize. A PrometheusRule has the following form:

7.2 Tasks: Alertmanager

Mon, 01 Jan 0001 00:00:00 +0000

Task 7.2.1: Configure Slack as alert receiver

As part of our lab setup we already configured an alert receiver for Alertmanager alerts.

We deployed the Mail catcher , which is a very simple component. We can send Emails to the server, and those are then displayed in the Web UI. The Emails are not send anywhere, and this setup should only be used for demo purposes.

When we enabled the alertmanager configuration in the charts/user-monitoring/values.yaml we also deployed the following AlertmanagerConfig Custom Resource

7.3 Tasks: Alertrules and alerts

Mon, 01 Jan 0001 00:00:00 +0000

Note

For doing the alerting lab it’s useful to have a “real” application so that alerts can be provoked. You will use the demo app installed in your monitoring-demo namespace for this purpose.

The example app exposes metrics which are already scraped from lab one.

{job="example-web-python"}

The Prometheus Operator allows you to configure Alerting Rules (PrometheusRules). This enables Kubernetes users to configure and maintain alerting rules for their projects. Furthermore it is possible to treat Alerting Rules like any other Kubernetes resource and lets you manage them in Helm or Kustomize. A PrometheusRule has the following form:

4.3 Tasks: Exporter as a sidecar

Mon, 01 Jan 0001 00:00:00 +0000

Task 4.3.1: Deploy a database and use a sidecar container to expose metric

Task description:

As we’ve learned in Lab 4 - Prometheus exporters when applications do not expose metrics in the Prometheus format, there are a lot of exporters available to convert metrics into the correct format. In Kubernetes this is often done by deploying so called sidecar containers along with the actual application.

Use the following command to deploy a MariaDB database your monitoring or application namespace on CAAST.

4.3 Tasks: Exporter as a sidecar

Mon, 01 Jan 0001 00:00:00 +0000

Task 4.3.1: Deploy a database and use a sidecar container to expose metric

This lab demonstrates that with a mariadb database. In the first step, we are going to install a plain mariadb with a Secret (username password to access the database), a Service and the Deployment .

5.1 Tasks: kube-prometheus metrics

Mon, 01 Jan 0001 00:00:00 +0000

Task 5.1.1: Memory usage of Prometheus

Task description:

Display the memory usage of both Prometheus pods
Use a filter to just display metrics from your prometheus containers

Use the [Thanos Querier web UI](https://<user>-thanos-query.training.cluster.acend.ch to execute the queries.

Note

Search for a metric with memory_working_set in its name

Hints

container_memory_working_set_bytes{pod=~"prometheus-prometheus-0", container="prometheus",namespace=~"<user>.*"}

Task 5.1.2: Kubernetes pod count

Task description:

Display how many pods are currently running on your Kubernetes platform

Hints

There are different ways to archive this. You can for example query all running containers and group them by pod and namespace.

5.2 Tasks: Prometheus Operator

Mon, 01 Jan 0001 00:00:00 +0000

5.2.1: Prometheus storage

By default, the Prometheus operator stack does not persist the data of the deployed monitoring stack. Therefore, any pod restart would result in a reset of all data. Let’s learn about persistence for Prometheus.

Task description:

See this example of how to configure storage for Prometheus

To get the current values, either look at the file in git or use kubectl -n $USER-monitoring get prometheus prometheus -oyaml.

Hints

To enable storage for our Prometheus we would simply add the following section to our Prometheus definition.

Footer

Mon, 01 Jan 0001 00:00:00 +0000

Header

Mon, 01 Jan 0001 00:00:00 +0000

- acend gmbh

8.1 Tasks: Relabeling

Mon, 01 Jan 0001 00:00:00 +0000

Relabeling in Prometheus can be used to perform numerous tasks using regular expressions, such as

adding, modifying or removing labels to/from metrics or alerts,
filtering metrics based on labels, or
enabling horizontal scaling of Prometheus by using hashmod relabeling.

It is a very powerful part of the Prometheus configuration, but it can also get quite complex and confusing. Thus, we will only take a look at some basic/simple examples.

There are four types of relabelings:

8.2 Tasks: Recording Rules

Mon, 01 Jan 0001 00:00:00 +0000

Prometheus recording rules allow you to precompute queries at a defined interval (global.evaluation_interval or interval in rule_group) and save them to a new set of time series.

In this lab you are going to create your first own recording rules. Recording rules are very useful when it comes to queries, which are very complex and take a long time to compute. The naming convention dictates to use the following format when naming recording rules level:metric:operation. Additional information regarding naming best-practices can be found here .

8.3 Tasks: Troubleshoot Kubernetes Service Discovery

Mon, 01 Jan 0001 00:00:00 +0000

Task 8.3.1: Troubleshooting Kubernetes Service Discovery

We will now deploy an application with an error in the monitoring configration.

Deploy Loki in the monitoring namespace.

Create a deployment training_loki-deployment.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
 labels:
 app: loki
 name: example-loki
spec:
 replicas: 1
 selector:
 matchLabels:
 app: loki
 template:
 metadata:
 labels:
 app: loki
 spec:
 containers:
 - image: quay.balgroupit.com/acend/loki
 imagePullPolicy: Always
 name: loki
 volumeMounts:
 - mountPath: /loki/chunks
 name: chunks
 - mountPath: /loki/boltdb-shipper-cache
 name: boltdb-shipper-cache
 - mountPath: /loki/boltdb-shipper-active
 name: shipper-active
 - mountPath: /loki/wal
 name: wal
 - mountPath: /loki/compactor
 name: compactor
 restartPolicy: Always
 volumes:
 - name: chunks
 emptyDir: {}
 - name: boltdb-shipper-cache
 emptyDir: {}
 - name: shipper-active
 emptyDir: {}
 - name: wal
 emptyDir: {}
 - name: compactor
 emptyDir: {}

Create a Service training_service-loki.yaml.

8.3 Tasks: Troubleshoot Kubernetes Service Discovery

Mon, 01 Jan 0001 00:00:00 +0000

Task 8.3.1: Troubleshooting Kubernetes Service Discovery

We will now deploy an application with an error in the monitoring configration.

Deploy Loki in your namespace by adding the following files to your git repo in the user-demo/ folder:

---
apiVersion: apps/v1
kind: Deployment
metadata:
 labels:
 app: loki
 name: loki
spec:
 replicas: 1
 selector:
 matchLabels:
 app: loki
 template:
 metadata:
 labels:
 app: loki
 spec:
 containers:
 - image: mirror.gcr.io/grafana/loki:latest
 name: loki

---
apiVersion: v1
kind: Service
metadata:
 labels:
 app: loki
 name: loki
spec:
 ports:
 - name: http
 port: 3100
 protocol: TCP
 targetPort: 3100
 selector:
 app: loki
 type: NodePort

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
 labels:
 app.kubernetes.io/name: loki
 name: loki
spec:
 endpoints:
 - interval: 30s
 port: http
 scheme: http
 path: /metrics
 selector:
 matchLabels:
 prometheus-monitoring: 'true'

Add, commit and push changes to your git repository and let ArgoCD sync your app again.

8.4 Tasks: Cleanup your monitoring workspace

Mon, 01 Jan 0001 00:00:00 +0000

Task 8.4.1: Cleanup your monitoring workspace

Make sure to remove all files with the training_ prefix in your monitoring directory.

About

Mon, 01 Jan 0001 00:00:00 +0000

About Prometheus Basics Training