Scribd
Upload
39 views

Assignment 3

The document provides instructions for setting up monitoring and alerting capabilities in Prometheus including installing Node Exporter and configuring alerts based on its metrics, performing queries, advanced Grafana dashboarding with variables and transformations, configuring recording and multi-level alerting rules, blackbox monitoring with the Blackbox Exporter, and backup and restore of Prometheus data.

Uploaded by

azatabdirashituly
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
39 views

Assignment 3

The document provides instructions for setting up monitoring and alerting capabilities in Prometheus including installing Node Exporter and configuring alerts based on its metrics, performing queries, advanced Grafana dashboarding with variables and transformations, configuring recording and multi-level alerting rules, blackbox monitoring with the Blackbox Exporter, and backup and restore of Prometheus data.

Uploaded by

azatabdirashituly
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 13

Practical work 3 - Monitoring and Alerting Setup

Task 1 – 4 are done in previous assignments.


Task 5: Expand Monitoring Capabilities
1. Install and configure Node Exporter.
a) Download Node Exporter from: https://prometheus.io/download/
b) Configure Prometheus to scrape metrics from Node Exporter.
- job_name: "node-exporter"
static_configs:
- targets: ["host.docker.internal:9100"]
prometheus.yml file.
node-exporter:
container_name: node-exporter
image: prom/node-exporter
network_mode: bridge
depends_on:
- prometheus
ports:
- 9100:9100
restart: unless-stopped
docker-compose.yml file.
c) Import a Grafana dashboard for Node Exporter metrics.
2. Set up an alert based on Node Exporter metrics.
a) Go to alerting section and choose metric, for example, high CPU usage.

b) Create an alert based on chosen metric.


Task 6: Querying and Exploration in Prometheus
Execution of basic and complex queries:

up: Shows the uptime of all instances that Prometheus is scraping.

node_cpu_seconds_total: Provides the total CPU time in seconds.


rate(node_cpu_seconds_total[5m]): Computes the per-second average rate of time series in the
last five minutes.

node_cpu_seconds_total{job="node-exporter", instance="localhost:9100"}: Narrows down the


metric to the node-exporter job from the host.docker.internal:9100 instance.

sum(rate(node_cpu_seconds_total[5m])) by (job): Aggregates the CPU rate for each job


separately.
Task 7: Advanced Grafana Dashboarding

Adding a Variable:

{job=~".+"}: Fetches all jobs.

Adding a Panel with Multiple Metrics:


Adding Transformation:

Applying overrides:
Task 8: Advances Alerting and Recording Rules

Objective 1: Set up recording rules in Prometheus.

Open prometheus.yml file and paste the following:


rule_files:
- "network_traffic.yml"

After, create a new file specified in rule_files network_traffic.yml and paste the following:
groups:
- name: recording_rules
rules:
- record: incoming_network_traffics
expr: rate(node_network_receive_bytes_total{job="node-exporter"}[5m])

Finally, restart the prometheus to apply a new configuration.


Objective 2: Configure multi-stage alerting.
Creating a multi-condition alert in Grafana:
Configuring the alert to have multiple levels:

Severity = {{if gt $values.B.Value 0.7 }}critical{{else}}warning{{end}}


Explanation of provided snipped: It is a conditional logic to determine whether an alert should be
considered "critical" or "warning" based on a metrics value.
Task 9: Blackbox Monitoring with Prometheus
Objective 1: Install and configure the Blackbox Exporter.
Download Blackbox Exporter from: https://prometheus.io/download/
Open docker-compose.yml file and paste the following configuration:
blackbox:
container_name: blackbox_exporter
image: prom/blackbox-exporter
volumes:
- ./blackbox/:/etc/blackbox_exporter/blackbox.yml
ports:
- 9115:9115
restart: unless-stopped

After, in prometheus.yml file specify the following:


- job_name: "blackbox"
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
- https://www.youtube.com/
- host.docker.internal:9115
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: host.docker.internal:9115

Then run the docker.


Creating a Panel for Visualization
Task 10: Backup and Restore
Objective 1: Backup your Prometheus data.
Stop the Prometheus service and Create a backup of the Prometheus data directory

Run the Prometheus.


Objective 2: Restore Prometheus from a backup.
Stop the Prometheus.

Restore the data directory from the backup.

Make sure that the data is not corrupted after starting Prometheus.

You might also like