Lesson Description - Query Operators

PromQL includes a wide variety of operators you can use to manipulate data in your
queries. In this lesson, we provide an overview of what the various types of operators
can do. We also demonstrate how to use operators by building some queries.

Relevant Documentation

• Operators

Lesson Reference

Access the Prometheus expression browser for your Prometheus server in a web
browser. Be sure to use the public IP address of your Prometheus server:

http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090

Use an arithmetic operator:

node_cpu_seconds_total * 2

Experiment with combining datasets using an arithmetic operator:

node_cpu_seconds_total{mode="system"} + ignoring(mode)
node_cpu_seconds_total{mode="user"}

node_cpu_seconds_total{mode="system"} + on(cpu)
node_cpu_seconds_total{mode="user"}

Use a comparison operator to filter results:

node_cpu_seconds_total == 0

Use the bool keyword to display the results of a comparison:

 node_cpu_seconds_total == bool 0

Experiment with combining datasets using logical set operators:

node_cpu_seconds_total

node_cpu_guest_seconds_total

node_cpu_seconds_total and node_cpu_guest_seconds_total

node_cpu_seconds_total or node_cpu_guest_seconds_total

Use an aggregation operator:

node_cpu_seconds_total{mode="idle"}

avg(node_cpu_seconds_total{mode="idle"})

sum(node_cpu_seconds_total{mode="idle"})
Lesson Description - Configuring
Prometheus
Prometheus has a wide variety of configuration options that can allow you to
customize its behavior to meet your needs. While there are too many options to
cover all of them in detail, it is useful to be aware of how you can go about
configuring Prometheus. In this lesson, we discuss the Prometheus configuration file
and where to find detailed information on configuration options in the Prometheus
documentation. We also demonstrate the process of making a configuration change
to a Prometheus server.

Relevant Documentation

• Prometheus Configuration
• Example Prometheus Config File

Lesson Reference

Log in to your Prometheus server.

Edit the Prometheus configuration file:

sudo vi /etc/prometheus/prometheus.yml

Locate the global.scrape_interval setting and change it to 10s:

global:
scrape_interval: 10s

Reload the Prometheus configuration:

sudo killall -HUP prometheus

Another way to reload the configuration is to simply restart Prometheus (you do not
need to do this if you used the killall -HUP method):
 sudo systemctl restart prometheus

Query the Prometheus API to verify your changes took effect:

curl localhost:9090/api/v1/status/config

You should see global:\n scrape_interval: 10s in the output.

Lesson Description - Configuring an
Exporter
In order to fully utilize Prometheus, you will need to configure exporters. Exporters
are sources of metric data that Prometheus periodically collects. In this lesson, we
set up monitoring for a Linux server. We will install Node Exporter on the server and
configure Prometheus to scrape metrics from that exporter. This will enable us to
query Prometheus for the new Linux server's metric data.

Relevant Documentation

• Scrape Config
• Monitoring a Linux Host

Lesson Reference

Configure a New Server to Be Monitored

Create a new Cloud Playground server:

• Distribution: Ubuntu 18.04 Bionic Beaver LTS

• Size: Small
• Tag: Linux Server

Log in to the new server. We will configure this new server for Prometheus
monitoring using Node Exporter.

Create a user for Node Exporter:

sudo useradd -M -r -s /bin/false node_exporter

Download and extract the Node Exporter binary:

wget https://github.com/prometheus/node_exporter/releases/download/
v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz

tar xvfz node_exporter-0.18.1.linux-amd64.tar.gz

Copy the Node Exporter binary to the appropriate location and set ownership:

sudo cp node_exporter-0.18.1.linux-amd64/node_exporter /usr/local/

bin/

sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

Create a systemd unit file for Node Exporter:

sudo vi /etc/systemd/system/node_exporter.service

Define the Node Exporter service in the unit file:

[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Start and enable the node_exporter service:

sudo systemctl daemon-reload

sudo systemctl start node_exporter

sudo systemctl enable node_exporter

You can retrieve the metrics served by Node Exporter like so:
 curl localhost:9100/metrics

Configure Prometheus to Scrape Metrics

Log in to your Prometheus server and configure Prometheus to scrape metrics from
the new server.

Edit the Prometheus config file:

sudo vi /etc/prometheus/prometheus.yml

Locate the scrape_configs section and add a new entry under that section. You will
need to supply the private IP address of your new playground server for targets.

...

- job_name: 'Linux Server'

static_configs:
- targets: ['<PRIVATE_IP_ADDRESS_OF_NEW_SERVER>:9100']

...

Reload the Prometheus config:

sudo killall -HUP prometheus

Navigate to the Prometheus expression browser in your web browser using the public
IP address of your Prometheus server: <PROMETHEUS_SERVER_PUBLIC_IP>:9090.

Run some queries to retrieve metric data about your new server:

up

node_filesystem_avail_bytes
Lesson Description - Metrics and Labels
Prometheus uses a combination of metrics and labels to uniquely identify each set of
time-series data. In this lesson, we discuss metric names and labels. We also briefly
demonstrate a few simple queries using metric names and labels to give you a
general idea of how they are used.

Relevant Documentation

• Prometheus Metric Names and Labels

Lesson Reference

Access the Prometheus expression browser for your Prometheus server in a web
browser. Be sure to use the public IP address of your Prometheus server:

http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090

Run a simple query using a metric name:

node_cpu_seconds_total

Add a label to the query to retrieve usage data only for a specific CPU:

node_cpu_seconds_total{cpu="0"}

Add additional labels to the query to retrieve an even narrower dataset:

node_cpu_seconds_total{cpu="0",mode="idle"}
Lesson Description - Metric Types
Prometheus exporters use a handful of metric types to represent different kinds of
data. In this lesson, we will discuss what metric types are. We will also discuss and
demonstrate examples of the four Prometheus metric types.

Relevant Documentation

• Metric Types
• Histograms and Summaries

Lesson Reference

Access the Prometheus expression browser for your Prometheus server in a web
browser. Be sure to use the public IP address of your Prometheus server:

http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090

Query a range of data on a counter. Note the values never decrease:

node_cpu_seconds_total[5m]

Query a range of data on a gauge. Note the values can both increase and decrease:

node_memory_MemAvailable_bytes[5m]

Query a histogram's buckets:

prometheus_http_request_duration_seconds_bucket{handler="/metrics"}

Check the histogram's sum metric:

prometheus_http_request_duration_seconds_sum{handler="/metrics"}
Query the histogram's count metric:

prometheus_http_request_duration_seconds_count{handler="/metrics"}

Query a quantile metric:

go_gc_duration_seconds{job="Linux Server"}

Query a quantile metric's sum:

go_gc_duration_seconds_sum{job="Linux Server"}

Query a quantile metric's count:

go_gc_duration_seconds_count{job="Linux Server"}
Lesson Description - Query Basics
Prometheus Query Language provides a robust interface for working with your metric
data. In this lesson, we will introduce the basic concepts of writing Prometheus
Queries. We will demonstrate these concepts by writing queries using the
Prometheus Expression Browser.

Relevant Documentation

• Querying Prometheus

Lesson Reference

Access the Prometheus expression browser for your Prometheus server in a web
browser. Be sure to use the public IP address of your Prometheus server:

http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090

Run a simple query using a time-series selector:

node_cpu_seconds_total

Use a time-series selector with a label:

node_cpu_seconds_total{cpu="0"}

Run some queries to experiment with various types of label matching:

node_cpu_seconds_total{cpu="0"}

node_cpu_seconds_total{cpu!="0"}

node_cpu_seconds_total{mode=~"s.*"}
 node_cpu_seconds_total{mode=~"user|system"}

node_cpu_seconds_total{mode!~"user|system"}

Use a range vector selector to select time-series values over a period of two minutes:

node_cpu_seconds_total{cpu="0"}[2m]

Select data from the past using an offset modifier:

node_cpu_seconds_total{cpu="0"} offset 1h

Combine a range vector selector with an offset modifier:

node_cpu_seconds_total{cpu="0"}[5m] offset 1h
Lesson Description - Query Operators
PromQL includes a wide variety of operators you can use to manipulate data in your
queries. In this lesson, we provide an overview of what the various types of operators
can do. We also demonstrate how to use operators by building some queries.

Relevant Documentation

• Operators

Lesson Reference

Access the Prometheus expression browser for your Prometheus server in a web
browser. Be sure to use the public IP address of your Prometheus server:

http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090

Use an arithmetic operator:

node_cpu_seconds_total * 2

Experiment with combining datasets using an arithmetic operator:

node_cpu_seconds_total{mode="system"} + ignoring(mode)
node_cpu_seconds_total{mode="user"}

node_cpu_seconds_total{mode="system"} + on(cpu)
node_cpu_seconds_total{mode="user"}

Use a comparison operator to filter results:

node_cpu_seconds_total == 0

Use the bool keyword to display the results of a comparison:

 node_cpu_seconds_total == bool 0

Experiment with combining datasets using logical set operators:

node_cpu_seconds_total

node_cpu_guest_seconds_total

node_cpu_seconds_total and node_cpu_guest_seconds_total

node_cpu_seconds_total or node_cpu_guest_seconds_total

Use an aggregation operator:

node_cpu_seconds_total{mode="idle"}

avg(node_cpu_seconds_total{mode="idle"})

sum(node_cpu_seconds_total{mode="idle"})
Lesson Description - Query Functions
Prometheus functions provide a wide range of built-in functionality that can greatly
simplify the process of writing queries. In this lesson, we introduce Prometheus
query functions. We will provide a few examples of functions and demonstrate their
usage by executing some queries that include those functions.

Relevant Documentation

• Functions

Lesson Reference

Access the Prometheus expression browser for your Prometheus server in a web
browser. Be sure to use the public IP address of your Prometheus server:

http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090

Use the clamp_max function to clamp result values to a set maximum:

clamp_max(node_cpu_seconds_total{cpu="0"}, 1000)

Use the rate function to get the rate of increase in total CPU seconds:

rate(node_cpu_seconds_total[1h])
Lesson Description - Prometheus HTTP
API
One way to execute queries is through the Prometheus HTTP API. This API is a great
way to interact with Prometheus from the context of your own custom tools and
applications. In this lesson, we introduce the HTTP API and demonstrate the process
of running some basic queries via the API.

Relevant Documentation

• HTTP API

Lesson Reference

Log in to your Prometheus server.

Make an HTTP request to the /api/v1/query endpoint to execute a query via the
HTTP API:

curl localhost:9090/api/v1/query?query=node_cpu_seconds_total

For queries that contain certain characters, you may need to URL-encode the query:

curl localhost:9090/api/v1/query --data-urlencode

"query=node_cpu_seconds_total{cpu=\"0\"}"

Use the /api/v1/query_range endpoint to query a range vector. Supply start and en
d parameters to specify the start and end of the time-range.

The step parameter determines how detailed the results will be. A step duration of 1
m will return values for every one minute during the specified time range:

start=$(date --date '-5 min' +'%Y-%m-%dT%H:%M:%SZ')

end=$(date +'%Y-%m-%dT%H:%M:%SZ')
curl "localhost:9090/api/v1/query_range?
query=node_cpu_seconds_total&start=$start&end=$end&step=1m"