Documentation

2.17.0 (May 11, 2021)

Percona Technical Documentation Team

Percona LLC, © 2021

 Table of contents

Table of contents

1. Welcome 5

1.1 Setting up 5

1.2 How it works 5

1.3 Documentation site map 7

2. Setting up 8

2.1 Setting up 8

2.2 Server 10

2.3 Client 56

3. Using 115

3.1 Using 115

3.2 User Interface 116

3.3 Integrated Alerting 121

3.4 Query Analytics 129

3.5 Percona Platform 139

4. How to 153

4.1 How to 153

4.2 Configure 154

4.3 Upgrade 166

4.4 Secure 167

4.5 Optimize 169

4.6 Annotate 170

4.7 Backup 171

4.8 Render dashboard images 173

4.9 Troubleshoot 175

5. Details 178

5.1 Details 178

5.2 Architecture 179

5.3 UI components 182

5.4 Dashboards 186

5.5 Commands 287

5.6 API 309

5.7 VictoriaMetrics 311

5.8 Glossary 312

6. FAQ 314

6.1 How can I contact the developers? 314

2 of 359 Percona LLC, © 2021

 Table of contents

6.2 What are the minimum system requirements for PMM? 314

6.3 How can I upgrade from PMM version 1? 314

6.4 How to control data retention for PMM? 314

6.5 How often are NGINX logs in PMM Server rotated? 314

6.6 What privileges are required to monitor a MySQL instance? 314

6.7 Can I monitor multiple service instances? 314

6.8 Can I rename instances? 315

6.9 Can I add an AWS RDS MySQL or Aurora MySQL instance from a non-default AWS partition? 315

6.10 How do I troubleshoot communication issues between PMM Client and PMM Server? 315

6.11 What resolution is used for metrics? 315

6.12 How do I set up Alerting in PMM? 316

6.13 How do I use a custom Prometheus configuration file inside PMM Server? 316

6.14 How to troubleshoot an Update? 316

6.15 What are my login credentials when I try to connect to a Prometheus Exporter? 316

6.16 How to provision PMM Server with non-default admin password? 316

7. Release Notes 318

7.1 Release Notes 318

7.2 Percona Monitoring and Management 2.17.0 319

7.3 Percona Monitoring and Management 2.16.0 321

7.4 Percona Monitoring and Management 2.15.1 324

7.5 Percona Monitoring and Management 2.15.0 325

7.6 Percona Monitoring and Management 2.14.0 329

7.7 Percona Monitoring and Management 2.13.0 331

7.8 Percona Monitoring and Management 2.12.0 333

7.9 Percona Monitoring and Management 2.11.1 335

7.10 Percona Monitoring and Management 2.11.0 336

7.11 Percona Monitoring and Management 2.10.1 337

7.12 Percona Monitoring and Management 2.10.0 338

7.13 Percona Monitoring and Management 2.9.1 340

7.14 Percona Monitoring and Management 2.9.0 341

7.15 Percona Monitoring and Management 2.8.0 342

7.16 Percona Monitoring and Management 2.7.0 343

7.17 Percona Monitoring and Management 2.6.1 344

7.18 Percona Monitoring and Management 2.6.0 345

7.19 Percona Monitoring and Management 2.5.0 347

7.20 Percona Monitoring and Management 2.4.0 348

7.21 Percona Monitoring and Management 2.3.0 350

7.22 Percona Monitoring and Management 2.2.2 351

3 of 359 Percona LLC, © 2021

 Table of contents

7.23 Percona Monitoring and Management 2.2.1 352

7.24 Percona Monitoring and Management 2.2.0 354

7.25 Percona Monitoring and Management 2.1.0 356

7.26 Percona Monitoring and Management 2.0.1 358

7.27 Percona Monitoring and Management 2.0.0 359

4 of 359 Percona LLC, © 2021

 1. Welcome

1. Welcome

Percona Monitoring and Management (PMM) is a free, open-source monitoring tool for MySQL,
PostgreSQL, MongoDB, and ProxySQL, and the servers they run on. PMM helps you improve the
performance of databases, simplify their management, and strengthen their security.

This documentation is for the latest release: PMM 2.17.0

With PMM, you can:

• Visualize a wide range of out-of-the-box system performance metrics

• Collect and analyze data across complex multi-vendor system topologies

• Drill-down and discover the cause of inefficiencies, anticipate performance issues, or troubleshoot
existing ones

• Watch for potential security issues and remedy them

Try the live demo: pmmdemo.percona.com

PMM is efficient, quick to set up and easy to use. It runs in cloud, on-prem, or across hybrid platforms. It is
supported by our legendary expertise in open source databases, and by a vibrant developer and user
community.

1.1 Setting up
PMM Server can run as:

• a Docker container

• a virtual machine

• an Amazon AWS EC2 instance

PMM Client runs on all hosts you want to monitor. The setup varies according to the type of system:

• MySQL and variants

• MongoDB

• PostgreSQL

• Amazon RDS

• Microsoft Azure

• Google Cloud Platform

• ProxySQL

• Linux

• External services

• HAProxy

Quickstart installation

1.2 How it works

PMM is a client/server application built by us with our own and third-party open-source tools. (Read more
in Architecture.)

5 of 359 Percona LLC, © 2021

 1.2 How it works

PMM Server

PMM Server is the heart of PMM. It receives data from clients, collates it and stores it. Metrics are drawn as
tables, charts and graphs within dashboards, each a part of the web-based user interface.

This is the home dashboard from pmmdemo:

PMM Client

6 of 359 Percona LLC, © 2021

 1.3 Documentation site map

PMM Client runs on every database host or node you want to monitor. The client collects server metrics,
general system metrics, and query analytics data, and sends it to the server.

Percona Platform

Percona Platform (in development) provides value-added services for PMM.

1.3 Documentation site map

Welcome

Setting up Using How to Details

Server Client User Configure Dashboards Architecture

interface

Upgrade User Interface

Docker Docker Insight
Integrated components
Alerting
Secure
Virtual MySQL PMM
Appliance Commands
Query
Analytics Optimize
Percona OS
AWS Server pmm-admin
Marketplace Render
Percona Prometheus
Platform dashboard
MongoDB pmm-agent
images
MySQL
PostgreSQL Security API
Troubleshoot
Threat Tool
MongoDB
ProxySQL VictoriaMetrics

PostgreSQL
Amazon RDS Glossary

ProxySQL
Microsoft Azure

HA
Google Cloud Platform

Linux

External
Services

HAProxy

7 of 359 Percona LLC, © 2021

 2. Setting up

2. Setting up

2.1 Setting up
The PMM setting-up process can be broken into three key stages:

1. Setting up at least one PMM Server

2. Setting up one or more PMM Clients

3. Configuring and adding services for monitoring

2.1.1 Setting up PMM Server

You must set up at least one PMM Server. A server can run:

• with Docker

• as a virtual appliance

• on an Amazon AWS EC2 instance

2.1.2 Setting up PMM Client

You must set up PMM Client on each node where there is a service to be monitored. You can do this:

1. with a package manager ( apt , apt-get , dnf , yum )

2. by manually downloading and installing .deb or .rpm packages

3. by manually downloading and unpacking a binary package ( .tar.gz )

4. with a Docker image

8 of 359 Percona LLC, © 2021

 2.1.3 Configure and add services

2.1.3 Configure and add services

You must configure your services and add them to PMM Server’s inventory of monitored systems. This is
different for each type of service:

• MySQL and variants (Percona Server for MySQL, Percona XtraDB Cluster, MariaDB)

• MongoDB

• PostgreSQL

• ProxySQL

• Amazon RDS

• Microsoft Azure

• Google Cloud Platform (MySQL and PostgreSQL)

• Linux

• External services

• HAProxy

You do this on each node/service being monitored.

If you have configured everything correctly, you’ll see data in the PMM user interface, in one of the
dashboards specific to the type of service.

9 of 359 Percona LLC, © 2021

 2.2 Server

2.2 Server

2.2.1 Setting up PMM Server

System requirements

Disk

Approximately 1 GB of storage per monitored database node with data retention set to one week. By default,
retention is 30 days.

Disable table statistics to decrease the VictoriaMetrics database size.

Memory

A minimum of 2 GB per monitored database node. The increase in memory usage is not proportional to the
number of nodes. For example, data from 20 nodes should be easily handled with 16 GB.

Architecture

Your CPU must support the SSE4.2 instruction set, a requirement of ClickHouse, a third-party column-
oriented database used by Query Analytics. If your CPU is lacking this instruction set you won’t be able to
use Query Analytics.

Choose how you want to run PMM Server:

• with Docker

• as a virtual appliance

• on an Amazon AWS EC2 instance

When PMM Server is running, set up PMM Client for each node or service.

Single line installer

Caution This is a technical preview and is subject to change.

curl -fsSL https://raw.githubusercontent.com/percona/pmm/PMM-2.0/get-pmm.sh -o get-pmm2.sh ;

chmod +x get-pmm2.sh ; ./get-pmm2.sh

Warning We highly recommend you review get-pmm2.sh prior to running on your system, to ensure the
content is as expected.

This command will:

• if Docker is not already installed, install it

• if there is a PMM Server docker container running, stop it and back it up

• pull and run the latest PMM Server docker image

10 of 359 Percona LLC, © 2021

 2.2.2 Docker

2.2.2 Docker

We maintain a Docker image for PMM Server. This section shows how to run PMM Server as a Docker
container, directly and with Docker compose. (The tags used here are for the latest version of PMM 2
(2.17.0). Other tags are available.)

System requirements

Software

• Docker 1.12.6 or higher.

• (Optional) Docker compose

Running PMM Server with Docker

1. Pull the image.

sudo docker pull percona/pmm-server:2

2. Create a persistent data container.

sudo docker create --volume /srv \

--name pmm-data percona/pmm-server:2 /bin/true

PMM Server expects the data volume (specified with --volume ) to be /srv . Using any other value will
result in data loss when upgrading.

3. Run the image to start PMM Server.

sudo docker run --detach --restart always \

--publish 443:443 \
--volumes-from pmm-data --name pmm-server \
percona/pmm-server:2

Optionally you can enable http (insecure) by including --publish 80:80 in the above docker run
command however note that PMM Client requires TLS to communication with the server so will only work
on the secure port.

You can disable manual updates via the Home Dashboard PMM Upgrade panel by adding -e
DISABLE_UPDATES=true to the docker run command.

4. In a web browser, visit https://server hostname:443 (or //server hostname:80 if optionally enabled) to
see the PMM user interface.

PRO Tip: Eliminate browser certificate warnings by configuring a trusted certificate

11 of 359 Percona LLC, © 2021

 2.2.2 Docker

DOCKER ENVIRONMENT VARIABLES

It is possible to change some server setting by using environment variables when starting the Docker
container. Use -e var=value in your pmm-server run command.

Variable Description

DISABLE_UPDATES Disable automatic updates

DISABLE_TELEMETRY Disable built-in telemetry and disable STT if telemetry is disabled

METRICS_RESOLUTION High metrics resolution in seconds

METRICS_RESOLUTION_HR High metrics resolution (same as above)

METRICS_RESOLUTION_MR Medium metrics resolution in seconds

METRICS_RESOLUTION_LR Low metrics resolution in seconds

DATA_RETENTION How many days to keep time-series data in ClickHouse

ENABLE_VM_CACHE Enable cache in VM

ENABLE_ALERTING Enable integrated alerting

ENABLE_AZUREDISCOVER Enable support for discovery of Azure databases

ENABLE_BACKUP_MANAGEMENT Enable integrated backup tools

PERCONA_TEST_SAAS_HOST SaaS server hostname

PERCONA_TEST_DBAAS Enable testing DBaaS features. (Will be deprecated in future versions.)

ENABLE_DBAAS Enable DBaaS features

PMM_DEBUG Enables a more verbose log level

PMM_TRACE Enables a more verbose log level including trace-back information

Ignored variables

These variables will be ignored by pmm-managed when starting the server. If any other variable is found, it
will be considered invalid and the server won’t start.

Variable Description

_ , HOME , HOSTNAME , LANG , PATH , PWD , SHLVL , Default environment variables

TERM

GF_* Grafana’s environment variables

SUPERVISOR_ Supervisord environment variables

PERCONA_TEST_ Unknown variable but won’t prevent the server to

start

PERCONA_TEST_DBAAS Deprecated. Use ENABLE_DBAAS

12 of 359 Percona LLC, © 2021

 2.2.2 Docker

Backup and upgrade

You can test a new release of the PMM Server Docker image by making backups of your current pmm-server
and pmm-data containers which you can restore if you need to.

1. Find out which release you have now.

sudo docker exec -it pmm-server curl -u admin:admin https://localhost/v1/version

Tip: Use jq to extract the quoted string value.

sudo apt install jq # Example for Debian, Ubuntu

sudo docker exec -it pmm-server curl -u admin:admin https://localhost/v1/version |
jq .version

2. Check the container mount points are the same ( /srv ).

sudo docker inspect pmm-data | grep Destination

sudo docker inspect pmm-server | grep Destination

With jq :

sudo docker inspect pmm-data | jq '.[].Mounts[].Destination'

sudo docker inspect pmm-server | jq '.[].Mounts[].Destination'

3. Stop the container and create backups.

sudo docker stop pmm-server

sudo docker rename pmm-server pmm-server-backup
mkdir pmm-data-backup && cd $_
sudo docker cp pmm-data:/srv .

4. Pull the latest image and run the container.

sudo docker pull percona/pmm-server:2

sudo docker run \
--detach \
--restart always \
--publish 443:443 \
--volumes-from pmm-data \
--name pmm-server \
percona/pmm-server:2

5. (Optional) Repeat step 1 to confirm the version, or check the PMM Upgrade panel on the Home
Dashboard.

Restore

1. Stop and remove the running version.

sudo docker stop pmm-server

sudo docker rm pmm-server

2. Restore backups.

13 of 359 Percona LLC, © 2021

 2.2.2 Docker

sudo docker rename pmm-server-backup pmm-server

# cd to wherever you saved the backup
sudo docker cp srv pmm-data:/

3. Restore permissions.

sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R root:root /srv
&& \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R pmm:pmm /srv/
alertmanager && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R root:pmm /srv/
clickhouse && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R
grafana:grafana /srv/grafana && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R pmm:pmm /srv/
logs && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R
postgres:postgres /srv/postgres && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R pmm:pmm /srv/
prometheus && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R pmm:pmm /srv/
victoriametrics && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R
postgres:postgres /srv/logs/postgresql.log

4. Start (don’t run) the image.

sudo docker start pmm-server

14 of 359 Percona LLC, © 2021

 2.2.2 Docker

Running PMM Server with Docker compose

With this approach, data is stored in a volume, not in a pmm-data container.

1. Copy and paste this text into a file called docker-compose.yml .

version: '2'
services:
pmm-server:
image: percona/pmm-server:2
hostname: pmm-server
container_name: pmm-server
restart: always
logging:
driver: json-file
options:
max-size: "10m"
max-file: "5"
ports:
- "443:443"
volumes:
- data:/srv
volumes:
data:

2. Run:

sudo docker-compose up

3. Access PMM Server on https://X.X.X.X:443 where X.X.X.X is the IP address of the PMM Server host.

See also Run PMM Client with Docker compose

Removing PMM Server

1. Stop pmm-server container.

docker stop pmm-server

2. Remove containers.

docker rm pmm-server pmm-data

3. Remove the image.

docker rmi $(docker images | grep "percona/pmm-server" | awk {'print $3'})

15 of 359 Percona LLC, © 2021

 2.2.2 Docker

Hosts with no internet connectivity

If the host where you will run PMM Server has no internet connection, you can download the Docker image
on a separate (internet-connected) host and securely copy it.

1. On an internet-connected host, download the Docker image and its checksum file.

wget https://downloads.percona.com/downloads/pmm2/2.17.0/docker/pmm-server-2.17.0.docker
wget https://downloads.percona.com/downloads/pmm2/2.17.0/docker/pmm-server-2.17.0.sha256sum

2. Copy both files to where you will run PMM Server.

3. Open a terminal on the PMM Server host.

4. (Optional) Check the Docker image file integrity.

shasum -ca 256 pmm-server-2.17.0.sha256sum

5. Load the image.

sudo docker load -i pmm-server-2.17.0.docker

6. Create the pmm-data persistent data container.

sudo docker create --volume /srv \

--name pmm-data percona/pmm-server:2.17.0 /bin/true

7. Run the container.

sudo docker run \

--detach \
--restart always \
--publish 443:443 \
--volumes-from pmm-data \
--name pmm-server \
percona/pmm-server:2.17.0

16 of 359 Percona LLC, © 2021

 2.2.3 Virtual Appliance

2.2.3 Virtual Appliance

Run PMM Server as a virtual machine by downloading and importing the PMM 2.17.0 Open Virtual Appliance
(OVA) file into any virtualization software supporting the OVF standard.

This page shows how to set up PMM Server as a virtual machine in VMware Workstation Player and Oracle
VM VirtualBox.

17 of 359 Percona LLC, © 2021

 2.2.3 Virtual Appliance

Most steps can be done with either a user interface or on the command line, but some steps can only be
done in one or the other. Sections are labeled UI for user interface or CLI for command line instructions.

18 of 359 Percona LLC, © 2021

 2.2.3 Virtual Appliance

Terminology

• Host is the desktop or server machine running the hypervisor.

• Hypervisor is software (e.g. VirtualBox, VMware) that runs the guest OS as a virtual machine.

• Guest is the CentOS virtual machine that runs PMM Server.

OVA file details

Item Value

Download page https://www.percona.com/downloads/pmm2/2.17.0/ova

File name pmm-server-2.17.0.ova

VM name PMM2-Server-2021-05-11-N ( N =build number)

VM specifications

Component Value

OS CentOS 7.9 (64-bit)

CPU 1

Base memory 4096 MB

Disks LVM, 2 physical volumes

Disk 1 (sda) VMDK (SCSI, 40 GB)

Disk 2 (sdb) VMDK (SCSI, 400 GB)

Users

Default Username Default password

root percona

admin admin

Download

UI

1. Open a web browser.

2. Visit the PMM Server download page.

3. Choose a Version or use the default (the latest).

4. Click the link for pmm-server-2.17.0.ova to download it. Note where your browser saves it.

5. Right click the link for pmm-server-2.17.0.sha256sum and save it in the same place as the .ova file.

6. (Optional) Verify.

CLI

Download the latest PMM Server OVA and checksum files

wget https://www.percona.com/downloads/pmm2/2.17.0/ova/pmm-server-2.17.0.ova
wget https://www.percona.com/downloads/pmm2/2.17.0/ova/pmm-server-2.17.0.sha256sum

19 of 359 Percona LLC, © 2021

 2.2.3 Virtual Appliance

Verify

CLI

Verify the checksum of the downloaded .ova file

shasum -ca 256 pmm-server-2.17.0.sha256sum

PMM Server as a virtual appliance on VMware Workstation Player

IMPORT

UI

1. Select File → Import.

2. Click Choose file….

3. Navigate to the downloaded .ova file and select it.

4. Click Open.

5. Click Continue.

6. In the Save as dialog:

a. (Optional) Change the directory or file name.

b. Click Save.

7. Choose one of:

• (Optional) Click Finish. This starts the virtual machine.

• (Recommended) Click Customize Settings. This opens the VM’s settings page without starting the
machine.

CLI

1. Install ovftool . (You need to register.)

2. Import and convert the OVA file. ( ovftool can’t change CPU or memory settings during import but it can
set the default interface.)

Choose one of: - Download and import the OVA file.

ovftool --name="PMM Server" --net:NAT=Wi-Fi\

https://www.percona.com/downloads/pmm2/2.17.0/ova/pmm-server-2.17.0.ova\
pmm-server-2.17.0.vmx

- Import an already-downloaded OVA file.

ovftool --name="PMM Server" --net:NAT=WiFi\

pmm-server-2.17.0.ova\
pmm-server.vmx

RECONFIGURE INTERFACE

When using the command line, the interface is remapped during import.

20 of 359 Percona LLC, © 2021

 2.2.3 Virtual Appliance

UI

1. If started, shut down the virtual machine.

2. In the VMware main window, select the imported virtual machine.

3. Click Virtual Machine → Settings…

4. Click Network Adapter.

5. In the Bridged Networking section, select Autodetect.

6. Close the settings window.

START GUEST AND GET IP ADDRESS

UI

1. In the VMware main window, select the imported virtual machine.

2. Click the play button  or select Virtual Machine → Start Up.

3. When the instance has booted, note the IP address in the guest console.

CLI/UI

1. Start the virtual machine in GUI mode. (There’s no way to redirect a VMware VM’s console to the host.)

vmrun -gu root -gp percona start \

pmm-server.vmx gui

2. When the instance has booted, note the IP address in the guest console.

3. (Optional) Stop and restart the instance in headless mode.

vmrun stop pmm-server.vmx

vmrun -gu root -gp percona start \
pmm-server.vmx nogui

PMM Server as a virtual appliance on Oracle VM VirtualBox

IMPORT

UI

1. Select File → Import appliance….

2. In the File field, type the path to the downloaded .ova file, or click the folder icon to navigate and open it.

3. Click Continue.

4. On the Appliance settings page, review the settings and click Import.

5. Click Start.

6. When the guest has booted, note the IP address in the guest console.

21 of 359 Percona LLC, © 2021

 2.2.3 Virtual Appliance

CLI

1. Open a terminal and change directory to where the downloaded .ova file is.

2. (Optional) Do a ‘dry run’ import to see what values will be used.

VBoxManage import pmm-server-2.17.0.ova --dry-run

3. Import the image. Choose one of:

• With the default settings.

VBoxManage import pmm-server-2.17.0.ova

• With custom settings (in this example, Name: “PMM Server”, CPUs: 2, RAM: 8192 MB).

VBoxManage import --vsys 0 --vmname "PMM Server"\

--cpus 2 --memory 8192 pmm-server-2.17.0.ova

RECONFIGURE INTERFACE

UI

1. Click Settings.

2. Click Network.

3. In the Adapter 1 field, click Attached to and change to Bridged Adapter.

4. In the Name field, select your host’s active network interface (e.g. en0: Wi-Fi (Wireless) ).

5. Click OK.

CLI

1. Show the list of available bridge interfaces.

VBoxManage list bridgedifs

2. Find the name of the active interface you want to bridge to (one with Status: Up and a valid IP address).
Example: en0: Wi-Fi (Wireless)

3. Bridge the virtual machine’s first interface ( nic1 ) to the host’s en0 ethernet adapter.

VBoxManage modifyvm 'PMM Server'\

--nic1 bridged --bridgeadapter1 'en0: Wi-Fi (Wireless)'

4. Redirect the console output into a host file.

VBoxManage modifyvm 'PMM Server'\

--uart1 0x3F8 4 --uartmode1 file /tmp/pmm-server-console.log

START GUEST AND GET IP ADDRESS

UI

1. Select the PMM Server virtual machine in the list.

2. Click Start.

3. When the guest has booted, note the IP address in the guest console.

22 of 359 Percona LLC, © 2021

 2.2.3 Virtual Appliance

CLI

1. Start the guest.

VBoxManage startvm --type headless 'PMM Server'

2. (Optional) Watch the log file.

tail -f /tmp/pmm-server-console.log

3. Wait for one minute for the server to boot up.

4. Choose one of:

• Read the IP address from the tailed log file.

• Extract the IP address from the log file.

grep -e "^IP:" /tmp/pmm-server-console.log | cut -f2 -d' '

5. (Optional) Stop the guest:

VBoxManage controlvm "PMM Server" poweroff

23 of 359 Percona LLC, © 2021

 2.2.3 Virtual Appliance

Log into PMM user interface

UI

1. Open a web browser and visit the guest IP address.

2. The PMM login screen appears.

3. Enter the default username and password in the relevant fields and click Log in.

• username: admin

• password: admin

4. (Recommended) Follow the prompts to change the default password.

5. The PMM Home Dashboard appears.

24 of 359 Percona LLC, © 2021

 2.2.3 Virtual Appliance

(Optional) Change root password

UI

1. Start the virtual machine in GUI mode.

2. Log in with the default superuser credentials:

• Username: root

• Password: percona

3. Follow the prompts to change the password.

(Optional) Set up SSH

UI/CLI

1. Create a key pair for the admin user.

ssh-keygen -f admin

2. Log into the PMM user interface.

3. Select PMM → PMM Settings → SSH Key.

4. Copy and paste the contents of the admin.pub file into the SSH Key field.

5. Click Apply SSH Key. (This copies the public key to /home/admin/.ssh/authorized_keys in the guest).

6. Log in via SSH ( N.N.N.N is the guest IP address).

ssh -i admin [email protected]

(Optional) Set up static IP

When the guest OS starts, it will get an IP address from the hypervisor’s DHCP server. This IP can change
each time the guest OS is restarted. Setting a static IP for the guest OS avoids having to check the IP
address whenever the guest is restarted.

25 of 359 Percona LLC, © 2021

 2.2.3 Virtual Appliance

CLI

1. Start the virtual machine in non-headless (GUI) mode.

2. Log in as root .

3. Edit /etc/sysconfig/network-scripts/ifcfg-eth0

4. Change the value of BOOTPROTO :

BOOTPROTO=none

5. Add these values:

IPADDR=192.168.1.123 # Example
NETMASK=255.255.255.0
GATEWAY=192.168.1.1

6. Restart the interface.

ifdown eth0 && ifup eth0

7. Check the IP.

ip addr show eth0

Remove PMM Server

UI

1. Stop the virtual machine. Select Close → Power Off

2. Remove the virtual machine. Select Remove → Delete all files

26 of 359 Percona LLC, © 2021

 2.2.4 AWS Marketplace

2.2.4 AWS Marketplace

You can run an instance of PMM Server hosted at AWS Marketplace.

Assuming that you have an AWS (Amazon Web Services) account, locate Percona Monitoring and
Management Server in AWS Marketplace (or use this link).

Selecting a region and instance type in the Pricing Information section will give you an estimate of the costs
involved. This is only an indication of costs. You will choose regions and instance types in later steps.

Percona Monitoring and Management Server is provided at no cost, but you may need to pay for
infrastructure costs.

Disk space consumed by PMM Server depends on the number of hosts being monitored. Although each
environment will be unique, you can consider the data consumption figures for the PMM Demo web site
which consumes approximately 230MB/host/day, or ~6.9GB/host at the default 30 day retention period.

For more information, see our blog post How much disk space should I allocate for Percona Monitoring
and Management?.

1. Click Continue to Subscribe.

2. Subscribe to this software: Check the terms and conditions and click Continue to Configuration.

3. Configure this software:

a. Select a value for Software Version. (The latest is 2.17.0)

b. Select a region. (You can change this in the next step.)

c. Click Continue to Launch.

4. Launch this software:

a. Choose Action: Select a launch option. Launch from Website is a quick way to make your instance
ready. For more control, choose Launch through EC2.

b. EC2 Instance Type: Select an instance type.

c. VPC Settings: Choose or create a VPC (virtual private cloud).

d. Subnet Settings: Choose or create a subnet.

e. Security Group Settings: Choose a security group or click *Create New Based On Seller Settings

f. Key Pair Settings: Choose or create a key pair.

g. Click Launch.

Limiting Access to the instance: security group and a key pair

In the Security Group section, which acts like a firewall, you may use the preselected option Create new
based on seller settings to create a security group with recommended settings. In the Key Pair select an
already set up EC2 key pair to limit access to your instance.

27 of 359 Percona LLC, © 2021

 2.2.4 AWS Marketplace

It is important that the security group allow communication via the the following ports: 22, 80, and 443.
PMM should also be able to access port 3306 on the RDS that uses the instance.

Applying settings

Scroll up to the top of the page to view your settings. Then, click the Launch with 1 click button to continue
and adjust your settings in the EC2 console.

28 of 359 Percona LLC, © 2021

 2.2.4 AWS Marketplace

Your instance settings are summarized in a special area. Click the Launch with 1 click button to continue.

The Launch with 1 click button may alternatively be titled as Accept Software Terms & Launch with 1-Click.

Adjusting instance settings in the EC2 Console

Your clicking the Launch with 1 click button, deploys your instance. To continue setting up your instance,
run the EC2 console. It is available as a link at the top of the page that opens after you click the Launch
with 1 click button.

Your instance appears in the EC2 console in a table that lists all instances available to you. When a new
instance is only created, it has no name. Make sure that you give it a name to distinguish from other
instances managed via the EC2 console.

Running the instance

After you add your new instance it will take some time to initialize it. When the AWS console reports that the
instance is now in a running state, you many continue with configuration of PMM Server.

When started the next time after rebooting, your instance may acquire another IP address. You may
choose to set up an elastic IP to avoid this problem.

With your instance selected, open its IP address in a web browser. The IP address appears in the IPv4
Public IP column or as value of the Public IP field at the top of the Properties panel.

29 of 359 Percona LLC, © 2021

 2.2.4 AWS Marketplace

To run the instance, copy and paste its public IP address to the location bar of your browser. In the
Percona Monitoring and Management welcome page that opens, enter the instance ID.

You can copy the instance ID from the Properties panel of your instance, select the Description tab back in
the EC2 console. Click the Copy button next to the Instance ID field. This button appears as soon as you
hover the cursor of your mouse over the ID.

Hover the cursor over the instance ID for the Copy button to appear.

Paste the instance in the Instance ID field of the Percona Monitoring and Management welcome page and
click Submit.

PMM Server provides user access control, and therefore you will need user credentials to access it:

30 of 359 Percona LLC, © 2021

 2.2.4 AWS Marketplace

• Default user name: admin

• Default password: admin

You will be prompted to change the default password every time you log in.

The PMM Server is now ready and the home page opens.

31 of 359 Percona LLC, © 2021

 2.2.4 AWS Marketplace

You are creating a username and password that will be used for two purposes:

1. authentication as a user to PMM - the credentials to log in to PMM.

2. authentication between PMM Server and PMM Clients - you will re-use these credentials when configuring
PMM Client for the first time on a server, for example:

pmm-admin config --server-insecure-tls --server-url=https://admin:admin@<IP Address>:443

For instructions about how to access your instances by using an SSH client, see Connecting to Your
Linux Instance Using SSH

Make sure to replace the user name ec2-user used in this document with admin .

Resizing the EBS Volume

Your AWS instance comes with a predefined size which can become a limitation. To make more disk space
available to your instance, you need to increase the size of the EBS volume as needed and then your
instance will reconfigure itself to use the new size.

The procedure of resizing EBS volumes is described in the Amazon documentation: Modifying the Size,
IOPS, or Type of an EBS Volume on Linux.

After the EBS volume is updated, PMM Server instance will auto-detect changes in approximately 5 minutes
or less and will reconfigure itself for the updated conditions.

Upgrading PMM Server on AWS

UPGRADING EC2 INSTANCE CLASS

Upgrading to a larger EC2 instance class is supported by PMM provided you follow the instructions from the
AWS manual. The PMM AMI image uses a distinct EBS volume for the PMM data volume which permits
independent resizing of the EC2 instance without impacting the EBS volume.

EXPANDING THE PMM DATA EBS VOLUME

The PMM data volume is mounted as an XFS formatted volume on top of an LVM volume. There are two
ways to increase this volume size:

1. Add a new disk via EC2 console or API, and expand the LVM volume to include the new disk volume.

2. Expand existing EBS volume and grow the LVM volume.

32 of 359 Percona LLC, © 2021

 2.2.4 AWS Marketplace

EXPAND EXISTING EBS VOLUME

To expand the existing EBS volume for increased capacity, follow these steps.

33 of 359 Percona LLC, © 2021

 2.2.4 AWS Marketplace

1. Expand the disk from AWS Console/CLI to the desired capacity.

2. Login to the PMM EC2 instance and verify that the disk capacity has increased. For example, if you have
expanded disk from 16G to 32G, dmesg output should look like below:

[ 535.994494] xvdb: detected capacity change from 17179869184 to 34359738368

3. You can check information about volume groups and logical volumes with the vgs and lvs commands:

vgs

VG #PV #LV #SN Attr VSize VFree

DataVG 1 2 0 wz--n- <16.00g 0

lvs

LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
DataLV DataVG Vwi-aotz-- <12.80g ThinPool 1.74
ThinPool DataVG twi-aotz-- 15.96g 1.39 1.29

4. Now we can use the lsblk command to see that our disk size has been identified by the kernel correctly,
but LVM2 is not yet aware of the new size. We can use pvresize to make sure the PV device reflects the
new size. Once pvresize is executed, we can see that the VG has the new free space available.

lsblk | grep xvdb

xvdb 202:16 0 32G 0 disk

pvscan

PV /dev/xvdb VG DataVG lvm2 [<16.00 GiB / 0 free]

Total: 1 [<16.00 GiB] / in use: 1 [<16.00 GiB] / in no VG: 0 [0 ]

pvresize /dev/xvdb

Physical volume "/dev/xvdb" changed

1 physical volume(s) resized / 0 physical volume(s) not resized

pvs

PV VG Fmt Attr PSize PFree

/dev/xvdb DataVG lvm2 a-- <32.00g 16.00g

5. We then extend our logical volume. Since the PMM image uses thin provisioning, we need to extend both
the pool and the volume:

lvs

34 of 359 Percona LLC, © 2021

 2.2.4 AWS Marketplace

LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
DataLV DataVG Vwi-aotz-- <12.80g ThinPool 1.77
ThinPool DataVG twi-aotz-- 15.96g 1.42 1.32

lvextend /dev/mapper/DataVG-ThinPool -l 100%VG

Size of logical volume DataVG/ThinPool_tdata changed from 16.00 GiB (4096 extents) to 31.96
GiB (8183 extents).
Logical volume DataVG/ThinPool_tdata successfully resized.

lvs

LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
DataLV DataVG Vwi-aotz-- <12.80g ThinPool 1.77
ThinPool DataVG twi-aotz-- 31.96g 0.71 1.71

6. Once the pool and volumes have been extended, we need to now extend the thin volume to consume the
newly available space. In this example we’ve grown available space to almost 32GB, and already
consumed 12GB, so we’re extending an additional 19GB:

lvs

LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
DataLV DataVG Vwi-aotz-- <12.80g ThinPool 1.77
ThinPool DataVG twi-aotz-- 31.96g 0.71 1.71

lvextend /dev/mapper/DataVG-DataLV -L +19G

Size of logical volume DataVG/DataLV changed from <12.80 GiB (3276 extents) to <31.80 GiB
(8140 extents).
Logical volume DataVG/DataLV successfully resized.

lvs

LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
DataLV DataVG Vwi-aotz-- <31.80g ThinPool 0.71
ThinPool DataVG twi-aotz-- 31.96g 0.71 1.71

7. We then expand the XFS file system to reflect the new size using xfs_growfs , and confirm the file system
is accurate using the df command.

df -h /srv

Filesystem Size Used Avail Use% Mounted on

/dev/mapper/DataVG-DataLV 13G 249M 13G 2% /srv

xfs_growfs /srv

meta-data=/dev/mapper/DataVG-DataLV isize=512 agcount=103, agsize=32752 blks

= sectsz=512 attr=2, projid32bit=1

35 of 359 Percona LLC, © 2021

 2.2.4 AWS Marketplace

= crc=1 finobt=0 spinodes=0

data = bsize=4096 blocks=3354624, imaxpct=25
= sunit=16 swidth=16 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=768, version=2
= sectsz=512 sunit=16 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 3354624 to 8335360

df -h /srv

Filesystem Size Used Avail Use% Mounted on

/dev/mapper/DataVG-DataLV 32G 254M 32G 1% /srv

Remove PMM Server

1. Find the instance in the EC2 Console

1. Select “Instance state” menu and “Terminate instance”

1. Confirm termination operation

36 of 359 Percona LLC, © 2021

 2.2.4 AWS Marketplace

37 of 359 Percona LLC, © 2021

 2.2.5 DBaaS (Alpha)

2.2.5 DBaaS (Alpha)

Caution DBaaS functionality is currently in Alpha status.

The information on this page is subject to change and may be inaccurate.

Software prerequisites

DOCKER

Red Hat, CentOS

yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

yum -y install docker-ce
usermod -a -G docker centos
systemctl enable docker
systemctl start docker

Debian, Ubuntu

apt-add-repository https://download.docker.com/linux/centos/docker-ce.repo
systemctl enable docker
systemctl start docker

MINIKUBE

Please install minikube 1.16.0

Red Hat, CentOS

yum -y install curl

curl -Lo /usr/local/sbin/minikube https://github.com/kubernetes/minikube/releases/download/
v1.16.0/minikube-linux-amd64
chmod +x /usr/local/sbin/minikube
ln -s /usr/local/sbin/minikube /usr/sbin/minikube
alias kubectl='minikube kubectl --'

38 of 359 Percona LLC, © 2021

 2.2.5 DBaaS (Alpha)

Start PMM server and activate a DBaaS feature

• To start a fully-working 3 node XtraDB cluster, consisting of sets of 3x HAProxy, 3x PXC and 6x PMM
Client containers, you will need at least 9 vCPU available for minikube. (1x vCPU for HAProxy and PXC
and 0.5vCPU for each pmm-client containers).

• DBaaS does not depend on PMM Client.

• You can pass the environment variable --env ENABLE_DBAAS=1 to force the DBaaS feature when
starting up pmm-server container. You can omit the variable and enable the feature later using
PMM UI, please follow the link in step 3. below.

• Add the option --network minikube if you run PMM Server and minikube in the same Docker instance.
(This will share a single network and the kubeconfig will work.)

• Add the options --env PMM_DEBUG=1 and/or --env PMM_TRACE=1 if you need extended debug details

1. Start PMM server:

docker run --detach --publish 80:80 --publish 443:443 --name pmm-server percona/pmm-server:2

2. Change the default administrator credentials from CLI:

(This step is optional, because the same can be done from the web interface of PMM on first login.)

docker exec -t pmm-server bash -c 'ln -s /srv/grafana /usr/share/grafana/data; chown -R

grafana:grafana /usr/share/grafana/data; grafana-cli --homepath /usr/share/grafana admin reset-
admin-password <RANDOM_PASS_GOES_IN_HERE>'

3. IMPORTANT: Please follow instructions on How to activate the DBaaS feature in Advanced Settings
of PMM.

You need to enable the feature using PMM UI if you omitted --env ENABLE_DBAAS=1 when starting up the
container.

Install Percona operators in minikube

1. Configure and start minikube:

minikube config set cpus 16

minikube config set memory 32768
minikube config set kubernetes-version 1.16.15
minikube start

2. Deploy the Percona operators configuration for PXC and PSMDB in minikube:

# Prepare a set of base64 encoded values and non encoded for user and pass with administrator
privileges to pmm-server (DBaaS)
PMM_USER='admin';
PMM_PASS='<RANDOM_PASS_GOES_IN_HERE>';

PMM_USER_B64="$(echo -n "${PMM_USER}" | base64)";

PMM_PASS_B64="$(echo -n "${PMM_PASS}" | base64)";

# Install the PXC operator

curl -sSf -m 30 https://raw.githubusercontent.com/percona/percona-xtradb-cluster-operator/
K8SPXC-668-CUSTOM/deploy/bundle.yaml \
| kubectl apply -f -

# Install the PSMDB operator

curl -sSf -m 30 https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/

39 of 359 Percona LLC, © 2021

 2.2.5 DBaaS (Alpha)

v1.7.0/deploy/bundle.yaml \
| kubectl apply -f -

3. Check the operators are deployed:

minikube kubectl -- get nodes

minikube kubectl -- get pods
minikube kubectl -- wait --for=condition=Available deployment percona-xtradb-cluster-operator
minikube kubectl -- wait --for=condition=Available deployment percona-server-mongodb-operator

4. Get your kubeconfig details from minikube (to register your Kubernetes cluster with PMM Server):

minikube kubectl -- config view --flatten --minify

You will need to copy this output to your clipboard and continue with add a Kubernetes cluster to PMM.

Installing Percona operators on Amazon AWS EKS

1. Create your cluster via eksctl or the Amazon AWS interface. For example:

eksctl create cluster --write-kubeconfig --name=your-cluster-name --zones=us-west-2a,us-

west-2b --kubeconfig <PATH_TO_KUBECONFIG>

2. When your EKS cluster is running, install the PXC and PSMDB operators:

# Prepare a set of base64 encoded values and non encoded for user and pass with administrator
privileges to pmm-server (DBaaS)
PMM_USER='admin';
PMM_PASS='<RANDOM_PASS_GOES_IN_HERE>';

PMM_USER_B64="$(echo -n "${PMM_USER}" | base64)";

PMM_PASS_B64="$(echo -n "${PMM_PASS}" | base64)";

# Install the PXC operator

curl -sSf -m 30 https://raw.githubusercontent.com/percona/percona-xtradb-cluster-operator/
K8SPXC-668-CUSTOM/deploy/bundle.yaml \
| kubectl apply -f -

# Install the PSMDB operator

curl -sSf -m 30 https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/
v1.7.0/deploy/bundle.yaml \
| kubectl apply -f -

# Validate that the operators are running

kubectl get pods

40 of 359 Percona LLC, © 2021

 2.2.5 DBaaS (Alpha)

3. Modify your kubeconfig file, if it’s not utilizing the aws-iam-authenticator or client-certificate method
for authentication with Kubernetes. Here are two examples that you can use as templates to modify a
copy of your existing kubeconfig:

• For the aws-iam-authenticator method:

---
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: << CERT_AUTH_DATA >>
server: << K8S_CLUSTER_URL >>
name: << K8S_CLUSTER_NAME >>
contexts:
- context:
cluster: << K8S_CLUSTER_NAME >>
user: << K8S_CLUSTER_USER >>
name: << K8S_CLUSTER_NAME >>
current-context: << K8S_CLUSTER_NAME >>
kind: Config
preferences: {}
users:
- name: << K8S_CLUSTER_USER >>
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
command: aws-iam-authenticator
args:
- "token"
- "-i"
- "<< K8S_CLUSTER_NAME >>"
- --region
- << AWS_REGION >>
env:
- name: AWS_ACCESS_KEY_ID
value: "<< AWS_ACCESS_KEY_ID >>"
- name: AWS_SECRET_ACCESS_KEY
value: "<< AWS_SECRET_ACCESS_KEY >>"

• For the client-certificate method:

---
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: << CERT_AUTH_DATA >>
server: << K8S_CLUSTER_URL >>
name: << K8S_CLUSTER_NAME >>
contexts:
- context:
cluster: << K8S_CLUSTER_NAME >>
user: << K8S_CLUSTER_USER >>
name: << K8S_CLUSTER_NAME >>
current-context: << K8S_CLUSTER_NAME >>
kind: Config
preferences: {}
users:
- name: << K8S_CLUSTER_NAME >>
user:
client-certificate-data: << CLIENT_CERT_DATA >>
client-key-data: << CLIENT_KEY_DATA >>

41 of 359 Percona LLC, © 2021

 2.2.5 DBaaS (Alpha)

4. Follow the instructions for Add a Kubernetes cluster.

If possible, the connection details will show the cluster’s external IP (not possible with minikube).

Install Percona operators on Google GKE

Prerequisites

42 of 359 Percona LLC, © 2021

 2.2.5 DBaaS (Alpha)

You should have an account on GCP https://cloud.google.com/.

43 of 359 Percona LLC, © 2021

 2.2.5 DBaaS (Alpha)

1. Login into google cloud platform console https://console.cloud.google.com/

2. Navigate to Menu → Kubernetes Engine → Clusters

3. Click button Create cluster

4. You can specify cluster option in form or simply click on “My first cluster” and button Create

44 of 359 Percona LLC, © 2021

 2.2.5 DBaaS (Alpha)

5. Wait until cluster created

45 of 359 Percona LLC, © 2021

 2.2.5 DBaaS (Alpha)

6. Click on button Connect in a the cluster’s row

7. Click button Run in Cloud shell

46 of 359 Percona LLC, © 2021

 2.2.5 DBaaS (Alpha)

8. Click Authorize

47 of 359 Percona LLC, © 2021

 2.2.5 DBaaS (Alpha)

9. Set up PXC and PSMDB operators:

curl -sSf -m 30 https://raw.githubusercontent.com/percona/percona-xtradb-cluster-operator/pmm-

branch/deploy/bundle.yaml | kubectl apply -f -
curl -sSf -m 30 https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/pmm-
branch/deploy/bundle.yaml | kubectl apply -f -

48 of 359 Percona LLC, © 2021

 2.2.5 DBaaS (Alpha)

10. Check if it was set up successfully

kubectl api-resources --api-group='psmdb.percona.com'

kubectl api-resources --api-group='pxc.percona.com'

11. Check versions

kubectl api-versions | grep percona.com

49 of 359 Percona LLC, © 2021

 2.2.5 DBaaS (Alpha)

12. Create Service Account, copy and store kubeconfig - output of the following command

cat <<EOF | kubectl apply -f -

---
apiVersion: v1
kind: ServiceAccount
metadata:
name: percona-dbaas-cluster-operator
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: service-account-percona-server-dbaas-xtradb-operator
subjects:
- kind: ServiceAccount
name: percona-dbaas-cluster-operator
roleRef:
kind: Role
name: percona-xtradb-cluster-operator
apiGroup: rbac.authorization.k8s.io
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: service-account-percona-server-dbaas-psmdb-operator
subjects:
- kind: ServiceAccount
name: percona-dbaas-cluster-operator
roleRef:
kind: Role
name: percona-server-mongodb-operator
apiGroup: rbac.authorization.k8s.io
EOF

name=`kubectl get serviceAccounts percona-dbaas-cluster-operator -o json | jq -

r .secrets[].name`
certificate=`kubectl get secret $name -o json | jq -r '.data."ca.crt"'`

50 of 359 Percona LLC, © 2021

 2.2.5 DBaaS (Alpha)

token=`kubectl get secret $name -o json | jq -r '.data.token' | base64 -d`

server=`kubectl cluster-info | grep 'Kubernetes master' | cut -d ' ' -f 6`

echo "
apiVersion: v1
kind: Config
users:
- name: percona-dbaas-cluster-operator
user:
token: $token
clusters:
- cluster:
certificate-authority-data: $certificate
server: $server
name: self-hosted-cluster
contexts:
- context:
cluster: self-hosted-cluster
user: percona-dbaas-cluster-operator
name: svcs-acct-context
current-context: svcs-acct-context
"

51 of 359 Percona LLC, © 2021

 2.2.5 DBaaS (Alpha)

13. Start PMM Server on you local machine or other VM instance:

docker run --detach --name pmm-server --publish 80:80 --publish 443:443 \

--env ENABLE_DBAAS=1 perconalab/pmm-server-fb:PR-1240-07bef94;

14. Login into PMM and navigate to DBaaS

15. Register your GKE using kubeconfig from step 12.

Important Ensure there are no stray new lines in the kubeconfig, especially in long lines like certificate
or token.

52 of 359 Percona LLC, © 2021

 2.2.5 DBaaS (Alpha)

Deleting clusters

You should delete all installation operators as the operators own resources.

If a Public Address is set in PMM Settings, for each DB cluster an API Key is created which can be found
on the page /graph/org/apikeys . You should not delete them (for now, until issue PMM-8045 is fixed) –
once a DB cluster is removed from DBaaS, the related API Key is also removed.

If you only run eksctl delete cluster without cleaning up the cluster first, there will be a lot of orphaned
resources as Cloud Formations, Load Balancers, EC2 instances, Network interfaces, etc.

In the pmm-managed repository, in the deploy directory there are 2 example bash scripts to install and delete
the operators from the EKS cluster.

53 of 359 Percona LLC, © 2021

 2.2.5 DBaaS (Alpha)

The install script:

#!/bin/bash

TOP_DIR=$(git rev-parse --show-toplevel)

PMM_USER="$(echo -n 'admin' | base64)";
PMM_PASS="$(echo -n 'admin_password' | base64)";
KUBECTL_CMD="kubectl --kubeconfig ${HOME}/.kube/config_eks"

# Install the PXC operator

cat ${TOP_DIR}/deploy/pxc_operator.yaml | ${KUBECTL_CMD} apply -f -

# Install the PSMDB operator

cat ${TOP_DIR}/deploy/psmdb_operator.yaml | ${KUBECTL_CMD} apply -f -

The delete script:

#!/bin/bash

TOP_DIR=$(git rev-parse --show-toplevel)

PMM_USER="$(echo -n 'admin' | base64)";
PMM_PASS="$(echo -n 'admin_password' | base64)";
KUBECTL_CMD="kubectl --kubeconfig ${HOME}/.kube/config_eks"

# Delete the PXC operator

cat ${TOP_DIR}/deploy/pxc_operator.yaml | ${KUBECTL_CMD} delete -f -

# Delete the PSMDB operator

cat ${TOP_DIR}/deploy/psmdb_operator.yaml | ${KUBECTL_CMD} delete -f -

(Both scripts are similar except the install script command is apply while in the delete script it is delete .)

After deleting everything in the EKS cluster, run this command (using your own configuration path) and
wait until the output only shows service/kubernetes before deleting the cluster with the eksclt delete
command.

kubectl --kubeconfig ~/.kube/config_eks get all

Example output:

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

service/kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 4d5h

If you don’t need the cluster anymore, you can uninstall everything in it and destroy it:

# Delete all volumes created by the operators:

kubectl [--kubeconfig <config file>] delete pvc --all
# Delete the cluster
eksctl delete cluster --name=your-cluster-name

Run PMM Server as a Docker container for DBaaS

1. Start PMM server from a feature branch:

54 of 359 Percona LLC, © 2021

 2.2.5 DBaaS (Alpha)

docker run --detach --name pmm-server --publish 80:80 --publish 443:443 --env ENABLE_DBAAS=1
percona/pmm-server:2;

Important

• Use --network minikube if running PMM Server and minikube in the same Docker instance. This way
they will share single network and the kubeconfig will work.

• Use Docker variables --env PMM_DEBUG=1 --env PMM_TRACE=1 to see extended debug details.

2. Change the default administrator credentials:

This step is optional, because the same can be done from the web interface of PMM on the first login.

docker exec -t pmm-server bash -c 'ln -s /srv/grafana /usr/share/grafana/data; chown -R

grafana:grafana /usr/share/grafana/data; grafana-cli --homepath /usr/share/grafana admin reset-
admin-password <RANDOM_PASS_GOES_IN_HERE>'

3. Set the public address for PMM Server in PMM settings UI

4. Follow the steps for Add a Kubernetes cluster.

5. Follow the steps for Add a DB Cluster.

6. Get the IP address to connect your app/service:

minikube kubectl get services

Exposing PSMDB and XtraDB clusters for access by external clients

To make services visible externally, you create a LoadBalancer service or manually run commands to
expose ports:

kubectl expose deployment hello-world --type=NodePort.

See also

• DBaaS Dashboard

• Install minikube

• Setting up a Standalone MYSQL Instance on Kubernetes & exposing it using Nginx Ingress Controller

• Use a Service to Access an Application in a Cluster

• Exposing applications using services

55 of 359 Percona LLC, © 2021

 2.3 Client

2.3 Client

2.3.1 Setting up PMM Client

PMM Client is a collection of agents and exporters that run on the host being monitored.

These sections cover the different ways to install PMM Client on a Linux node and register it with PMM
Server. The options are:

• Option 1 – For Debian- or Red Hat-based distributions, install percona-release and use a Linux
package manager ( apt / dnf ) to install PMM Client.

• Option 2 – For Debian- or Red Hat-based distributions, download .deb / .rpm PMM Client packages and
install them

• Option 3 – For other Linux distributions, download and unpack generic PMM Client Linux binaries

• Option 4 – run PMM Client as a Docker container

• Option 5 – run PMM Client with Docker compose

When you have installed PMM Client, you must:

• Register the node with PMM Server

• Configure and add services according to type

Here is an overview of the steps involved for each option.

Before you start

• PMM Server is installed and running with a known IP address accessible from the client node.

• You have superuser (root) access on the client host.

56 of 359 Percona LLC, © 2021

 2.3.1 Setting up PMM Client

• You have superuser access to any database servers that you want to monitor.

• These Linux packages are installed: curl , gnupg , sudo , wget .

• If using Docker, install it.

• System requirements

• Operating system – PMM Client runs on any modern 64-bit Linux distribution. It is tested on
supported versions of Debian, Ubuntu, CentOS, and Red Hat Enterprise Linux.

• Disk – A minimum of 100 MB of storage is required for installing the PMM Client package. With a
good connection to PMM Server, additional storage is not required. However, the client needs to
store any collected data that it cannot dispatch immediately, so additional storage may be required
if the connection is unstable or the throughput is low. (Caching only applies to Query Analytics
data; VictoriaMetrics data is never cached on the client side.)

Install PMM Client with a package manager

Tip If you have used percona-release before, disable and re-enable the repository:

sudo percona-release disable all

sudo percona-release enable original release

INSTALL ON DEBIAN-BASED DISTRIBUTIONS

1. Configure repositories.

wget https://repo.percona.com/apt/percona-release_latest.generic_all.deb
sudo dpkg -i percona-release_latest.generic_all.deb

2. Install the PMM Client package.

sudo apt update

sudo apt install -y pmm2-client

3. Check.

pmm-admin --version

4. Register the node.

INSTALL ON RED HAT-BASED DISTRIBUTIONS

1. Configure repositories.

sudo yum install -y https://repo.percona.com/yum/percona-release-latest.noarch.rpm

2. Install the PMM Client package.

sudo yum install -y pmm2-client

3. Check.

pmm-admin --version

4. Register the node.

57 of 359 Percona LLC, © 2021

 2.3.1 Setting up PMM Client

Download and install PMM Client packages manually

1. Visit the Percona Monitoring and Management 2 download page.

2. Under Version:, select the one you want (usually the latest).

3. Under Software:, select the item matching your software platform.

4. Click to download the package file:

• For Debian, Ubuntu: .deb

• For Red Hat, CentOS, Oracle Linux: .rpm

(Alternatively, copy the link and use wget to download it.)

Here are the download page links for each supported platform.

• Debian 9 (“Stretch”)

• Debian 10 (“Buster”)

• Red Hat/CentOS/Oracle 7

• Red Hat/CentOS/Oracle 8

• Ubuntu 16.04 (“Xenial Xerus”)

• Ubuntu 18.04 (“Bionic Beaver”)

• Ubuntu 20.04 (“Focal Fossa”)

INSTALL ON DEBIAN-BASED DISTRIBUTIONS

sudo dpkg -i *.deb

INSTALL ON RED HAT-BASED DISTRIBUTIONS

sudo dnf localinstall *.rpm

Download and unpack generic Linux binary package

1. Download the PMM Client package:

sudo wget https://downloads.percona.com/downloads/pmm2/2.17.0/binary/tarball/pmm2-

client-2.17.0.tar.gz

2. Download the PMM Client package checksum file:

sudo wget https://downloads.percona.com/downloads/pmm2/2.17.0/binary/tarball/pmm2-

client-2.17.0.tar.gz.sha256sum

3. Verify the download.

sha256sum -c pmm2-client-2.17.0.tar.gz.sha256sum

4. Unpack the package and move into the directory.

sudo tar xfz pmm2-client-2.17.0.tar.gz && cd pmm2-client-2.17.0

5. Run the installer.

58 of 359 Percona LLC, © 2021

 2.3.1 Setting up PMM Client

sudo ./install_tarball

6. Change the path.

PATH=$PATH:/usr/local/percona/pmm2/bin

7. Set up the agent

sudo pmm-agent setup --config-file=/usr/local/percona/pmm2/config/pmm-agent.yaml --server-

address=192.168.1.123 --server-insecure-tls --server-username=admin --server-password=admin

8. Open a new terminal and run the agent.

PATH=$PATH:/usr/local/percona/pmm2/bin pmm-agent --config-file=/usr/local/percona/pmm2/config/

pmm-agent.yaml

9. In the first terminal, check.

pmm-admin status

Remove PMM Client with a package manager

REMOVE ON DEBIAN-BASED DISTRIBUTIONS

1. Uninstall the PMM Client package.

sudo apt remove -y pmm2-client

2. Remove the Percona repository

sudo dpkg -r percona-release

REMOVE ON RED HAT-BASED DISTRIBUTIONS

1. Uninstall the PMM Client package.

sudo yum remove -y pmm2-client

2. Remove the Percona repository

sudo yum remove -y percona-release

59 of 359 Percona LLC, © 2021

 2.3.1 Setting up PMM Client

Run PMM Client as a Docker container

The PMM Client Docker image is a convenient way to run PMM Client as a preconfigured Docker container.

1. Pull the PMM Client docker image.

docker pull \
percona/pmm-client:2

2. Use the image as a template to create a persistent data store that preserves local data when the image is
updated.

docker create \
--volume /srv \
--name pmm-client-data \
percona/pmm-client:2 /bin/true

3. Run the container to start PMM Agent in setup mode. Set X.X.X.X to the IP address of your PMM Server.
(Do not use the docker --detach option as PMM agent only logs to the console.)

PMM_SERVER=X.X.X.X:443
docker run \
--rm \
--name pmm-client \
-e PMM_AGENT_SERVER_ADDRESS=${PMM_SERVER} \
-e PMM_AGENT_SERVER_USERNAME=admin \
-e PMM_AGENT_SERVER_PASSWORD=admin \
-e PMM_AGENT_SERVER_INSECURE_TLS=1 \
-e PMM_AGENT_SETUP=1 \
-e PMM_AGENT_CONFIG_FILE=pmm-agent.yml \
--volumes-from pmm-client-data \
percona/pmm-client:2

4. Check status.

docker exec pmm-client \

pmm-admin status

In the PMM user interface you will also see an increase in the number of monitored nodes.

You can now add services with pmm-admin by prefixing commands with docker exec pmm-client .

Tip

• Adjust host firewall and routing rules to allow Docker communications. (Read more in the FAQ.)

• For help:

docker run --rm percona/pmm-client:2 --help

Remove PMM Client as a Docker container

1. Stop pmm-client container.

docker stop pmm-client

2. Remove containers.

60 of 359 Percona LLC, © 2021

 2.3.1 Setting up PMM Client

docker rm pmm-client

3. Remove the image.

docker rmi $(docker images | grep "percona/pmm-client" | awk {'print $3'})

Run PMM Client with Docker compose

1. Copy and paste this text into a file called docker-compose.yml .

version: '2'
services:
pmm-client:
image: percona/pmm-client:2
hostname: pmm-client-myhost
container_name: pmm-client
restart: always
ports:
- "42000:42000"
- "42001:42001"
logging:
driver: json-file
options:
max-size: "10m"
max-file: "5"
volumes:
- ./pmm-agent.yaml:/etc/pmm-agent.yaml
environment:
- PMM_AGENT_CONFIG_FILE=/etc/pmm-agent.yaml
- PMM_AGENT_SERVER_USERNAME=admin
- PMM_AGENT_SERVER_PASSWORD=admin
- PMM_AGENT_SERVER_ADDRESS=X.X.X.X:443
- PMM_AGENT_SERVER_INSECURE_TLS=true
entrypoint: pmm-agent setup

• Check the values in the environment section match those for your PMM Server. ( X.X.X.X is the IP
address of your PMM Server.)

• Use unique hostnames across all PMM Clients (value for services.pmm-client.hostname ).

2. Ensure a writable agent configuration file.

touch pmm-agent.yaml && chmod 0666 pmm-agent.yaml

3. Run the PMM Agent setup. This will run and stop.

sudo docker-compose up

4. Edit docker-compose.yml , comment out the entrypoint line (insert a # ) and save.

...
# entrypoint: pmm-agent setup

5. Run again, this time with the Docker detach option.

sudo docker-compose up -d

6. Verify.

61 of 359 Percona LLC, © 2021

 2.3.1 Setting up PMM Client

On the command line.

sudo docker exec pmm-client pmm-admin status

In the GUI.

• Select PMM Dashboards → System (Node) → Node Overview

• In the Node Names menu, select the new node

• Change the time range to see data

Caution pmm-agent.yaml contains sensitive credentials and should not be shared.

Register node with PMM Server

Register your node ( X.X.X.X is the IP address of your PMM Server).

pmm-admin config --server-insecure-tls --server-url=https://admin:[email protected]:443

• X.X.X.X is the address of your PMM Server.

• 443 is the default port number.

• admin / admin is the default PMM username and password. This is the same account you use to log into
the PMM user interface, which you had the option to change when first logging in.

Examples

Register on PMM Server with IP address 192.168.33.14 using the default admin/admin username and
password, a node with IP address 192.168.33.23 , type generic , and name mynode .

pmm-admin config --server-insecure-tls --server-url=https://admin:[email protected]:443 192.

168.33.23 generic mynode

Configure and add services

You should continue by adding services according to the service type.

• MySQL and variants (Percona Server for MySQL, Percona XtraDB Cluster, MariaDB)

• MongoDB

• PostgreSQL

• ProxySQL

• Amazon RDS

• Microsoft Azure

• Google Cloud Platform (MySQL and PostgreSQL)

• Linux

• External services

• HAProxy

Note To change the parameters of a previously-added service, remove the service and re-add it with new
parameters.

62 of 359 Percona LLC, © 2021

 2.3.1 Setting up PMM Client

Remove services from monitoring

You should specify service type and service name for removing from monitoring One of next types has to be
set: mysql, mongodb, postgresql, proxysql, haproxy, external

pmm-admin remove <service-type> <service-name>

See also

• Percona release

• PMM Client architecture

• Thanks to https://gist.github.com/paskal for Docker compose files

63 of 359 Percona LLC, © 2021

 2.3.2 MySQL and variants

2.3.2 MySQL and variants

PMM Client collects metrics from MySQL, Percona Server for MySQL, Percona XtraDB Cluster, and MariaDB.
(Amazon RDS is also supported and explained in a separate section.)

This page shows you how to set up PMM to monitor a MySQL or MySQL-based database instance. (You
should read it completely before starting work.)

Here is an overview of the steps involved.

64 of 359 Percona LLC, © 2021

 2.3.2 MySQL and variants

65 of 359 Percona LLC, © 2021

 2.3.2 MySQL and variants

Before you start

Check that:

• PMM Server is installed and running with a known IP address accessible from the client node.

• PMM Client is installed and the node is registered with PMM Server.

• You have superuser (root) access on the client host.

• You have superuser access to any database servers that you want to monitor.

Create a database account for PMM

It is good practice to use a non-superuser account to connect PMM Client to the monitored database
instance. This example creates a database user with name pmm , password pass , and the necessary
permissions.

CREATE USER 'pmm'@'localhost' IDENTIFIED BY 'pass' WITH MAX_USER_CONNECTIONS 10;

GRANT SELECT, PROCESS, SUPER, REPLICATION CLIENT, RELOAD ON *.* TO 'pmm'@'localhost';

Choose and configure a source

Decide which source of metrics to use, and configure your database server for it. The choices are Slow
query log and Performance Schema.

While you can use both at the same time we recommend using only one–there is some overlap in the data
reported, and each incurs a small performance penalty. The choice depends on the version and variant of
your MySQL instance, and how much detail you want to see.

Here are the benefits and drawbacks of Slow query log and Performance Schema metrics sources.

 Benefits  Drawbacks

Slow query log More detail. PMM Client must be on the same host as
the database server or have access to the
slow query log.

Lower resource impact (with query Log files grow and must be actively
sampling feature in Percona Server managed.
for MySQL).

Performance Faster parsing. Less detail.

Schema

Enabled by default on newer

versions of MySQL.

Data source recommendations

Database server Versions Recommended source

MySQL 5.1-5.5 Slow query log

MySQL 5.6+ Performance Schema

MariaDB 10.0+ Performance Schema

Percona Server for MySQL 5.7, 8.0 Slow query log

Percona XtraDB Cluster 5.6, 5.7, 8.0 Slow query log

66 of 359 Percona LLC, © 2021

 2.3.2 MySQL and variants

SLOW QUERY LOG

This section covers how to configure a MySQL-based database server to use the slow query log as a source
of metrics.

Applicable versions

Server Versions

MySQL 5.1-5.5

MariaDB 10.1.2+

Percona Server for MySQL 5.7.10+, 8.0.12+

Percona XtraDB Cluster 5.6, 5.7, 8.0

The slow query log records the details of queries that take more than a certain amount of time to complete.
With the database server configured to write this information to a file rather than a table, PMM Client parses
the file and sends aggregated data to PMM Server via the Query Analytics part of PMM Agent.

Settings

Variable Value Description

slow_query_log ON Enables the slow query log.

                                                       

log_output 'FILE' Ensures the log is sent to a file. (This is

the default on MariaDB.)

long_query_time 0 The slow query threshold in seconds. In

heavily-loaded applications, many quick
queries can affect performance more than
a few slow ones. Setting this value to 0
ensures all queries are captured.

log_slow_admin_statements ON Includes the logging of slow

administrative statements.

log_slow_slave_statements ON Enables logging for queries that have

taken more than long_query_time seconds
to execute on the replica.

Examples

Configuration file

slow_query_log=ON
log_output=FILE
long_query_time=0
log_slow_admin_statements=ON
log_slow_slave_statements=ON

Session

67 of 359 Percona LLC, © 2021

 2.3.2 MySQL and variants

SET GLOBAL slow_query_log = 1;

SET GLOBAL log_output = 'FILE';
SET GLOBAL long_query_time = 0;
SET GLOBAL log_slow_admin_statements = 1;
SET GLOBAL log_slow_slave_statements = 1;

Slow query log – extended

Some MySQL-based database servers support extended slow query log variables.

Applicable versions

Server Versions

Percona Server for MySQL 5.7.10+, 8.0.12+

Percona XtraDB Cluster 5.6, 5.7, 8.0

MariaDB 10.0

68 of 359 Percona LLC, © 2021

 2.3.2 MySQL and variants

Settings

Variable Value Description

log_slow_rate_limit 100 Defines the rate of queries

                                                                       captured by the slow query log.
A good rule of thumb is 100
queries logged per second. For
example, if your Percona Server
instance processes 10,000
queries per second, you should
set log_slow_rate_limit to 100
and capture every 100 th query
for the slow query log.
Depending on the amount of
traffic, logging could become
aggressive and resource
consuming. This variable
throttles the level of intensity of
the data capture without
compromising information.

log_slow_rate_type ‘query’ Set so that it applies to queries,

rather than sessions.

slow_query_log_always_write_time 1 Specifies which queries should

ignore sampling. With query
sampling this ensures that
queries with longer execution
time will always be captured by
the slow query log, avoiding the
possibility that infrequent slow
queries might not get captured
at all.

log_slow_verbosity ‘full’ Ensures that all information

about each captured query is
stored in the slow query log.

slow_query_log_use_global_control ‘all’ Configure the slow query log

during runtime and apply these
settings to existing connections.
(By default, slow query log
settings apply only to new
sessions.)

Examples

Configuration file (Percona Server for MySQL, Percona XtraDB Cluster)

log_slow_rate_limit=100
log_slow_rate_type='query'
slow_query_log_always_write_time=1
log_slow_verbosity='full'
slow_query_log_use_global_control='all'

Configuration file (MariaDB)

69 of 359 Percona LLC, © 2021

 2.3.2 MySQL and variants

log_slow_rate_limit=100

Session (Percona Server for MySQL, Percona XtraDB Cluster)

SET GLOBAL log_slow_rate_limit = 100;

SET GLOBAL log_slow_rate_type = 'query';
SET GLOBAL slow_query_log_always_write_time = 1;
SET GLOBAL log_slow_verbosity = 'full';
SET GLOBAL slow_query_log_use_global_control = 'all';

Slow query log rotation

Slow query log files can grow quickly and must be managed.

When adding a service with the command line use the pmm-admin option --size-slow-logs to set at what
size the slow query log file is rotated. (The size is specified as a number with a suffix. See pmm-admin add
mysql .)

When the limit is reached, PMM Client will:

• remove the previous .old slow log file,

• rename the current file by adding the suffix .old ,

• execute the MySQL command FLUSH LOGS .

Only one .old file is kept. Older ones are deleted.

You can manage log rotation yourself, for example, with logrotate . If you do, you can disable PMM
Client’s log rotation with the --slow-log-rotation=false option when adding a service with pmm-admin add .

PERFORMANCE SCHEMA

This section covers how to configure a MySQL-based database server to use Performance Schema as a
source of metrics.

Applicable versions

Server Versions

Percona Server for MySQL 5.6, 5.7, 8.0

Percona XtraDB Cluster 5.6, 5.7, 8.0

MariaDB 10.3+

PMM’s MySQL Performance Schema Details dashboard charts the various performance_schema metrics.

To use Performance Schema, set these variables.

Variable Value Description

performance_schema ON Enables Performance Schema metrics. This is the

default in MySQL 5.6.6 and higher.

performance-schema-instrument 'statement/ Configures Performance Schema instruments.

%=ON'

performance-schema-consumer- ON Configures the statements-digest consumer.

statements-digest

innodb_monitor_enable all Enables InnoDB metrics counters.

70 of 359 Percona LLC, © 2021

 2.3.2 MySQL and variants

Examples

Configuration file

performance_schema=ON
performance-schema-instrument='statement/%=ON'
performance-schema-consumer-statements-digest=ON
innodb_monitor_enable=all

Session

( performance_schema cannot be set in a session and must be set at server start-up.)

UPDATE performance_schema.setup_consumers
SET ENABLED = 'YES' WHERE NAME LIKE '%statements%';
SET GLOBAL innodb_monitor_enable = all;

MariaDB 10.5.7 or lower

There is no Explain or Example data shown by default in Query Analytics when monitoring MariaDB
instances version 10.5.7 or lower. A workaround is to set this variable.

Variable Value Description

performance_schema.setup_instruments 'statement/%' List of instrumented object classes.

Session

UPDATE performance_schema.setup_instruments SET ENABLED = 'YES', TIMED = 'YES' WHERE NAME LIKE
'statement/%';
UPDATE performance_schema.setup_consumers SET ENABLED = 'YES' WHERE NAME LIKE '%statements%';

Query response time

Query time distribution is a chart in the Details tab of Query Analytics showing the proportion of query time
spent on various activities. It is enabled with the query_response_time_stats variable and associated
plugins.

Applicable versions

Server Versions

Percona Server for MySQL 5.7 (not Percona Server for MySQL 8.0.)

MariaDB 10.0.4

Set this variable to see query time distribution charts.

Variable Value Description

query_response_time_stats ON Report query response time distributions.

                                    (Requires plugin installation. See below.)

Configuration file

query_response_time_stats=ON

71 of 359 Percona LLC, © 2021

 2.3.2 MySQL and variants

You must also install the plugins.

Session

1. Check that /usr/lib/mysql/plugin/query_response_time.so exists.

2. Install the plugins and activate.

For MariaDB 10.3:

INSTALL PLUGIN QUERY_RESPONSE_TIME_AUDIT SONAME 'query_response_time.so';

INSTALL PLUGIN QUERY_RESPONSE_TIME SONAME 'query_response_time.so';
SET GLOBAL query_response_time_stats = ON;

For Percona Server for MySQL 5.7:

INSTALL PLUGIN QUERY_RESPONSE_TIME_AUDIT SONAME 'query_response_time.so';

INSTALL PLUGIN QUERY_RESPONSE_TIME SONAME 'query_response_time.so';
INSTALL PLUGIN QUERY_RESPONSE_TIME_READ SONAME 'query_response_time.so';
INSTALL PLUGIN QUERY_RESPONSE_TIME_WRITE SONAME 'query_response_time.so';
SET GLOBAL query_response_time_stats = ON;

Tablestats

Some table metrics are automatically disabled when the number of tables exceeds a default limit of 1000
tables. This prevents PMM Client from affecting the performance of your database server.

The limit can be changed when adding a service on the command line with the two pmm-admin options:

pmm-admin option Description

--disable-tablestats Disables tablestats collection when the default

                                                                  limit is reached.

--disable-tablestats-limit=N Sets the number of tables ( N ) for which

tablestats collection is disabled. 0 means no
limit. A negative number means tablestats is
completely disabled (for any number of
tables).

User statistics

Applicable versions

User activity, individual table and index access details are shown on the MySQL User Details dashboard
when the userstat variable is set.

Server Versions

Percona Server for MySQL 5.6, 5.7, 8.0

Percona XtraDB Cluster 5.6, 5.7, 8.0

MariaDB 5.2.0+

Examples

Configuration file

userstat=ON

72 of 359 Percona LLC, © 2021

 2.3.2 MySQL and variants

Session

SET GLOBAL userstat = ON;

Add a service

When you have configured your database server, you can add a MySQL service with the user interface or on
the command line.

When adding a service with the command line, you must use the pmm-admin --query-source=SOURCE option
to match the source you’ve chosen and configured the database server for.

With the PMM user interface, you select Use performance schema, or deselect it to use slow query log.

WITH THE USER INTERFACE

1. Select PMM → PMM Add Instance.

2. Select MySQL – Add a remote instance.

73 of 359 Percona LLC, © 2021

 2.3.2 MySQL and variants

3. Enter values for these fields.

74 of 359 Percona LLC, © 2021

 2.3.2 MySQL and variants

Section Field Required Description Default pmm-admin parameter

Main
details

Hostname Hostname --address

or IP
address of
the service

Service Service --name

name name

Port Port for 3306 port in --

accessing address=address[:port]
the service

Username MySQL --username

user name

Password MySQL --password

user
password

Labels

Environment --environment

Region

Availability
zone

Replication --replication-set
set

Cluster --cluster

Custom --custom-labels
labels

Additional
options

Skip --skip-connection-
connection check
check

Use TLS for --tls

database
connections

Skip TLS --tls-skip-verify

certificate
and
hostname
validation

Table
statistics
limit

→ Disabled --disable-tablestats

75 of 359 Percona LLC, © 2021

 2.3.2 MySQL and variants

Section Field Required Description Default pmm-admin parameter

→ Default --disable-tablestats-
limit

→ Custom --disable-tablestats-
limit

Use --perfschema if
performance selected, --slowlog if
schema not.

4. Click Add service.

ON THE COMMAND LINE

Add the database server as a service using one of these example commands. If successful, PMM Client will
print MySQL Service added with the service’s ID and name. Use the --environment and -custom-labels
options to set tags for the service to help identify them.

Examples – TLS connection

sudo pmm-admin add mysql --username=user --password=pass --tls --tls-skip-verify --tls-

ca=pathtoca.pem --tls-cert=pathtocert.pem --tls-key=pathtocertkey.pem --server-url=http://
admin:[email protected] --query-source=perfschema name localhost:3306

Examples – Slow query log

Default query source ( slowlog ), service name ( {node name}-mysql ), and service address/port
( 127.0.0.1:3306 ), with database server account pmm and password pass .

sudo pmm-admin add mysql --username=pmm --password=pass

Slow query log source and log size limit (1 gigabyte), service name ( MYSQL_NODE ) and service address/port
( 191.168.1.123:3306 ).

sudo pmm-admin add mysql --query-source=slowlog --size-slow-logs=1GB --username=pmm --

password=pass MYSQL_NODE 192.168.1.123:3306

Slow query log source, disabled log management (use logrotate or some other log management tool),
service name ( MYSQL_NODE ) and service address/port ( 191.168.1.123:3306 ).

sudo pmm-admin add mysql --query-source=slowlog --size-slow-logs=false --username=pmm --

password=pass MYSQL_NODE 192.168.1.123:3306

Default query source ( slowlog ), service name ( {node}-mysql ), connect via socket.

sudo pmm-admin add mysql --username=pmm --password=pass --socket=/var/run/mysqld/mysqld.sock

Examples – Performance Schema

Performance schema query source, service name ( MYSQL_NODE ) and default service address/port
( 127.0.0.1:3306 )

sudo pmm-admin add mysql --query-source=perfschema --username=pmm --password=pass MYSQL_NODE

76 of 359 Percona LLC, © 2021

 2.3.2 MySQL and variants

Performance schema query source, service name ( MYSQL_NODE ) and default service address/port
( 127.0.0.1:3306 ) specified with flags.

sudo pmm-admin add mysql --query-source=perfschema --username=pmm --password=pass --service-

name=MYSQL_NODE --host=127.0.0.1 --port=3306

Examples – Identifying services

Default query source ( slowlog ), environment labeled test , custom labels setting source to slowlog . (This
example uses positional parameters for service name and service address.)

pmm-admin add mysql --environment=test --custom-labels='source=slowlog' --username=root --

password=password --query-source=slowlog MySQLSlowLog localhost:3306

Check the service

Check service - PMM user interface

1. Select  Configuration→PMM Inventory→Inventory list.

2. Look in the Services tab for a matching Service Type (MySQL), Service name, Addresses, and any other
details entered in the form.

3. Look in the Agents tab to check the desired data source is being used.

Check service - Command line

Look for your service in the output of this command.

pmm-admin inventory list services

Check data

1. Open the MySQL Instance Summary dashboard.

2. Set the Service Name to the newly-added service.

Percona Server for MySQL, MariaDB

If query response time plugin was installed, check for data in the MySQL Query Response Time Details
dashboard or select a query in PMM Query Analytics to see the Query time distribution bar.

Percona XtraDB Cluster

Open the PXC/Galera Cluster Summary dashboard.

See also

• Percona Server for MySQL – Slow Query Log Extended

• Percona Server for MySQL – User Statistics

• MariaDB – Slow Query Log Overview

• MariaDB – Slow Query Log Extended Statistics

• MariaDB – User Statistics

• Percona Blog – PERFORMANCE_SCHEMA vs Slow Query Log

• Percona Blog – MySQL’s INNODB_METRICS table

• Percona Blog – Rotating MySQL Slow Logs Safely

• Percona Blog – Impact of logging on MySQL’s performance

77 of 359 Percona LLC, © 2021

 2.3.3 MongoDB

2.3.3 MongoDB

Configuring MongoDB for Monitoring in PMM Query Analytics

In Query Analytics, you can monitor MongoDB metrics and queries. Run the pmm-admin add command to
use these monitoring services.

Supported versions of MongoDB

Query Analytics supports MongoDB version 3.2 or higher.

Setting Up the Required Permissions

For MongoDB monitoring services to work in Query Analytics, you need to set up the mongodb_exporter
user.

Here is an example for the MongoDB shell that creates and assigns the appropriate roles to the user.

db.createRole({
role: "explainRole",
privileges: [{
resource: {
db: "",
collection: ""
},
actions: [
"listIndexes",
"listCollections",
"dbStats",
"dbHash",
"collStats",
"find"
]
}],
roles:[]
})

db.getSiblingDB("admin").createUser({
user: "mongodb_exporter",
pwd: "s3cR#tpa$$worD",
roles: [
{ role: "explainRole", db: "admin" },
{ role: "clusterMonitor", db: "admin" },
{ role: "read", db: "local" }
]
})

Enabling Profiling

For MongoDB to work correctly with Query Analytics, you need to enable profiling in your mongod
configuration. (Profiling is not enabled by default because it may reduce the performance of your MongoDB
server.)

ENABLING PROFILING ON COMMAND LINE

You can enable profiling from command line when you start the mongod server. This command is useful if
you start mongod manually.

Run this command as root or by using the sudo command

78 of 359 Percona LLC, © 2021

 2.3.3 MongoDB

mongod --dbpath=DATABASEDIR --profile 2 --slowms 200 --rateLimit 100

Note that you need to specify a path to an existing directory that stores database files with the --dpbath .
When the --profile option is set to 2, mongod collects the profiling data for all operations. To decrease the
load, you may consider setting this option to 1 so that the profiling data are only collected for slow
operations.

The --slowms option sets the minimum time for a slow operation. In the given example, any operation
which takes longer than 200 milliseconds is a slow operation.

The --rateLimit option, which is available if you use PSMDB instead of MongoDB, refers to the number of
queries that the MongoDB profiler collects. The lower the rate limit, the less impact on the performance.
However, the accuracy of the collected information decreases as well.

ENABLING PROFILING IN THE CONFIGURATION FILE

If you run mongod as a service, you need to use the configuration file which by default is /etc/mongod.conf .

In this file, you need to locate the operationProfiling: section and add the following settings:

operationProfiling:
slowOpThresholdMs: 200
mode: slowOp

These settings affect mongod in the same way as the command line options. Note that the configuration file
is in the YAML format. In this format the indentation of your lines is important as it defines levels of nesting.

Restart the mongod service to enable the settings.

Run this command as root or by using the sudo command

service mongod restart

Adding MongoDB Service Monitoring

Add monitoring as follows:

pmm-admin add mongodb --username=pmm --password=pmm

where username and password are credentials for the monitored MongoDB access, which will be used
locally on the database host. Additionally, two positional arguments can be appended to the command line
flags: a service name to be used by PMM, and a service address. If not specified, they are substituted
automatically as <node>-mongodb and 127.0.0.1:27017 .

The command line and the output of this command may look as follows:

pmm-admin add mongodb --username=pmm --password=pmm mongo 127.0.0.1:27017

MongoDB Service added.

Service ID : /service_id/f1af8a88-5a95-4bf1-a646-0101f8a20791
Service name: mongo

Beside positional arguments shown above you can specify service name and service address with the
following flags: --service-name , --host (the hostname or IP address of the service), and --port (the port

79 of 359 Percona LLC, © 2021

 2.3.3 MongoDB

number of the service). If both flag and positional argument are present, flag gains higher priority. Here is
the previous example modified to use these flags:

pmm-admin add mongodb --username=pmm --password=pmm --service-name=mongo --host=127.0.0.1 --

port=27017

You can add a MongoDB instance using a UNIX socket with the --socket option:

pmm-admin add mongodb --socket=/tmp/mongodb-27017.sock

If the password contains special symbols like the ‘at’ ( @ ) symbol, the host might not be detected
correctly. Make sure that you insert the password with special characters replaced with their escape
sequences. The simplest way is to use the encodeURIComponent JavaScript function in your browser’s web
console (usually found under Development Tools). Evaluate the function with your password as the
parameter. For example:

encodeURIComponent('$ecRet_pas$w@rd')
"%24ecRet_pas%24w%40rd"

Passing SSL parameters to the MongoDB monitoring service

SSL/TLS related parameters are passed to an SSL enabled MongoDB server as monitoring service
parameters along with the pmm-admin add command when adding the MongoDB monitoring service.

Run this command as root or by using the sudo command

pmm-admin add mongodb --tls

Supported SSL/TLS Parameters

--tls

Enable a TLS connection with mongo server

--tls-skip-verify

Skip TLS certificates validation

--tls-certificate-key-file=PATHTOCERT

Path to TLS certificate file.

--tls-certificate-key-file-password=IFPASSWORDTOCERTISSET

Password for TLS certificate file.

--tls-ca-file=PATHTOCACERT

Path to certificate authority file.

80 of 359 Percona LLC, © 2021

 2.3.4 PostgreSQL

2.3.4 PostgreSQL

PMM Client collects metrics from PostgreSQL and Percona Distribution for PostgreSQL databases.

This page shows how to set up PMM to monitor a PostgreSQL database instance. (Read it completely before
starting work.)

Here is an overview of the steps involved.

81 of 359 Percona LLC, © 2021

 2.3.4 PostgreSQL

82 of 359 Percona LLC, © 2021

 2.3.4 PostgreSQL

Before you start

Check that:

• PMM Server is installed and running with a known IP address accessible from the client node.

• PMM Client is installed and the node is registered with PMM Server.

• You have superuser (root) access on the client host.

• You have superuser access to any database servers that you want to monitor.

(PMM follows PostgreSQL’s end-of-life policy. For specific details on supported platforms and versions, see
Percona’s Software Platform Lifecycle page.)

Create a database account for PMM

We recommend creating a PMM database account that can connect to the postgres database with the
SUPERUSER role.

1. Create a user. This example uses pmm . (Replace ****** with a strong password of your choice.)

CREATE USER pmm WITH SUPERUSER ENCRYPTED PASSWORD '******';

If your database runs on Amazon RDS:

CREATE USER pmm WITH rds_superuser ENCRYPTED PASSWORD '******';

2. PMM must be able to log in locally as this user to the PostgreSQL instance. To enable this, edit the
pg_hba.conf file. If not already enabled by an existing rule, add:

local all pmm md5

# TYPE DATABASE USER ADDRESS METHOD

(Ignore the second line. It is a comment to show field alignment.)

3. Reload the configuration:

sudo su - postgres
psql -c "select pg_reload_conf()"

4. Check local login.

psql postgres pmm -c "\conninfo"

5. Enter the password for the pmm user when prompted.

Choose and configure an extension

Decide which database extension to use, and configure your database server for it. The choices are:

1. pg_stat_statements , the original extension created by PostgreSQL, part of the postgresql-contrib

package available on Linux.

2. pg_stat_monitor is a new extension created by Percona. It is based on and compatible with

pg_stat_statements . pg_stat_monitor has all the features of pg_stat_statements , but adds bucket-based
data aggregation.

83 of 359 Percona LLC, © 2021

 2.3.4 PostgreSQL

We recommend choosing only one of these. If you use both, you will get duplicate metrics.

Important While we recommend use of the newer pg_stat_monitor extension, be aware it is currently in
beta phase and unsupported.

Here are the benefits and drawbacks of each.

 Benefits  Drawbacks

pg_stat_statements Part of official postgresql-contrib No aggregated statistics or

package histograms

pg_stat_monitor Builds on pg_stat_monitor features Beta software

Bucket-based aggregation

About bucket-based data aggregation

pg_stat_monitor collects statistics and aggregates data in a data collection unit called a bucket. These
are linked together to form a bucket chain.

You can specify:

• the number of buckets (the length of the chain)

• how much space is available for all buckets

• a time limit for each bucket’s data collection (the bucket expiry)

When a bucket’s expiration time is reached, accumulated statistics are reset and data is stored in the next
available bucket in > the chain.

When all buckets in the chain have been used, the first bucket is reused and its contents are overwritten.

If a bucket fills before its expiration time is reached, data is discarded.

PG_STAT_STATEMENTS

Install

• Debian/Ubuntu

sudo apt install -y postgresql-contrib

• Red Hat/CentOS

sudo yum install -y postgresql-contrib

84 of 359 Percona LLC, © 2021

 2.3.4 PostgreSQL

Configure

1. Add these lines to your postgresql.conf file:

shared_preload_libraries = 'pg_stat_statements'
track_activity_query_size = 2048 # Increase tracked query string size
pg_stat_statements.track = all # Track all statements including nested
track_io_timing = on # Capture read/write stats

2. Restart the database server.

3. Install the extension.

psql postgres postgres -c "CREATE EXTENSION pg_stat_statements SCHEMA public"

You can now add the service.

PG_STAT_MONITOR

pg_stat_monitor has been tested with:

• PostgreSQL versions 11, 12, 13.

• Percona Distribution for PostgreSQL versions 11, 12, 13.

Important pg_stat_monitor is currently in beta phase and unsupported.

Install

• If you use Percona Distribution for PostgreSQL, you can install the extension with your Linux package
manager. See Installing Percona Distribution for PostgreSQL.

• If you use PostgreSQL you can install by downloading and compiling the source code. See Installing
pg_stat_monitor .

Configure

1. Set or change the value for shared_preload_library .

In your postgresql.conf file:

shared_preload_libraries = 'pg_stat_monitor'

2. Set configuration values.

You can get a list of available settings with SELECT * FROM pg_stat_monitor_settings; .

See pg_stat_monitor GitHub repository for details about available parameters.

3. Set bucket time to 60 seconds

ALTER SYSTEM SET pg_stat_monitor.pgsm_bucket_time=60;

4. Start or restart your PostgreSQL instance.

5. In a psql session:

CREATE EXTENSION pg_stat_monitor;

6. Check the version.

SELECT pg_stat_monitor_version();

85 of 359 Percona LLC, © 2021

 2.3.4 PostgreSQL

Add a service

When you have configured your database server, you can add a PostgreSQL service with the user interface
or on the command line.

WITH THE USER INTERFACE

1. Select  Configuration → PMM Inventory → Add Instance.

2. Select PostgreSQL – Add a remote instance.

3. Enter or select values for the fields.

4. Click Add service.

ON THE COMMAND LINE

Add the database server as a service using one of these example commands. If successful, PMM Client will
print PostgreSQL Service added with the service’s ID and name. Use the --environment and -custom-
labels options to set tags for the service to help identify them.

Examples

Add instance with default node ( <node>-postgresql ) and service name.

pmm-admin add postgresql \

--username=pmm \
--password=password \
--server-url=https://admin:[email protected]:443 \
--server-insecure-tls

• <user name> : The PostgreSQL PMM user

• <password> : The PostgreSQL user credentials.

The service name and service ID will be automatically chosen.

Add instance with specified node and service name.

pmm-admin add postgresql \

--username=pmm \
--password=password \
--server-url=https://admin:[email protected]:443 \
--server-insecure-tls \

Add instance to connect with a UNIX socket.

pmm-admin add postgresql --socket=/var/run/postgresql

See [pmm-admin man page]

Check the service

Check service - PMM user interface

1. Select  Configuration → PMM Inventory → Inventory list

2. Look in the Services tab for a matching Service Type (PostgreSQL), Service name, Addresses, and any
other details entered in the form.

3. Look in the Agents tab to check the desired data source is being used.

Check service - Command line

86 of 359 Percona LLC, © 2021

 2.3.4 PostgreSQL

Look for your service in the output of this command.

pmm-admin inventory list services

If using Docker, use sudo docker exec pmm-client pmm-admin inventory list services

Check data

1. Open the PostgreSQL Instance Summary dashboard.

2. Set the Service Name to the newly-added service.

See also - Configuring Percona Repositories with percona-release

87 of 359 Percona LLC, © 2021

 2.3.5 ProxySQL

2.3.5 ProxySQL

Use the proxysql alias to enable ProxySQL performance metrics monitoring.

USAGE

pmm-admin add proxysql --username=admin --password=admin

where username and password are credentials for the administration interface of the monitored ProxySQL
instance. Additionally, two positional arguments can be appended to the command line flags: a service
name to be used by PMM, and a service address. If not specified, they are substituted automatically as
<node>-proxysql and 127.0.0.1:6032 .

The output of this command may look as follows:

pmm-admin add proxysql --username=admin --password=admin

ProxySQL Service added.

Service ID : /service_id/f69df379-6584-4db5-a896-f35ae8c97573
Service name: ubuntu-proxysql

Beside positional arguments shown above you can specify service name and service address with the
following flags: --service-name , and --host (the hostname or IP address of the service) and --port (the
port number of the service), or --socket (the UNIX socket path). If both flag and positional argument are
present, flag gains higher priority. Here is the previous example modified to use these flags for both host/
port or socket connections:

pmm-admin add proxysql --username=pmm --password=pmm --service-name=my-new-proxysql --host=127.

0.0.1 --port=6032
pmm-admin add proxysql --username=pmm --password=pmm --service-name=my-new-proxysql --socket=/
tmp/proxysql_admin.sock

88 of 359 Percona LLC, © 2021

 2.3.6 Amazon RDS

2.3.6 Amazon RDS

Required settings

It is possible to use PMM for monitoring Amazon RDS (just like any remote MySQL instance). In this case,
the PMM Client is not installed on the host where the database server is deployed. By using the PMM web
interface, you connect to the Amazon RDS DB instance. You only need to provide the IAM user access key
(or assign an IAM role) and PMM discovers the Amazon RDS DB instances available for monitoring.

First of all, ensure that there is the minimal latency between PMM Server and the Amazon RDS instance.

Network connectivity can become an issue for VictoriaMetrics to scrape metrics with 1 second resolution.
We strongly suggest that you run PMM Server on AWS (Amazon Web Services) in the same availability zone
as Amazon RDS instances.

It is crucial that enhanced monitoring be enabled for the Amazon RDS DB instances you intend to monitor.

Set the Enable Enhanced Monitoring option in the settings of your Amazon RDS DB instance.

Creating an IAM user with permission to access Amazon RDS DB instances

It is recommended that you use an IAM user account to access Amazon RDS DB instances instead of using
your AWS account. This measure improves security as the permissions of an IAM user account can be
limited so that this account only grants access to your Amazon RDS DB instances. On the other hand, you
use your AWS account to access all AWS services.

The procedure for creating IAM user accounts is well described in the Amazon RDS documentation. This
section only goes through the essential steps and points out the steps required for using Amazon RDS with
Percona Monitoring and Management.

The first step is to define a policy which will hold all the necessary permissions. Then, you need to associate
this policy with the IAM user or group. In this section, we will create a new user for this purpose.

Creating a policy

A policy defines how AWS services can be accessed. Once defined it can be associated with an existing user
or group.

To define a new policy use the IAM page at AWS.

89 of 359 Percona LLC, © 2021

 2.3.6 Amazon RDS

90 of 359 Percona LLC, © 2021

 2.3.6 Amazon RDS

1. Select the Policies option on the navigation panel and click the Create policy button.
2. On the Create policy page, select the JSON tab and replace the existing contents with the following JSON
document.

{ "Version": "2012-10-17",
"Statement": [{ "Sid": "Stmt1508404837000",
"Effect": "Allow",
"Action": [ "rds:DescribeDBInstances",
"cloudwatch:GetMetricStatistics",
"cloudwatch:ListMetrics"],
"Resource": ["*"] },
{ "Sid": "Stmt1508410723001",
"Effect": "Allow",
"Action": [ "logs:DescribeLogStreams",
"logs:GetLogEvents",
"logs:FilterLogEvents" ],
"Resource": [ "arn:aws:logs:*:*:log-group:RDSOSMetrics:*" ]}
]
}

3. Click Review policy and set a name to your policy, such as AmazonRDSforPMMPolicy . Then, click the Create
policy button.

Creating an IAM user

Policies are attached to existing IAM users or groups. To create a new IAM user, select Users on the Identity
and Access Management page at AWS. Then click Add user and complete the following steps:

91 of 359 Percona LLC, © 2021

 2.3.6 Amazon RDS

1. On the Add user page, set the user name and select the Programmatic access option under Select AWS
access type. Set a custom password and then proceed to permissions by clicking the Permissions button.

2. On the Set permissions page, add the new user to one or more groups if necessary. Then, click Review.

3. On the Add user page, click Create user.

Creating an access key for an IAM user

To discover an Amazon RDS DB instance in PMM, you either need to use the access key and secret access
key of an existing IAM user or an IAM role. To create an access key for use with PMM, open the IAM console
and click Users on the navigation pane. Then, select your IAM user.

To create the access key, open the Security credentials tab and click the Create access key button. The
system automatically generates a new access key ID and a secret access key that you can provide on the
PMM Add Instance dashboard to have your Amazon RDS DB instances discovered.

You may use an IAM role instead of IAM user provided your Amazon RDS DB instances are associated with
the same AWS account as PMM.

In case, the PMM Server and Amazon RDS DB instance were created by using the same AWS account, you
do not need create the access key ID and secret access key manually. PMM retrieves this information
automatically and attempts to discover your Amazon RDS DB instances.

Attaching a policy to an IAM user

The last step before you are ready to create an Amazon RDS DB instance is to attach the policy with the
required permissions to the IAM user.

First, make sure that the Identity and Access Management page is open and open Users. Then, locate and
open the IAM user that you plan to use with Amazon RDS DB instances. Complete the following steps, to
apply the policy:

1. On the Permissions tab, click the Add permissions button.

2. On the Add permissions page, click Attach existing policies directly.

3. Using the Filter, locate the policy with the required permissions (such as AmazonRDSforPMMPolicy ).

4. Select a check-box next to the name of the policy and click Review.

5. The selected policy appears on the Permissions summary page. Click Add permissions.

The AmazonRDSforPMMPolicy is now added to your IAM user.

92 of 359 Percona LLC, © 2021

 2.3.6 Amazon RDS

Setting up the Amazon RDS DB Instance

Query Analytics requires Configuring Performance Schema as the query source, because the slow query log
is stored on the AWS (Amazon Web Services) side, and QAN agent is not able to read it. Enable the
performance_schema option under Parameter Groups in Amazon RDS.

Caution Enabling Performance Schema on T2 instances is not recommended because it can easily run the
T2 instance out of memory.

When adding a monitoring instance for Amazon RDS, specify a unique name to distinguish it from the local
MySQL instance. If you do not specify a name, it will use the client’s host name.

Create the pmm user with the following privileges on the Amazon RDS instance that you want to monitor:

GRANT SELECT, PROCESS, REPLICATION CLIENT ON *.* TO 'pmm'@'%' IDENTIFIED BY 'pass' WITH
MAX_USER_CONNECTIONS 10;
GRANT SELECT, UPDATE, DELETE, DROP ON performance_schema.* TO 'pmm'@'%';

If you have Amazon RDS with a MySQL version prior to 5.5, REPLICATION CLIENT privilege is not available
there and has to be excluded from the above statement.

General system metrics are monitored by using the rds_exporter exporter which replaces node_exporter .
rds_exporter gives access to Amazon CloudWatch metrics.

node_exporter , used in versions of PMM prior to 1.8.0, was not able to monitor general system metrics
remotely.

Adding an Amazon RDS MySQL, Aurora MySQL or Remote Instance

The preferred method of adding an Amazon RDS database instance to PMM is via the 
Configuration→PMM Inventory→Add Instance menu option.

This method supports Amazon RDS database instances that use Amazon Aurora, MySQL, or MariaDB
engines, as well as any remote PostgreSQL, ProxySQL, MySQL and MongoDB instances.

93 of 359 Percona LLC, © 2021

 2.3.6 Amazon RDS

The following steps are needed to add an Amazon RDS database instance to PMM:

94 of 359 Percona LLC, © 2021

 2.3.6 Amazon RDS

1. In the PMM web interface, go to  Configuration→ PMM Inventory→ Add Instance

2. Select AWS RDS MySQL or Aurora MySQL – Add a remote instance.

3. Enter the access key ID and the secret access key of your IAM user.

4. Click the Discover button for PMM to retrieve the available Amazon RDS instances.

5. For the instance that you would like to monitor, select the Start monitoring button.

6. You will see a new page with the number of fields. The list is divided into the following groups: Main
details, RDS database, Labels, and Additional options. Some already known data, such as already entered
AWS access key, are filled in automatically, and some fields are optional.

95 of 359 Percona LLC, © 2021

 2.3.6 Amazon RDS

The Main details section allows you to specify the DNS hostname of your instance, the service name to use
within PMM, the port your service is listening on, and the database user name and password.

The Labels section allows you to specify labels for the environment, the AWS region and availability zone
to be used, the Replication set and Cluster names and also it allows you to set the list of custom labels in
a key:value format.

96 of 359 Percona LLC, © 2021

 2.3.6 Amazon RDS

The Additional options section contains specific flags which allow you to tune the RDS monitoring. They
can allow you to skip connection check, to use TLS for the database connection, not to validate the TLS
certificate and the hostname, as well as to disable basic and/or enhanced metrics collection for the RDS
instance to reduce costs.

Also this section contains a database-specific flag, which would allow Query Analytics for the selected
remote database:

• when adding some remote MySQL, AWS RDS MySQL or Aurora MySQL instance, you will be able to
choose using performance schema for the database monitoring;

• when adding a PostgreSQL instance, you will be able to activate using pg_stat_statements extension;

• when adding a MongoDB instance, you will be able to choose using Query Analytics MongoDB profiler.

7. Finally press the Add service button to start monitoring your instance.

97 of 359 Percona LLC, © 2021

 2.3.6 Amazon RDS

Adding an Amazon RDS PostgreSQL instance

For PostgreSQL, use the same method described above.

98 of 359 Percona LLC, © 2021

 2.3.6 Amazon RDS

1. In the PMM web interface, go to PMM > PMM Add Instance.

2. Select AWS RDS MySQL or Aurora MySQL – Add a remote instance. At the moment of writing this guide,
the Add button doesn’t mention PostgreSQL but the discovery function already supports it.

3. Follow steps 4 to 6 as in the previous section. Fill the form and remember to select PG Stat Statement to
enable Query Analytics. To get queries for Query Analytics, you need to enable pg_stat_statements in your
instance by running:

CREATE EXTENSION pg_stat_statements SCHEMA public;

99 of 359 Percona LLC, © 2021

 2.3.6 Amazon RDS

100 of 359 Percona LLC, © 2021

 2.3.7 Microsoft Azure

2.3.7 Microsoft Azure

Required settings

It is possible to use PMM for monitoring Azure database instances like other remote instances. In this case,
the PMM Client is not installed on the host where the database server is deployed. By using the PMM web
interface, you connect to the Azure DB instance. Discovery is not yet implemented in PMM but it is possible
to add known instances by providing the connection parameters.

First of all, ensure that there is the minimal latency between PMM Server and the Azure instance.

Second, add a firewall rule to enable access from PMM Client like this:

Setting up a MySQL instance

Query Analytics requires you to configure Performance Schema as the query source, because the slow query
log is stored on the Azure side, and QAN agent is not able to read it. Enable the performance_schema option
under Parameter Groups in Amazon RDS.

When adding a monitoring instance for Azure, specify a unique name to distinguish it from the local MySQL
instance. If you do not specify a name, it will use the client’s host name.

Create the pmm user with the following privileges on the Amazon RDS instance that you want to monitor:

GRANT SELECT, PROCESS, REPLICATION CLIENT ON *.* TO 'pmm'@'%' IDENTIFIED BY 'pass' WITH
MAX_USER_CONNECTIONS 10;
GRANT SELECT, UPDATE, DELETE, DROP ON performance_schema.* TO 'pmm'@'%';

2.3.8 Adding an Azure Instance

Follow the instructions for remotes instances explained here.

Example:

101 of 359 Percona LLC, © 2021

 2.3.9 MariaDB.

and be sure to set Performance Schema as the query collection method for Query Analytics.

2.3.9 MariaDB.

MariaDB up to version 10.2 works out of the box but starting with MariaDB 10.3 instrumentation is disabled
by default and cannot be enabled since there is no SUPER role in Azure-MariaDB. So, it is not possible to
run the required queries to enable instrumentation. Monitoring will work but Query Analytics won’t receive
any query data.

2.3.10 PostgreSQL

For PostgreSQL follow the same methods used for MySQL and MariaDB and enable track_io_timing in the
instance configuration to enable Query Analytics.

102 of 359 Percona LLC, © 2021

 2.3.10 PostgreSQL

For Query Analytics, set the server parameter:

pg_stat_statements.track = all

To discover databases on Azure

You need to get the Client ID, Client Secret, Tenant ID and Subscription ID.

You can follow the steps here: https://www.inkoop.io/blog/how-to-get-azure-api-credentials/

Navigate to:

• Create an application in Azure Active directory. https://portal.azure.com/#blade/Microsoft_AAD_IAM/

ActiveDirectoryMenuBlade/Overview

• Get the subscription ID.

• Get the tenant ID.

• Get the client ID.

• Get the client secret.

When you fill in all fields press the Discover button and you will see a list of available databases for
monitoring.

103 of 359 Percona LLC, © 2021

 2.3.10 PostgreSQL

You can monitor 6 types of databases:

• Microsoft.DBforMySQL/servers

• Microsoft.DBforMySQL/flexibleServers

• Microsoft.DBforMariaDB/servers

• Microsoft.DBforPostgreSQL/servers

• Microsoft.DBforPostgreSQL/flexibleServers

• Microsoft.DBforPostgreSQL/serversv2

You can find more details on how to create DB on Azure at:

• https://docs.microsoft.com/en-us/azure/postgresql/

• https://docs.microsoft.com/en-us/azure/mysql/

You will need to set pg_stat_statements.track = all in you PostgreSQL Server settings to use PMM Query
Analytics

In the list of databases on the Discovery page click Start Monitoring to add the selected Azure Database to
PMM.

Fill in all required fields and click Add service.

104 of 359 Percona LLC, © 2021

 2.3.10 PostgreSQL

PMM can use 3 exporters to collect metrics:

• Azure Metrics Exporter – collect “system” metrics related to DB.

• node_cpu_average

• azure_resource_info

• node_filesystem_size_bytes

• azure_memory_percent_average

• azure_storage_percent_average

• azure_storage_used_bytes_average

• node_network_receive_bytes_total

• node_network_transmit_bytes_total

• mysql_exporter or postgres_exporter – to collect database related metrics.

• PMM Agent to collect queries related metrics using pg_stat_statements for PostgreSQL or Performance
Schema for MySQL (MariaDB)

105 of 359 Percona LLC, © 2021

 2.3.11 Google Cloud Platform

2.3.11 Google Cloud Platform

PMM can monitor MySQL or PostgreSQL instances hosted on the Google Cloud Platform.

The connection can be direct, or indirect using Cloud SQL Proxy.

MySQL

1. Set up a MySQL instance on Google Cloud

2. The database server must be accessible by PMM Client. If PMM Client is not also hosted on GCP, you will
need to add a network interface with a public interface.

3. Configure Performance Schema on the MySQL server. Using the GCP console’s Cloud Shell or your own
gcloud installation, run:

gcloud sql instances patch <instance_name> --database-flags performance_schema=on

4. Log into the PMM user interface.

5. Select PMM → PMM Add Instance

6. Click MySQL Add a remote instance

7. Fill in the details for the remote MySQL instance.

• Ensure Use performance schema is selected

8. Click Add service

9. Check for values in the MySQL Instance Overview dashboard and in Query Analytics

PostgreSQL

1. Set up a PostgreSQL instance on Google Cloud

2. The database server must be accessible by PMM Client. If PMM Client is not also hosted on GCP, you will
need to add a network interface with a public interface.

3. Configure pg_stat_statements . Open an interactive SQL session with your GCP PostgreSQL server and
run:

CREATE EXTENSION pg_stat_statements;

4. Log into the PMM user interface.

5. Select PMM → PMM Add Instance

6. Select PostgreSQL Add a remote instance

7. Fill in the details for the remote PostgreSQL instance:

• In Stat tracking options, select PG Stat Statements

8. Click Add service

9. Check for values in the PostgreSQL Instance Overview dashboard and Query Analytics

Cloud SQL Proxy

MYSQL

1. Create instance on GCP

2. Note connection as <project_id>:<zone>:<db_instance_name>

106 of 359 Percona LLC, © 2021

 2.3.11 Google Cloud Platform

3. Enable Admin API and download the JSON credential file

4. Enable Performance Schema

5. Run Cloud SQL Proxy (runs on PMM Client node)

• As a Docker container:

docker run -d \
-v ~/path/to/admin-api-file.json:/config \
-p 127.0.0.1:3306:3306 \
gcr.io/cloudsql-docker/gce-proxy:1.19.1 \
/cloud_sql_proxy \
-instances=example-project-NNNN:us-central1:mysql-for-pmm=tcp:0.0.0.0:3306 \
-credential_file=/config

• On Linux:

wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy

chmod +x cloud_sql_proxy
./cloud_sql_proxy -instances=example-project-NNNN:us-central1:mysql-for-pmm=tcp:3306 \
-credential_file=/path/to/credential-file.json

6. Add instance

pmm-admin add mysql --host=127.0.0.1 --port=3306 \

--username=root --password=secret \
--service-name=MySQLGCP --query-source=perfschema

POSTGRESQL

1. Create instance on GCP

2. Note connection as <project_id>:<zone>:<db_instance_name>

3. Enable Admin API and download the JSON credential file

4. Run Cloud SQL Proxy

./cloud_sql_proxy -instances=example-project-NNNN:us-central1:pg-for-pmm=tcp:5432 \
-credential_file=/path/to/credential-file.json

5. Log into PostgreSQL

6. Load extension:

CREATE EXTENSION pg_stat_statements;

7. Add service:

pmm-admin add postgresql --host=127.0.0.1 --port=5432 \

--username="postgres" --password=secret --service-name=PGGCP

107 of 359 Percona LLC, © 2021

 2.3.12 Linux

2.3.12 Linux

Adding general system metrics service

PMM collects Linux metrics automatically starting from the moment when you have configured your node
for monitoring with pmm-admin config .

108 of 359 Percona LLC, © 2021

 2.3.13 External Services

2.3.13 External Services

Adding general external services

You can collect metrics from an external (custom) exporter on a node when:

• there is already a PMM Agent instance running and,

• this node has been configured using the pmm-admin config command.

USAGE

pmm-admin add external --service-name=<service-name> --listen-port=<listen-port> --metrics-

path=<metrics-path> --scheme=<scheme>

pmm-admin add external-serverless --external-name=<external-service-name> --host=<hostname> --

listen-port=<listen-port> --metrics-path=<metrics-path> --scheme=<scheme>

Getting data from external exporters

There two ways to get metrics from other exporters:

• external will collect metrics from the exporter that is run on the same host as PMM Client’s connection
to it by a port. (See more details with pmm-admin add external --help .)

• external-serverless is useful for collecting metrics from cloud services. You need a host and port
number to add it to PMM Server. (See more details with pmm-admin add external-serverless --help .)

Here are the differences between external and external-serverless types.

Connection schema of external exporter:

Connection schema of external serverless exporter:

How I can add something not supported by PMM

PMM can collect any metrics in Open metrics or Prometheus exposition format. You must specify the host
and port of these metrics using the pmm-admin add external or pmm-admin add external-serverless
commands.

109 of 359 Percona LLC, © 2021

 2.3.13 External Services

From this point, PMM will collect and store available metrics.

To browse and visualize collected metrics as a first step, we can look at the Advanced Data Exploration
dashboard and select informative services and metrics.

Another way is to create a new Grafana Dashboard to PMM as needed.

One more way is to search for an already created dashboard at https://grafana.com/grafana/dashboards

for the added exporter and import it into PMM.

THIRD-PARTY EXPORTERS

You can find more exporters on the official Prometheus page.

CUSTOM EXPORTER

You can write a custom external exporter or extend your application to expose metrics in Prometheus
format.

Please see more details here: https://prometheus.io/docs/instrumenting/writing_exporters/

EXAMPLES

root@mysql1:~# pmm-admin add external --group=processes --listen-port=9256

External Service added.
Service ID : /service_id/6485f4fd-745b-4dfb-8b72-328e300f8b50
Service name: mysql1-processes
Group : processes

• Add an exporter running on local port 9256 to the group called processes .

• Use the group and host names to automatically generate a service name.

• Use the default scheme and metrics path.

Adding an External service via UI

1. In the PMM web interface, go to PMM > PMM Add Instance.

2. Select External Service – Add a remote instance.

110 of 359 Percona LLC, © 2021

 2.3.13 External Services

3. Fill the form and set the external service endpoint.

The endpoint can be set manually:

111 of 359 Percona LLC, © 2021

 2.3.13 External Services

or by parsing required data from a URL string, in which case you only need to pass a valid URL.

112 of 359 Percona LLC, © 2021

 2.3.14 HAProxy

2.3.14 HAProxy

Adding HAProxy services

You can collect metrics from HAProxy on a node when:

• there is already configured haproxy

• How to configure HAProxy https://www.haproxy.com/blog/haproxy-exposes-a-prometheus-metrics-

endpoint

• After HAProxy is running (default address http://localhost:8404/metrics) you can add it to PMM.

• Use the haproxy alias to enable HAProxy metrics monitoring.

• there is already a PMM Agent instance running

• this node has been configured using the pmm-admin config command.

USAGE

pmm-admin add haproxy --listen-port=8404

where listen-port is the port number where HAProxy running. (This is the only required flag.)

The output of this command should look as follows:

HAProxy Service added.

Service ID : /service_id/c481183f-70a2-443f-91e5-cae5cecd06a2
Service name: Ubuntu-haproxy

Additionally, one positional argument can be appended to the command line flags: a service name to be
used by PMM. If not specified, they are substituted automatically as <node>-haproxy .

During adding here is connection check (can be skipped by flag --skip-connection-check ). If HAProxy
doesn’t run properly on the given port then you will see an error message:

Connection check failed: Get "http://127.0.0.1:8404/metrics": dial tcp 127.0.0.1:8404: connect:

connection refused.

Beside positional argument shown above you can specify service name with the following flags: --username
, --password , --metrics-path (path for scraping metrics, default: /metrics) and --scheme (http or https).
Here are some examples:

pmm-admin add haproxy --listen-port=8404 --username=pmm --password=pmm new-haproxy

pmm-admin add haproxy --listen-port=8404 --metrics-path=/prom-metrics --scheme=https

Here you can check list of all available flags: pmm-admin

You can also add HAProxy by UI in Grafana. Add instance.

HAProxy data is visible in the Advanced Data Exploration dashboard:

113 of 359 Percona LLC, © 2021

 2.3.14 HAProxy

114 of 359 Percona LLC, © 2021

 3. Using

3. Using

3.1 Using
• User Interface

• Using the web-based user interface

• Finding dashboards

• Rendering dashboard images

• Viewing graph details

• Annotating events

• Integrated alerting

• Query Analytics, a specialized dashboard for detailed query analysis

• Percona Platform

• Security Threat Tool: Enabling and seeing the results of database security checks

115 of 359 Percona LLC, © 2021

 3.2 User Interface

3.2 User Interface

PMM’s user interface is a browser application based on Grafana. This page explains how to log in, how the
user interface is laid out, and what the controls do.

3.2.1 Logging in

1. Start a web browser and in the address bar enter the server name or IP address of the PMM server host.

2. The page loads showing the PMM login screen.

3. Enter the username and password given to you by your system administrator. The defaults are:

• Username: admin

• Password: admin

4. Click Log in

5. If this is your first time logging in, you’ll be asked to set a new password. (We recommend you do.) Enter
a new password in both fields and click Submit. You can click Skip to carry on with the default password.

6. The PMM Home dashboard loads.

116 of 359 Percona LLC, © 2021

 3.2.2 Dashboards

3.2.2 Dashboards

The interface is a collection of web pages called dashboards.

Dashboards are grouped into folders. You can customize these, by renaming them or creating new ones.

The area inside dashboards is populated by panels. Some are in collapsible panel groups. A panel can show
a value, a graph, a chart, or a visual representation of a set.

3.2.3 Controls

These menus and controls appear on all dashboards:

1. Main menu (also Grafana menu, side menu)

2. Navigation bar

3. View controls

4. View selectors (dynamic contents)

5. Shortcut menu (dynamic contents)

(For details see UI Components.)

117 of 359 Percona LLC, © 2021

 3.2.4 Navigation

3.2.4 Navigation

Search for a dashboard by name

There are two ways to open the dashboard search page. (Each takes you to the same search screen.)

• Click the  icon in the main menu.

• Click the dashboard name in the navigation bar (top row, to the right of the  icon). (To search within
the current folder, click the folder name instead of the dashboard name.)

118 of 359 Percona LLC, © 2021

 3.2.4 Navigation

1. Click Search dashboards by name and begin typing any part of the dashboard name (in this example,
“Instances”).

2. Click one of the search results to go to that dashboard. Change the search text to refine the list.

3. To abandon the search, click the  icon at the end of the search bar.

Open a dashboard with the menu

In the main menu, the PMM Dashboards icon reveals a submenu containing links to all PMM dashboards
grouped by service type. (This menu will eventually replace the shortcut menu which has links to commonly-
used dashboards.

119 of 359 Percona LLC, © 2021

 3.2.5 Panels

3.2.5 Panels

Charts, graphs and set-based panels reveal extra information when the mouse is moved over them.

Some panels have an information icon  in the top left corner. Mouse over this to reveal panel information.

Panel menu

At the top of each panel and to the right of the panel name is the panel menu.

Tip The menu is hidden until you mouse over it. Look for the  symbol in the title bar of a panel.

Item Description

 View Open the panel in full window mode

 Share Render the panel’s image for sharing

 Explore Run PromQL queries

 Inspect See the panel’s data or definition

 More (Only charts and graphs) Additional options

VIEW

The View menu items opens panels in full-window mode. This is useful for graphs with several metrics.

Exit a panel’s full window mode by pressing Escape or clicking the left arrow  next to the dashboard
name.

See also

• How to render dashboard images

• How to annotate special events

120 of 359 Percona LLC, © 2021

 3.3 Integrated Alerting

3.3 Integrated Alerting

Integrated Alerting lets you know when certain system events occur.

Caution Integrated Alerting is a technical preview and is subject to change.

To activate Integrated Alerting, select PMM→PMM Settings→Advanced Settings, turn on Integrated

Alerting and click Apply changes.

3.3.1 Definitions

• Alerts are generated when their criteria (alert rules) are met; an alert is the result of an alert rule
expression evaluating to true.

• Alert rules are based on alert rule templates. We provide a default set of templates. You can also create
your own.

PMM’s Integrated Alerting is a customized and separate instance of the Prometheus Alertmanager, and
distinct from Grafana’s alerting functionality.

3.3.2 Prerequisites

Set up a communication channel: When the Communication tab appears, select it. Enter details for Email or
Slack. (Read more)

3.3.3 Open the Integrated Alerting page

• From the left menu, select  Alerting,  Integrated Alerting

The Alerting menu also lists  Alert Rules and  Notification Channels. These are for Grafana’s alerting
functionality.

121 of 359 Percona LLC, © 2021

 3.3.3 Open the Integrated Alerting page

This page has four tabs.

122 of 359 Percona LLC, © 2021

 3.3.3 Open the Integrated Alerting page

1. Alerts: Shows alerts (if any).

2. Alert Rules: Shows rule definitions.

3. Alert Rule Templates: Lists rule templates.

123 of 359 Percona LLC, © 2021

 3.3.4 Add a Notification Channel

4. Notification Channels: Lists notification channels.

3.3.4 Add a Notification Channel

1. On the Integrated Alerting page, go to the Notification Channels tab.

2. Click  Add.

3. Fill in the details:

124 of 359 Percona LLC, © 2021

 3.3.5 Add an Alert Rule

• Name

• Type

• Email:

• Addresses

• Pager Duty

• Routing key

• Service key

• Slack

• Channel

4. Click Add to add the notification channel, or Cancel to abort the operation.

3.3.5 Add an Alert Rule

1. On the Integrated Alerting page, go to the Alert Rules tab.

2. Click  Add.

3. Fill in the details

125 of 359 Percona LLC, © 2021

 3.3.6 Add an Alert Rule Template

• Template

• Name

• Threshold

• Duration(s)

• Severity

• Filters

• Channels

• Activate

4. Click Add to add the alert rule, or Cancel to abort the operation.

3.3.6 Add an Alert Rule Template

1. On the Integrated Alerting page, go to the Alert Rule Templates tab.

2. Click  Add.

3. Enter a template in the Alert Rule Template text box.

---
templates:
- name: mysql_too_many_connections
version: 1
summary: MySQL connections in use
tiers: [anonymous, registered]
expr: |-
max_over_time(mysql_global_status_threads_connected[5m]) / ignoring (job)

126 of 359 Percona LLC, © 2021

 3.3.6 Add an Alert Rule Template

mysql_global_variables_max_connections
* 100
> [[ .threshold ]]
params:
- name: threshold
summary: A percentage from configured maximum
unit: '%'
type: float
range: [0, 100]
value: 80
for: 5m
severity: warning
labels:
foo: bar
annotations:
description: |-
More than [[ .threshold ]]% of MySQL connections are in use on
{{ $labels.instance }}
VALUE = {{ $value }}
LABELS: {{ $labels }}
summary: MySQL too many connections (instance {{ $labels.instance }})

127 of 359 Percona LLC, © 2021

 3.3.7 Video

Alert Rule Template parameters

The parameters used in the template follow a format and might include different fields depending on
their type :

• name (required): the name of the parameter. Spaces and special characters not allowed.

• summary (required): a short description of what this parameter represents.

• type (required): PMM currently supports the float type. (More will be available in the future, such
as string or bool .)

• unit (optional): PMM currently supports either s (seconds) or % (percentage).

• value (optional): the parameter value itself.

• range (optional): only for float parameters, defining the boundaries for the value.

Restrictions

• Value strings must not include any of these special characters: < > ! @ # $ % ^ & * ( )
_ / \ ' + - = (space)

• Any variables must be predefined.

4. Click Add to add the alert rule template, or Cancel to abort the operation.

3.3.7 Video

This short (3:36) video shows how to activate and configure Integrated Alerting.

Your browser does not support playing this video.

128 of 359 Percona LLC, © 2021

 3.4 Query Analytics

3.4 Query Analytics

The Query Analytics dashboard shows how queries are executed and where they spend their time. It helps
you analyze database queries over time, optimize database performance, and find and remedy the source
of problems.

Query Analytics supports MySQL, MongoDB and PostgreSQL. The minimum requirements for MySQL are:

• MySQL 5.1 or later (if using the slow query log)

• MySQL 5.6.9 or later (if using Performance Schema)

Query Analytics displays metrics in both visual and numeric form. Performance-related characteristics
appear as plotted graphics with summaries.

The dashboard contains three panels:

• Filters Panel

• Overview Panel

• Details Panel

Query Analytics data retrieval is not instantaneous and can be delayed due to network conditions. In such
situations no data is reported and a gap appears in the sparkline.

129 of 359 Percona LLC, © 2021

 3.4.1 Filters Panel

3.4.1 Filters Panel

• The Filter panel occupies the left side of the dashboard. It lists filters, grouped by category. Selecting
one reduces the Overview list to those items matching the filter.

• The first five of each category are shown. If there are more, the list is expanded by clicking Show all
beside the category name, and collapsed again with Show top 5.

• Applying a filter may make other filters inapplicable. These become grayed out and inactive.

• Click the chart symbol  to navigate directly to an item’s associated dashboard.

• Separately, the global Time range setting filters results by time, either your choice of Absolute time
range, or one of the predefined Relative time ranges.

130 of 359 Percona LLC, © 2021

 3.4.2 Overview Panel

3.4.2 Overview Panel

To the right of the Filters panel and occupying the upper part of the dashboard is the Overview panel.

Each row of the table represents the metrics for a chosen object type, one of:

• Query

• Service Name

• Database

• Schema

• User Name

• Client Host

At the top of the second column is the dimension menu. Use this to choose the object type.

131 of 359 Percona LLC, © 2021

 3.4.2 Overview Panel

On the right side of the dimension column is the Dimension Search bar.

Enter a string and press Enter to limit the view to queries containing only the specified keywords.

Delete the search text and press Enter to see the full list again.

Columns

• The first column is the object’s identifier. For Query, it is the query’s Fingerprint.

• The second column is the Main metric, containing a reduced graphical representation of the metric over
time, called a sparkline, and a horizontal meter, filled to reflect a percentage of the total value.

• Additional values are revealed as mouse-over tool-tips.

Tool-tips

• For the Query dimension, hovering over the information icon  reveals the query ID and its example.

• Hovering on a column header reveals an informative tool-tip for that column.

• Hovering on the main metric sparkline highlights the data point and a tooltip shows the data value
under the cursor.

132 of 359 Percona LLC, © 2021

 3.4.2 Overview Panel

133 of 359 Percona LLC, © 2021

 3.4.2 Overview Panel

• Hovering on the main metric meter reveals the percentage of the total, and other details specific to the
main metric.

• Hovering on column values reveals more details on the value. The contents depends on the type of
value.

134 of 359 Percona LLC, © 2021

 3.4.3 Details Panel

Adding and removing columns

• Metrics columns are added with the Add column button.

• When clicked, a text field and list of available metrics are revealed. Select a metric or enter a search
string to reduce the list. Selecting a metric adds it to the panel.

• A metric column is removed by clicking on the column heading and selecting Remove column.

• The value plotted in the main metric column can be changed by clicking a metric column heading and
selecting Swap with main metric.

Sorting

• The entire list is sorted by one of the columns.

• Click either the up or down caret to sort the list by that column’s ascending or descending values.

Pagination

• The pagination device lets you move forwards or backwards through pages, jump to a specific page,
and choose how many items are listed per page.

• Queries are grouped into pages of 25, 50 or 100 items.

3.4.3 Details Panel

• Selecting an item in the Overview panel opens the Details panel with a Details Tab.

• If the dimension is Query, the panel also contains the Examples Tab, Explain Tab, and Tables Tab.

Details Tab

The Details tab contains a Query time distribution bar (only for MySQL databases) and a set of Metrics in
collapsible subpanels.

135 of 359 Percona LLC, © 2021

 3.4.3 Details Panel

• The Query time distribution bar shows a query’s total time made up of colored segments, each
segment representing the proportion of time spent on one of the follow named activities:

• query_time : Statement execution time.

• lock_time : Time to acquire locks.

• blk_read_time : Total time the statement spent reading blocks (if track_io_timing is enabled,
otherwise zero).

• blk_write_time : Total time the statement spent writing blocks (if track_io_timing is enabled,
otherwise zero).

• innodb_io_r_wait : Time for InnoDB to read the data from storage.

• innodb_queue_wait : Time the query spent either waiting to enter the InnoDB queue, or in it pending
execution.

• innodb_rec_lock_wait : Time the query waited for row locks.

• other : Remaining uncategorized query time.

• Metrics is a table with these headings:

• Metric: The Metric name, with a question-mark tool-tip that reveals a description of the metric on
mouse-over.

• Rate/Second: A sparkline chart of real-time values per unit time.

• Sum: A summation of the metric for the selected query, and the percentage of the total.

• Per Query Stats: The value of the metric per query.

• Each row in the table is a metric. The contents depends on the chosen dimension.

Examples Tab

(For Query dimension.)

The Examples tab shows an example of the selected query’s fingerprint or table element.

Explain Tab

(For Query dimension.)

136 of 359 Percona LLC, © 2021

 3.4.4 Query Analytics for MongoDB

The Explain tab shows the explain output for the selected query, in Classic or JSON formats:

• MySQL: Classic and JSON

• MongoDB: JSON only

• PostgreSQL: Not supported

Tables Tab

(For Query dimension.)

The Tables tab shows information on the tables and indexes involved in the selected query.

3.4.4 Query Analytics for MongoDB

MongoDB is conceptually different from relational database management systems, such as MySQL and
MariaDB.

Relational database management systems store data in tables that represent single entities. Complex
objects are represented by linking tables.

137 of 359 Percona LLC, © 2021

 3.4.4 Query Analytics for MongoDB

In contrast, MongoDB uses the concept of a document where all essential information for a complex object
is stored in one place.

Query Analytics can monitor MongoDB queries. Although MongoDB is not a relational database management
system, you analyze its databases and collections in the same interface using the same tools.

138 of 359 Percona LLC, © 2021

 3.5 Percona Platform

3.5 Percona Platform

3.5.1 About Percona Platform

Percona Platform provides value-added services to PMM.

The services comprise:

• Security Threat Tool

• DBaaS (Alpha)

139 of 359 Percona LLC, © 2021

 3.5.2 Security Threat Tool

3.5.2 Security Threat Tool

The Security Threat Tool runs regular checks against connected databases, alerting you if any servers pose
a potential security threat.

All checks run on the PMM Client side. Results are sent to PMM Server where a summary count is shown on
the Home Dashboard, with details in the PMM Database Checks dashboard.

Checks are automatically downloaded from Percona Platform and run every 24 hours. (This period is not
configurable.)

Check results data always remains on the PMM Server. It is not related to anonymous data sent for
Telemetry purposes.

The Failed security checks panel on the Home Dashboard shows the number of failed checks classed as
critical (red), major (amber), and trivial (blue).

Key

Critical / Major / Trivial

Details are in the PMM Database Checks dashboard (select PMM→PMM Database Checks).

How to enable

The Security Threat Tool (STT) is disabled by default. Enable it in PMM Settings→Advanced Settings.

140 of 359 Percona LLC, © 2021

 3.5.2 Security Threat Tool

Enabling STT in the settings also causes the PMM server to download STT checks from Percona Platform
and run them once. This operation runs in the background so even though the settings update finishes
instantly it might take some time for the checks to complete download and execution and the results (if any)
to be visible in the PMM Database Checks dashboard.

Change a check’s interval

1. Navigate to PMM→PMM Database Checks

2. Activate the All Checks tab

3. In the Actions column for a chosen check, click the  Interval icon

4. Chose an interval from the options

5. Click Save

List of checks made

Check ID Description

mongodb_auth MongoDB authentication is disabled

mongodb_version MongoDB/PSMDB version is not the latest

mysql_anonymous_users There are accounts with no username

mysql_empty_password There are users without passwords

mysql_version MySQL/PS/MariaDB version is not the latest

postgresql_super_role PostgreSQL has users (besides postgres , rdsadmin , and pmm_user ) with the role
‘SUPER’

postgresql_version PostgreSQL version is not the latest

141 of 359 Percona LLC, © 2021

 3.5.3 DBaaS Dashboard (Alpha)

3.5.3 DBaaS Dashboard (Alpha)

Caution DBaaS functionality is currently in Alpha status.

The information on this page is subject to change and may be inaccurate.

The DBaaS dashboard is where you add, remove, and operate on Kubernetes and database clusters.

Activate a DBaaS feature

To activate the DBaaS feature, select  PMM → PMM Settings → Advanced settings. Then turn on the
DBaaS feature by clicking on  toggle in the Technical preview features section of the page.

To open the DBaaS dashboard:

• From the left menu, select  DBaaS.

Kubernetes clusters

ADD A KUBERNETES CLUSTER

1. Click Register new Kubernetes Cluster

2. Enter values for the Kubernetes Cluster Name and Kubeconfig file in the corresponding fields.

142 of 359 Percona LLC, © 2021

 3.5.3 DBaaS Dashboard (Alpha)

3. Click Register.

4. A message will momentarily display telling you whether the registration was successful or not.

UNREGISTER A KUBERNETES CLUSTER

You can’t unregister a Kubernetes cluster if there DB clusters associated with it.

1. Click Unregister.

2. Confirm the action by clicking Proceed, or abandon by clicking Cancel.

VIEW A KUBERNETES CLUSTER’S CONFIGURATION

1. Find the row with the Kubernetes cluster you want to see.

2. In the Actions column, open the  menu and click Show configuration.

143 of 359 Percona LLC, © 2021

 3.5.3 DBaaS Dashboard (Alpha)

MANAGE ALLOWED COMPONENT VERSIONS

Administrators can select allowed and default versions of components versions for each cluster.

144 of 359 Percona LLC, © 2021

 3.5.3 DBaaS Dashboard (Alpha)

1. Find the row with the Kubernetes cluster you want to manage.
2. In the Actions column, open the  menu and click Manage versions.

3. Select an Operator and Component from the drop-down menus.

145 of 359 Percona LLC, © 2021

 3.5.3 DBaaS Dashboard (Alpha)

4. Activate or deactivate allowed versions, and select a default in the Default menu.

5. Click Save.

KUBERNETES OPERATOR STATUS

The Kubernetes Cluster tab shows the status of operators.

DB clusters

ADD A DB CLUSTER

You must create at least one Kubernetes cluster to create a DB cluster.

146 of 359 Percona LLC, © 2021

 3.5.3 DBaaS Dashboard (Alpha)

To monitor a DB cluster, set up a public address for PMM Server first.

147 of 359 Percona LLC, © 2021

 3.5.3 DBaaS Dashboard (Alpha)

1. Select the DB Cluster tab.

2. Click Create DB Cluster.

3. In section 1, Basic Options:

a. Enter a value for Cluster name that complies with domain naming rules.

b. Select a cluster from the Kubernetes Cluster menu.

c. Select a database type from the Database Type menu.

148 of 359 Percona LLC, © 2021

 3.5.3 DBaaS Dashboard (Alpha)

4. Expand section 2, Advanced Options.

a. Select Topology, either Cluster or Single Node.

b. Select the number of nodes. (The lower limit is 3.)

c. Select a preset for Resources per Node.

Small, Medium and Large are fixed preset values for Memory, CPU, and Disk.

Values for the Custom preset can be edited.

Beside each resource type is an estimate of the required and available resources represented
numerically in absolute and percentage values, and graphically as a colored, segmented bar showing
the projected ratio of used to available resources. A red warning triangle  is shown if the requested
resources exceed those available.

5. When both Basic Options and Advanced Options section icons are green, the Create Cluster button
becomes active. (If inactive, check the values for fields in sections whose icon is red.)

Click Create Cluster to create your cluster.

6. A row appears with information on your cluster:

149 of 359 Percona LLC, © 2021

 3.5.3 DBaaS Dashboard (Alpha)

• Name: The cluster name

• Database type: The cluster database type

• Connection:

• Host: The hostname

• Port: The port number

• Username: The connection username

• Password: The connection password (click the eye icon  to reveal)

• DB Cluster Parameters:

• K8s cluster name: The Kubernetes cluster name

• CPU: The number of CPUs allocated to the cluster

• Memory: The amount of memory allocated to the cluster

• Disk: The amount of disk space allocated to the cluster

• Cluster Status:

• PENDING: The cluster is being created

• ACTIVE: The cluster is active

• FAILED: The cluster could not be created

• DELETING: The cluster is being deleted

DELETE A DB CLUSTER

1. Find the row with the database cluster you want to delete.

2. In the Actions column, open the  menu and click Delete.

3. Confirm the action by clicking Proceed, or abandon by clicking Cancel.

150 of 359 Percona LLC, © 2021

 3.5.3 DBaaS Dashboard (Alpha)

Important Deleting a cluster in this way also deletes any attached volumes.

EDIT A DB CLUSTER

1. Select the DB Cluster tab.

2. Find the row with the database cluster you want to change.

3. In the Actions column, open the  menu and click Edit.

A paused cluster can’t be edited.

RESTART A DB CLUSTER

1. Select the DB Cluster tab.

2. Identify the database cluster to be changed.

3. In the Actions column, open the  menu and click Restart.

SUSPEND OR RESUME A DB CLUSTER

1. Select the DB Cluster tab.

2. Identify the DB cluster to suspend or resume.

151 of 359 Percona LLC, © 2021

 3.5.3 DBaaS Dashboard (Alpha)

3. In the Actions column, open the  menu and click the required action:
• For active clusters, click Suspend.

• For paused clusters, click Resume.

See also Setting up a development environment for DBaaS

152 of 359 Percona LLC, © 2021

 4. How to

4. How to

4.1 How to
• Configure via the PMM Settings page

• Upgrade PMM Server via the user interface

• Secure your PMM installation

• Optimize the performance of your PMM installation

• Annotate charts to mark significant events

• Render dashboard images to save or share

• Troubleshoot:

• Integrated Alerting

153 of 359 Percona LLC, © 2021

 4.2 Configure

4.2 Configure
The Settings page is where you configure PMM.

Open the Settings page from the main menu with  Configuration→Settings. The page opens with the
Metrics Resolution settings tab selected.

154 of 359 Percona LLC, © 2021

 4.2.1 Metrics resolution

On the left are the selector tabs:

• Configure

• Metrics resolution

• Advanced Settings

• Data Retention

• Telemetry

• Check for updates

• Security Threat Tool

• Public address

• DBaaS

• Integrated Alerting

• Microsoft Azure Monitoring

• Public Address

• SSH Key

• Alertmanager integration

• Percona Platform

• Login

• Sign up

• Password Reset

• Password Forgotten

• Change Password after Login

• Communication

• Email

• Slack

Important Click Apply changes to save any changes made here.

Diagnostics

On all tabs is a Diagnostics section (top-right). Click Download server diagnostics to retrieve PMM
diagnostics data which can be examined and/or shared with our support team should you need help.

4.2.1 Metrics resolution

Metrics are collected at three intervals representing low, medium and high resolutions.

The Metrics Resolution settings tab contains a radio button with three fixed presets (Rare, Standard and
Frequent) and one editable custom preset (Custom).

155 of 359 Percona LLC, © 2021

 4.2.1 Metrics resolution

Each preset is a group of low, medium and high resolutions. The values are in seconds.

Short time intervals are high resolution metrics. Longer time intervals are low resolution. So:

• A low resolution interval increases the time between collection, resulting in low-resolution metrics and
lower disk usage.

• A high resolution interval decreases the time between collection, resulting in high-resolution metrics
and higher disk usage.

The default values (in seconds) for the fixed presets and their resolution names are:

Editable? Preset Low Medium High

No Rare 300 180 60

No Standard 60 10 5

No Frequent 30 5 1

Yes Custom 60 10 5
(defaults)

Values for the Custom preset can be entered as values, or changed with the arrows.

If there is poor network connectivity between PMM Server and PMM Client, or between PMM Client and the
database server it is monitoring, scraping every second may not be possible when the network latency is
greater than 1 second.

156 of 359 Percona LLC, © 2021

 4.2.2 Advanced Settings

4.2.2 Advanced Settings

Data Retention

Data retention specifies how long data is stored by PMM Server. By default, time-series data is stored for 30
days. You can adjust the data retention time to balance your system’s available disk space with your
metrics history requirements.

Telemetry

The Telemetry switch enables gathering and sending basic anonymous data to Percona, which helps us to
determine where to focus the development and what is the uptake of the various versions of PMM.
Specifically, gathering this information helps determine if we need to release patches to legacy versions
beyond support, determining when supporting a particular version is no longer necessary, and even
understanding how the frequency of release encourages or deters adoption.

Currently, only the following information is gathered:

• PMM Version,

• Installation Method (Docker, AMI, OVF),

• the Server Uptime,

• Security Threat Tool Status (enabled or disabled),

• Integrated Alerting Status (enabled or disabled).

We do not gather anything that would make the system identifiable, but the following two things are to be
mentioned:

1. The Country Code is evaluated from the submitting IP address before it is discarded.

2. We do create an “instance ID” - a random string generated using UUID v4. This instance ID is generated
to distinguish new instances from existing ones, for figuring out instance upgrades.

157 of 359 Percona LLC, © 2021

 4.2.2 Advanced Settings

The first telemetry reporting of a new PMM Server instance is delayed by 24 hours to allow sufficient time to
disable the service for those that do not wish to share any information.

There is a landing page for this service, available at check.percona.com, which clearly explains what this
service is, what it’s collecting, and how you can turn it off.

Grafana’s anonymous usage statistics is not managed by PMM. To activate it, you must change the PMM
Server container configuration after each update.

As well as via the PMM Settings page, you can also disable telemetry with the -e DISABLE_TELEMETRY=1
option in your docker run statement for the PMM Server.

• If the Security Threat Tool is enabled in PMM Settings, Telemetry is automatically enabled.

• Telemetry is sent immediately; the 24 hour grace period is not honored.

Check for updates

When active, PMM will automatically check for updates and put a notification in the home page Updates
dashboard if any are available.

Security Threat Tool

The Security Threat Tool performs a range of security-related checks on a registered instance and reports
the findings. It is off by default.

To see the results of checks, select PMM Database Checks to open the Security Checks/Failed Checks
dashboard, and select the Failed Checks tab.

Checks are re-fetched and re-run at intervals. There are three named intervals:

Interval name Value (hours)

Rare interval 78

Standard interval (default) 24

Frequent interval 4

The values for each named interval are fixed.

158 of 359 Percona LLC, © 2021

 4.2.3 Public address

Checks use the Standard interval by default. To change a check’s interval:

• Go to PMM Database Checks

• Select All Checks

• In the Actions column, select the  icon

• Select an interval and click Save

(Read more at Security Threat Tool.)

4.2.3 Public address

The address or hostname PMM Server will be accessible at. Click Get from browser to have your browser
detect and populate this field automatically.

DBaaS

Caution DBaaS functionality is a technical preview that must be turned on with a server feature flag. See
DBaaS.

Enables/disables DBaaS features on this server.

Important Deactivating DBaaS does not suspend or remove running DB clusters.

Integrated Alerting

Enables Integrated Alerting and reveals the Communication tab.

Microsoft Azure Monitoring

Caution This is a technical preview feature.

159 of 359 Percona LLC, © 2021

 4.2.4 SSH Key

Activates Microsoft Azure monitoring.

Public Address

Public address for accessing DBaaS features on this server.

4.2.4 SSH Key

This section lets you upload your public SSH key to access the PMM Server via SSH (for example, when
accessing PMM Server as a virtual appliance).

Enter your public key in the SSH Key field and click Apply SSH Key.

4.2.5 Alertmanager integration

Alertmanager manages alerts, de-duplicating, grouping, and routing them to the appropriate receiver or
display component.

This section lets you configure integration of VictoriaMetrics with an external Alertmanager.

• The Alertmanager URL field should contain the URL of the Alertmanager which would serve your PMM
alerts.

• The Prometheus Alerting rules field is used to specify alerting rules in the YAML configuration format.

160 of 359 Percona LLC, © 2021

 4.2.6 Percona Platform

Fill both fields and click the Apply Alertmanager settings button to proceed.

4.2.6 Percona Platform

This panel is where you create, and log into and out of your Percona Platform account.

Login

161 of 359 Percona LLC, © 2021

 4.2.6 Percona Platform

If you have a Percona Platform account, enter your credentials and click Login.

Click Sign out to log out of your Percona Platform account.

Sign up

To create a Percona Platform account:

1. Click Sign up

2. Enter a valid email address in the Email field

3. Choose and enter a strong password in the Password field

4. Select the check box acknowledging our terms of service and privacy policy

5. Click Sign up

A brief message will confirm the creation of your new account and you may now log in with these
credentials.

Your Percona Platform account is separate from your PMM User account.

Password Reset

PASSWORD FORGOTTEN

In case you forgot your password, click on the Forgot password link in the login page.

You will be redirected to a password reset page. Enter the email you are registered with in the field and click
on Reset via Email.

162 of 359 Percona LLC, © 2021

 4.2.6 Percona Platform

An email with a link to reset your password will be sent to you.

CHANGE PASSWORD AFTER LOGIN

If you did not forget your password but you still want to change it, go to https://okta.percona.com/
enduser/settings (make sure you are logged in).

163 of 359 Percona LLC, © 2021

 4.2.7 Communication

Insert you current password and the new password in the form to the bottom right of the page. If you
cannot see the form, you will need to click on the Edit Profile green button (you will be prompted for you
password).

Click on Change Password. If everything goes well, you will see a confirmation message.

4.2.7 Communication

Global communications settings for Integrated Alerting.

If there is no Communication tab, go to the Advanced Settings tab and activate Integrated Alerting.

(Integrated Alerting uses a separate instance of Alertmanager run by pmm-managed .)

Email

Settings for the SMTP email server:

• Server Address: The default SMTP smarthost used for sending emails, including port number.

• Hello: The default hostname to identify to the SMTP server.

• From: The sender’s email address.

• Auth type: Authentication type. Choose from:

• None

• Plain

• Login

• CRAM-MD5

• Username: SMTP Auth using CRAM-MD5, LOGIN and PLAIN.

• Password: SMTP Auth using CRAM-MD5, LOGIN and PLAIN.

164 of 359 Percona LLC, © 2021

 4.2.7 Communication

Slack

Settings for Slack notifications:

• URL: The Slack webhook URL to use for Slack notifications.

See also

Prometheus Alertmanager configuration

165 of 359 Percona LLC, © 2021

 4.3 Upgrade

4.3 Upgrade
Upgrade PMM Server before upgrading PMM Clients.

4.3.1 Updating a Server

Client and server components are installed and updated separately.

PMM Server can run natively, as a Docker image, a virtual appliance, or an AWS cloud instance. Each has
its own installation and update steps.

The preferred and simplest way to update PMM Server is with the PMM Upgrade panel on the Home page.

The panel shows:

• the current server version and release date;

• whether the server is up to date;

• the last time a check was made for updates.

Click the refresh button to manually check for updates.

If one is available, click the update button to update to the version indicated.

4.3.2 Upgrade from PMM 1

Because of the significant architectural changes between PMM1 and PMM2, there is no direct upgrade path.
The approach to making the switch from PMM version 1 to 2 is a gradual transition, outlined in this blog
post.

In short, it involves first standing up a new PMM2 server on a new host and connecting clients to it. As new
data is reported to the PMM2 server, old metrics will age out during the course of the retention period (30
days, by default), at which point you’ll be able to shut down your existing PMM1 server.

Any alerts configured through the Grafana UI will have to be recreated due to the target dashboard id’s not
matching between PMM1 and PMM2. In this instance we recommend moving to Alertmanager recipes in
PMM2 for alerting which, for the time being, requires a separate Alertmanager instance. However, we are
working on integrating this natively into PMM2 Server and expect to support your existing Alertmanager
rules.

166 of 359 Percona LLC, © 2021

 4.4 Secure

4.4 Secure
You can improve the security of your PMM installation with:

• SSL encryption to secure traffic between client and server;

• Grafana HTTPS secure cookies

To see which security features are enabled:

pmm-admin status

Tip You can gain an extra level of security by keeping PMM Server isolated from the internet, if possible.

4.4.1 SSL encryption

You need valid SSL certificates to encrypt traffic between client and server.

With our Docker, OVF and AMI images, self-signed certificates are in /srv/nginx .

To use your own, you can either:

• mount the local certificate directory to the same location, or,

• copy your certificates to a running PMM Server container.

Mounting certificates

For example, if your own certificates are in /etc/pmm-certs :

docker run -d -p 443:443 --volumes-from pmm-data \

--name pmm-server -v /etc/pmm-certs:/srv/nginx \
--restart always percona/pmm-server:2

• The certificates must be owned by root. You can do this with: sudo chown 0:0 /etc/pmm-certs/*

• The mounted certificate directory ( /etc/pmm-certs in this example) must contain the files
certificate.crt , certificate.key , ca-certs.pem and dhparam.pem .

• For SSL encryption, the container must publish on port 443 instead of 80.

Copying certificates

If PMM Server is running as a Docker image, use docker cp to copy certificates. This example copies
certificate files from the current working directory to a running PMM Server docker container.

docker cp certificate.crt pmm-server:/srv/nginx/certificate.crt

docker cp certificate.key pmm-server:/srv/nginx/certificate.key
docker cp ca-certs.pem pmm-server:/srv/nginx/ca-certs.pem
docker cp dhparam.pem pmm-server:/srv/nginx/dhparam.pem

Enabling SSL when connecting PMM Client to PMM Server

pmm-admin config --server-url=https://<user>:<password>@<server IP>

167 of 359 Percona LLC, © 2021

 4.4.2 Grafana HTTPS secure cookies

4.4.2 Grafana HTTPS secure cookies

To enable:

1. Start a shell within the Docker container: docker exec -it pmm-server bash

2. Edit /etc/grafana/grafana.ini

3. Enable cookie_secure and set the value to true

4. Restart Grafana: supervisorctl restart grafana

168 of 359 Percona LLC, © 2021

 4.5 Optimize

4.5 Optimize
4.5.1 Improving PMM Performance with Table Statistics Options

If a MySQL instance has a lot of schemas or tables, there are two options to help improve the performance
of PMM when adding instances with pmm-admin add : --disable-tablestats and --disable-tablestats-
limit .

• These settings can only be used when adding an instance. To change them, you must remove and re-
add the instances.

• You can only use one of these options when adding an instance.

4.5.2 Disable per-table statistics for an instance

When adding an instance with pmm-admin add , the --disable-tablestats option disables table statistics
collection when there are more than the default number (1000) of tables in the instance.

USAGE

sudo pmm-admin add mysql --disable-tablestats

4.5.3 Change the number of tables beyond which per-table statistics is disabled

When adding an instance with pmm-admin add , the --disable-tablestats-limit option changes the number
of tables (from the default of 1000) beyond which per-table statistics collection is disabled.

USAGE

sudo pmm-admin add mysql --disable-tablestats-limit=<LIMIT>

EXAMPLE

Add a MySQL instance, disabling per-table statistics collection when the number of tables in the instance
reaches 2000.

sudo pmm-admin add mysql --disable-tablestats-limit=2000

169 of 359 Percona LLC, © 2021

 4.6 Annotate

4.6 Annotate
Annotations mark a moment in time. They are useful for marking system changes or other significant
application events. They can be set globally or for specific nodes or services.

You create them on the command line with the pmm-admin annotate command.

Annotations show as a vertical dashed line on a dashboard graph. Reveal the annotation text by mousing
over the caret indicator below the line.

You turn annotations on or off with the PMM Annotations switch in the second row menu bar.

170 of 359 Percona LLC, © 2021

 4.7 Backup

4.7 Backup
1. Navigate to PMM Settings→Advanced Settings

2. Activate Backup Management

3. Click Apply changes

4. In the left menu bar, click →Backup

5. Click Add

6. Fill in the form fields

171 of 359 Percona LLC, © 2021

 4.7 Backup

• Name:

• Description:

• Type:

• S3:

• Local Client:

• Local Server:

• Endpoint:

• Bucket Name:

• Access Key:

• Secret Key:

7. Click Add to add the location or Test to test the connection.

172 of 359 Percona LLC, © 2021

 4.8 Render dashboard images

4.8 Render dashboard images

At the moment, PMM Server can’t render dashboard images exported by Grafana without these steps.

Part 1: Install dependencies

1. Connect to your PMM Server Docker container.

docker exec -it pmm-server bash

2. Install Grafana plugins.

grafana-cli plugins install grafana-image-renderer

3. Restart Grafana.

supervisorctl restart grafana

4. Install libraries.

yum install -y libXcomposite libXdamage libXtst cups libXScrnSaver pango \

atk adwaita-cursor-theme adwaita-icon-theme at at-spi2-atk at-spi2-core \
cairo-gobject colord-libs dconf desktop-file-utils ed emacs-filesystem \
gdk-pixbuf2 glib-networking gnutls gsettings-desktop-schemas \
gtk-update-icon-cache gtk3 hicolor-icon-theme jasper-libs json-glib \
libappindicator-gtk3 libdbusmenu libdbusmenu-gtk3 libepoxy \
liberation-fonts liberation-narrow-fonts liberation-sans-fonts \
liberation-serif-fonts libgusb libindicator-gtk3 libmodman libproxy \
libsoup libwayland-cursor libwayland-egl libxkbcommon m4 mailx nettle \
patch psmisc redhat-lsb-core redhat-lsb-submod-security rest spax time \
trousers xdg-utils xkeyboard-config alsa-lib

173 of 359 Percona LLC, © 2021

 4.8 Render dashboard images

Part 2 - Share the image

1. Navigate to the dashboard you want to share.

2. Open the panel menu.

3. Select Share to reveal the Share Panel.

4. Click Direct link rendered image.

5. A new browser tab opens. Wait for the image to be rendered then use your browser’s image save function
to download the image.

If the necessary plugins are not installed, a message in the Share Panel will say so.

174 of 359 Percona LLC, © 2021

 4.9 Troubleshoot

4.9 Troubleshoot
4.9.1 Update

If PMM server wasn’t updated properly, or if you have concerns about the release, you can force the update
process in 2 ways:

1. From the UI - Home panel: click with the Alt key on the reload icon in the Update panel (IMG needed) to
make the Update Button visible even if you are on the same version as available for update. Pressing this
button will force the system to rerun the update so that any broken or not installed components can be
installed. In this case, you’ll go through the usual update process with update logs and successful
messages at the end.

2. By API call (if UI not available): You can call the Update API directly with:

curl --user admin:admin --request POST 'http://PMM_SERVER/v1/Updates/Start'

Replace admin:admin with your username/password, and replace PMM_SERVER with your server address.

You will not see the logs using this method.

Refresh The Home page in 2-5 min and you should see that PMM was updated.

4.9.2 PMM Server/PMM Client connection

Broken network connectivity may be due to many reasons. Particularly, when using Docker, the container is
constrained by the host-level routing and firewall rules. For example, your hosting provider might have
default iptables rules on their hosts that block communication between PMM Server and PMM Client,
resulting in DOWN targets in VictoriaMetrics. If this happens, check the firewall and routing settings on the
Docker host.

PMM is also able to generate diagnostics data which can be examined and/or shared with our support team
to help quickly solve an issue. You can get collected logs from PMM Client using the pmm-admin summary
command.

Logs obtained in this way includes PMM Client logs and logs which were received from the PMM Server,
stored separately in the client and server folders. The server folder also contains its own client
subfolder with the self-monitoring client information collected on the PMM Server.

Beginning with PMM version 2.4.0, there is an additional flag that enables the fetching of pprof debug
profiles and adds them to the diagnostics data. To enable, run pmm-admin summary --pprof .

You can get PMM Server logs in two ways:

• In a browser, visit https://<address-of-your-pmm-server>/logs.zip .

• Go to PMM > PMM Settings and click Download server diagnostics. (See Diagnostics in PMM Settings.)

4.9.3 Integrated Alerting

No  Integrated Alerting icon

You are not logged in as a privileged user. You need either Admin or Editor roles to work with Integrated
Alerting.

175 of 359 Percona LLC, © 2021

 4.9.3 Integrated Alerting

 Integrated Alerting icon but no submenu

Integrated Alerting isn’t activated.

1. Go to PMM → PMM Settings → Advanced Settings

2. Enable Integrated Alerting

Unreachable external IP addresses

When I get an email or page from my system the IP is not reachable from outside my organization how
do I fix this?

You can configure your PMM Server’s Public Address by navigating to PMM → PMM Settings → Advanced
Settings, and supply an address to use in your alert notifications.

What is ‘Alertmanager integration’?

There’s already an Alertmanager integration tab without me turning it on, I know because I was using
your existing Alertmanager integration.

This will continue to work but will be renamed External Alertmanager.

Notification channels not working

I tried to setup a Slack/Email channel but nothing happened

Before you can use a notification channel you must provide your connection details.

1. Go to PMM → PMM Settings→ Communication

2. Define your SMTP server or Slack incoming webhook URL

For PagerDuty you can configure in the notification channel tab of Integrated Alerting by supplying your
server/routing key.

What’s the difference: Username/Password vs Identity/Secret

In configuring my email server I’m being asked for a Username and Password as well as Identity and
Secret. What is the difference between these and which do I use or do I need both?

It depends on what kind of authentication your system uses:

• LOGIN: Use Username/Password

• PLAIN: Use either Username or Identity and Password

• CRAM-MD5: Use Username and Secret

Alert Rule Templates is disabled

Built-In alerts are not editable.

However, you can copy them and edit the copies. (PMM >=2.14.0).

If you create a custom alert rule template you will have access to edit.

176 of 359 Percona LLC, © 2021

 4.9.3 Integrated Alerting

Creating rules

I’m ready to create my first rule! I’ve chosen a template and given it a name…what is the format of the
fields?

• Threshold - float value, it has different meanings depending on what template is used

• Duration - The duration the condition must be satisfied in seconds

• Filters - A Key, Evaluator, and Value. E.g. service_name=ps5.7

• Key must be an exact match. You can find a complete list of keys by using the Explore main
menu item in PMM

• Evaluator can be any of: = , =~

• Value is an exact match or when used with a ‘fuzzy’ evaluator (=~) can be a regular expression.
E.g. service_name=~ps.*

Variables in Templates

The concept of “template” implies things like variable substitutions…where can I use these? Where can I
find a complete list of them?

Here is a guide to creating templates for Alertmanager: https://prometheus.io/docs/prometheus/latest/

configuration/template_examples/

177 of 359 Percona LLC, © 2021

 5. Details

5. Details

5.1 Details
• Architecture: high-level architecture and main components.

• User interface components: Descriptions of the main menus and icons

• Dashboards reference: A complete list of dashboards by category, with screenshots

• Commands:

• pmm-admin: The manual page for the PMM administration tool

• pmm-agent: The manual page for the PMM Client agent program

• API: How to access the Swagger API

• VictoriaMetrics: the third-party monitoring solution and time-series database that replaced Prometheus
in PMM 2.12.0

• Glossary: A list of obscure terms and definitions

178 of 359 Percona LLC, © 2021

 5.2 Architecture

5.2 Architecture
PMM works on the client/server principle, where a single server instance communicates with one or more
clients.

Except when monitoring AWS RDS instances, a PMM Client must be running on the host to be monitored.

5.2.1 PMM context

The PMM Client package provides:

• Exporters for each database and service type. When an exporter runs, it connects to the database or
service instance, runs the metrics collection routines, and sends the results to PMM Server.

• pmm-agent : Run as a daemon process, it starts and stops exporters when instructed.

• vmagent : A VictoriaMetrics daemon process that sends metrics data (pushes) to PMM Server.

The PMM Server package provides:

• pmm-managed

• Query Analytics

• Grafana

• VictoriaMetrics

179 of 359 Percona LLC, © 2021

 5.2.2 PMM Server

5.2.2 PMM Server

PMM Server includes the following tools:

• Query Analytics (QAN) enables you to analyze MySQL query performance over periods of time. In
addition to the client-side QAN agent, it includes the following:

• QAN API is the back-end for storing and accessing query data collected by the QAN agent running
on a PMM Client.

• QAN Web App is a web application for visualizing collected Query Analytics data.

• Metrics Monitor provides a historical view of metrics that are critical to a MySQL or MongoDB server
instance. It includes the following:

• VictoriaMetrics, a scalable time-series database. (Replaced Prometheus in PMM 2.12.0.)

• ClickHouse is a third-party column-oriented database that facilitates the Query Analytics

functionality.

• Grafana is a third-party dashboard and graph builder for visualizing data aggregated (by
VictoriaMetrics or Prometheus) in an intuitive web interface.

• Percona Dashboards is a set of dashboards for Grafana developed by us.

180 of 359 Percona LLC, © 2021

 5.2.2 PMM Server

PMM Client

The PMM Client package consist of the following:

• pmm-admin is a command-line tool for managing PMM Client, for example, adding and removing
database instances that you want to monitor. (Read more.).

• pmm-agent is a client-side component a minimal command-line interface, which is a central entry point
in charge for bringing the client functionality: it carries on client’s authentication, gets the client
configuration stored on the PMM Server, manages exporters and other agents.

• node_exporter is an exporter that collects general system metrics.

• mysqld_exporter is an exporter that collects MySQL server metrics.

• mongodb_exporter is an exporter that collects MongoDB server metrics.

• postgres_exporter is an exporter that collects PostgreSQL performance metrics.

• proxysql_exporter is an exporter that collects ProxySQL performance metrics.

• rds_exporter is an exporter that collects Amazon RDS performance metrics.

• azure_database_exporter is an exporter that collects Azure database performance metrics.

To make data transfer from PMM Client to PMM Server secure, all exporters are able to use SSL/TLS
encrypted connections, and their communication with the PMM server is protected by the HTTP basic
authentication.

181 of 359 Percona LLC, © 2021

 5.3 UI components

5.3 UI components
1. Main menu (also Grafana menu, side menu)

2. Navigation bar

3. View controls

4. View selectors (dynamic contents)

5. Shortcut menu (dynamic contents)

182 of 359 Percona LLC, © 2021

 5.3.1 Main menu

5.3.1 Main menu

The main menu is part of the Grafana framework and is visible on every page.

Item (Top) Name Description

Home Link to home dashboard

 Search Search dashboards by name

 Create Create dashboards or folders, import dashboards

 Dashboards Manage dashboards, create playlists, manage snapshots

PMM Dashboards Replacement for shortcut menu

 Explore Run queries with PromQL

 Alerting Alerting, Integrated Alerting, Alert Rules, Notification

Channels

 Configuration

 Server Admin

 Backup Management Backup management and storage location configuration

PMM Database
Checks

 DBaaS

The DBaaS icon appears only if a server feature flag has been set.

The Backup Management icon appears when Backup Management is activated in PMM Settings→Advanced
Settings.

Icon (Bottom) Description

(Profile icon) User menu

 Help

Navigation bar

Item (left) Description

 (Display only)

(Name) / (Optional) Folder name

(Name) Dashboard name

 Mark as favorite

 Share dashboard

183 of 359 Percona LLC, © 2021

 5.3.1 Main menu

View controls

Item (right) Description

 Dashboard settings

 Cycle view mode

 (time range) Time range selector

 Time range zoom out

 Refresh dashboard

(Time interval) Refresh period

View selectors

This menu bar is context sensitive; it changes according to the page you are on. (With wide menus on small
screens, items may wrap to the next row.)

Item Description

Interval Data interval

Region Filter by region

Environment Filter by environment

Cluster Filter by cluster

Replication Set Filter by replication set

Node Name Filter by node name

Service Name Filter by service name

PMM Annotations View annotations

184 of 359 Percona LLC, © 2021

 5.3.1 Main menu

Shortcut menu

This menu contains shortcuts to other dashboards. The list changes according to the page you’re on.

Important This menu will be removed in future releases. Its function will be replaced by the PMM
Dashboards main menu entry.

Item Description

 Home Home dashboard

 Query Analytics Query Analytics

 Compare Nodes compare

(Service Type) Service type menu (see below)

 HA HA dashboards

 Services Services menu

 PMM PMM menu

The Compare menu links to the Instances Overview dashboard for the current service type.

Services menu

The Services menu choice determines the Service Type menu.

Menu Item Service type Description

menu

Services

MongoDB Instances Overview  MongoDB MongoDB dashboards

MySQL Instances Overview  MySQL MySQL dashboards

Nodes Overview  OS OS dashboards

PostgreSQL Instances  PostgreSQL PostgreSQL

Overview dashboards

PMM menu

This item lists shortcuts to utility pages.

Menu Item

PMM

PMM Add Instance

PMM Database Checks

PMM Inventory

PMM Settings

185 of 359 Percona LLC, © 2021

 5.4 Dashboards

5.4 Dashboards

5.4.1 Dashboards

Insight

• Advanced Data Exploration

• Home Dashboard

• Prometheus Exporter Status

• Prometheus Exporters Overview

• VictoriaMetrics

• VictoriaMetrics Agents Overview

PMM

• PMM Inventory

OS Dashboards

• CPU Utilization Details

• Disk Details

• Network Details

• Memory Details

• Node Temperature Details

• Nodes Compare

• Nodes Overview

• Node Summary

• NUMA Details

• Processes Details

Prometheus Dashboards

• Prometheus Exporter Status

• Prometheus Exporters Overview

MySQL Dashboards

• MySQL Amazon Aurora Details

• MySQL Command/Handler Counters Compare

• MySQL InnoDB Compression Details

• MySQL InnoDB Details

• MySQL MyISAM/Aria Details

• MySQL MyRocks Details

• MySQL Instance Summary

• MySQL Instances Compare

186 of 359 Percona LLC, © 2021

 5.4.1 Dashboards

• MySQL Instances Overview

• MySQL Wait Event Analyses Details

• MySQL Performance Schema Details

• MySQL Query Response Time Details

• MySQL Replication Summary

• MySQL Group Replication Summary

• MySQL Table Details

• MySQL User Details

• MySQL TokuDB Details

MongoDB Dashboards

• MongoDB Cluster Summary

• MongoDB Instance Summary

• MongoDB Instances Overview

• MongoDB Instances Compare

• MongoDB ReplSet Summary

• MongoDB InMemory Details

• MongoDB MMAPv1 Details

• MongoDB WiredTiger Details

PostgreSQL Dashboards

• PostgreSQL Instances Overview

• PostgreSQL Instance Summary

• PostgreSQL Instances Compare

ProxySQL Dashboards

• ProxySQL Instance Summary

HA Dashboards

• PXC/Galera Node Summary

• PXC/Galera Cluster Summary

• PXC/Galera Nodes Compare

• HAProxy Instance Summary

187 of 359 Percona LLC, © 2021

 5.4.2 Insight

5.4.2 Insight

Home Dashboard

The Home Dashboard is a high-level overview of your environment, the starting page of the PMM portal
from which you can open the tools of PMM, and browse to online resources.

On the PMM home page, you can also find the version number and a button to update your PMM Server.

GENERAL INFORMATION

This section contains links to online resources, such as PMM documentation, releases notes, and blogs.

SHARED AND RECENTLY USED DASHBOARDS

This section is automatically updated to show the most recent dashboards that you worked with. It also
contains the dashboards that you have bookmarked.

STATISTICS

This section shows the total number of hosts added to PMM and the total number of database instanced
being monitored. This section also current the version number. Use the Check for Updates Manually button
to see if you are using the most recent version of PMM.

188 of 359 Percona LLC, © 2021

 5.4.2 Insight

ENVIRONMENT OVERVIEW

This section lists all added hosts along with essential information about their performance. For each host,
you can find the current values of the following metrics:

• CPU Busy

• Memory Available

• Disk Reads

• Disk Writes

• Network IO

• DB Connections

• DB QPS

• Virtual CPUs

• RAM

• Host Uptime

• DB Uptime

189 of 359 Percona LLC, © 2021

 5.4.2 Insight

Advanced Data Exploration

The Advanced Data Exploration dashboard provides detailed information about the progress of a single
Prometheus metric across one or more hosts.

VIEW ACTUAL METRIC VALUES (GAUGE)

A gauge is a metric that represents a single numerical value that can arbitrarily go up and down.

190 of 359 Percona LLC, © 2021

 5.4.2 Insight

Gauges are typically used for measured values like temperatures or current memory usage, but also
“counts” that can go up and down, like the number of running goroutines.

VIEW METRIC RATE OF CHANGE (COUNTER)

A counter is a cumulative metric that represents a single numerical value that only ever goes up. A counter
is typically used to count requests served, tasks completed, errors occurred, etc. Counters should not be
used to expose current counts of items whose number can also go down, e.g. the number of currently
running goroutines. Use gauges for this use case.

METRIC RATES

Shows the number of samples Per second stored for a given interval in the time series.

This dashboard supports metrics related to NUMA. The names of all these metrics start with
node_memory_numa .

191 of 359 Percona LLC, © 2021

 5.4.2 Insight

VictoriaMetrics

No description

192 of 359 Percona LLC, © 2021

 5.4.2 Insight

VictoriaMetrics Agents Overview

No description

193 of 359 Percona LLC, © 2021

 5.4.3 PMM

5.4.3 PMM

PMM Inventory

The Inventory dashboard is a high level overview of all objects PMM “knows” about.

It contains three tabs (services, agents, and nodes) with lists of the correspondent objects and details
about them, so that users are better able to understand which objects are registered against PMM Server.
These objects are composing a hierarchy with Node at the top, then Service and Agents assigned to a Node.

• Nodes – Where the service and agents will run. Assigned a node_id , associated with a machine_id
(from /etc/machine-id ). Few examples are bare metal, virtualized, container.

• Services – Individual service names and where they run, against which agents will be assigned. Each
instance of a service gets a service_id value that is related to a node_id . Examples are MySQL,
Amazon Aurora MySQL. This feature also allows to support multiple mysqld instances on a single node,
with different service names, e.g. mysql1-3306, and mysql1-3307.

• Agents – Each binary (exporter, agent) running on a client will get an agent_id value.

• pmm-agent one is the top of the tree, assigned to a node_id

• node_exporter is assigned to pmm-agent agent_id

• mysqld_exporter & QAN MySQL Perfschema are assigned to a service_id .

Examples are pmm-agent , node_exporter , mysqld_exporter , QAN MySQL Perfschema.

194 of 359 Percona LLC, © 2021

 5.4.3 PMM

REMOVING ITEMS FROM THE INVENTORY

You can remove items from the inventory.

1. Open Home Dashboard > PMM Inventory

2. In the first column, select the items to be removed.

3. Click Delete. The interface will ask you to confirm the operation:

195 of 359 Percona LLC, © 2021

 5.4.4 OS Dashboards

5.4.4 OS Dashboards

CPU Utilization Details

OVERALL CPU UTILIZATION

The Overall CPU Utilization metric shows how much of the overall CPU time is used by the server. It has
these components:

Max Core Utilization

No description

System

This component the proportion of time the CPUs spent inside the Linux kernel for operations like context
switching, memory allocation and queue handling.

196 of 359 Percona LLC, © 2021

 5.4.4 OS Dashboards

User

This component is the time spent in the user space. Normally, most of the MySQL CPU time is in user
space. A high value of user time indicates a CPU bound workload.

Softirq

This component is the portion of time the CPU spent servicing software interrupts generated by the device
drivers. A high value of softirq may indicates a poorly configured device. The network devices are
generally the main source of high softirq values.

Steal

When multiple virtual machines share the same physical host, some virtual machines may be allowed to
use more of their share of CPU and that CPU time is accounted as Steal by the virtual machine from which
the time is taken.

Iowait

This component is the time the CPU spent waiting for disk IO requests to complete. A high value of iowait
indicates a disk bound load.

Nice

No description

In addition, sampling of the Max utilization of a single core is shown.

This metric presents global values: while there may be a lot of unused CPU, a single core may be saturated.
Look at the Max Core Utilization to see if any core is reaching close to 100%.

CURRENT CPU THREADS UTILIZATION

This shows the total utilization of each CPU core along with the average utilization of all CPU cores. Watch
for any core close to 100% utilization and investigate the root cause.

CPU THREADS FREQUENCY

No description

CURRENT CPU CORES TEMPERATURE

No description

OVERALL CPU THREADS UTILIZATION DETAILS

No description

197 of 359 Percona LLC, © 2021

 5.4.4 OS Dashboards

Disk Details

MOUNT POINT USAGE

Shows the percentage of disk space utilization for every mount point defined on the system. Having some of
the mount points close to 100% space utilization is not good because of the risk of a “disk full” error that
can block one of the services or even cause a crash of the entire system.

In cases where the mount point is close to 100% consider removing unused files or expanding the space
allocated to the mount point.

MOUNT POINT

Shows information about the disk space usage of the specified mount point.

Used is the amount of space used.

Free is the amount of space not in use.

Used+Free is the total disk space allocated to the mount point.

Having Free close to 0 B is not good because of the risk of a “disk full” error that can block one of the
services or even cause a crash of the entire system.

In cases where Free is close to 0 B consider removing unused files or expanding the space allocated to the
mount point.

DISK LATENCY

Shows average latency for Reads and Writes IO Devices. Higher than typical latency for highly loaded
storage indicates saturation (overload) and is frequent cause of performance problems. Higher than normal
latency also can indicate internal storage problems.

DISK OPERATIONS

Shows amount of physical IOs (reads and writes) different devices are serving. Spikes in number of IOs
served often corresponds to performance problems due to IO subsystem overload.

198 of 359 Percona LLC, © 2021

 5.4.4 OS Dashboards

DISK BANDWIDTH

Shows volume of reads and writes the storage is handling. This can be better measure of IO capacity usage
for network attached and SSD storage as it is often bandwidth limited. Amount of data being written to the
disk can be used to estimate Flash storage life time.

DISK LOAD

Shows how much disk was loaded for reads or writes as average number of outstanding requests at
different period of time. High disk load is a good measure of actual storage utilization. Different storage
types handle load differently - some will show latency increases on low loads others can handle higher load
with no problems.

DISK IO UTILIZATION

Shows disk Utilization as percent of the time when there was at least one IO request in flight. It is designed
to match utilization available in iostat tool. It is not very good measure of true IO Capacity Utilization.
Consider looking at IO latency and Disk Load Graphs instead.

AVG DISKS OPERATIONS MERGE RATIO

Shows how effectively Operating System is able to merge logical IO requests into physical requests. This is
a good measure of the IO locality which can be used for workload characterization.

DISK IO SIZE

Shows average size of a single disk operation.

199 of 359 Percona LLC, © 2021

 5.4.4 OS Dashboards

Network Details

LAST HOUR STATISTIC

This section reports the inbound speed, outbound speed, traffic errors and drops, and retransmit rate.

NETWORK TRAFFIC

This section contains the Network traffic and network utilization hourly metrics.

NETWORK TRAFFIC DETAILS

This section offers the following metrics:

• Network traffic by packets

• Network traffic errors

• Network traffic drop

• Network traffic multicast

NETWORK NETSTAT TCP

This section offers the following metrics:

• Timeout value used for retransmitting

• Min TCP retransmission timeout

• Max TCP retransmission timeout

• Netstat: TCP

• TCP segments

NETWORK NETSTAT UDP

In this section, you can find the following metrics:

• Netstat: UDP

• UDP Lite

200 of 359 Percona LLC, © 2021

 5.4.4 OS Dashboards

The graphs in the UDP Lite metric give statistics about:

InDatagrams

Packets received

OutDatagrams

Packets sent

InCsumErrors

Datagrams with checksum errors

InErrors

Datagrams that could not be delivered to an application

RcvbufErrors

Datagrams for which not enough socket buffer memory to receive

SndbufErrors

Datagrams for which not enough socket buffer memory to transmit

NoPorts

Datagrams received on a port with no listener

ICMP

This section has the following metrics:

• ICMP Errors

• Messages/Redirects

• Echos

• Timestamps/Mask Requests

ICMP Errors

InErrors

Messages which the entity received but determined as having ICMP-specific errors (bad ICMP checksums,
bad length, etc.)

OutErrors

Messages which this entity did not send due to problems discovered within ICMP, such as a lack of buffers

InDestUnreachs

Destination Unreachable messages received

OutDestUnreachs

Destination Unreachable messages sent

201 of 359 Percona LLC, © 2021

 5.4.4 OS Dashboards

InType3

Destination unreachable

OutType3

Destination unreachable

InCsumErrors

Messages with ICMP checksum errors

InTimeExcds

Time Exceeded messages received

Messages/Redirects

InMsgs

Messages which the entity received. Note that this counter includes all those counted by icmpInErrors

InRedirects

Redirect messages received

OutMsgs

Messages which this entity attempted to send. Note that this counter includes all those counted by
icmpOutErrors

OutRedirects

Redirect messages sent. For a host, this object will always be zero, since hosts do not send redirects

Echos

InEchoReps

Echo Reply messages received

InEchos

Echo (request) messages received

OutEchoReps

Echo Reply messages sent

OutEchos

Echo (request) messages sent

Timestamps/Mask Requests

InAddrMaskReps

Address Mask Reply messages received

202 of 359 Percona LLC, © 2021

 5.4.4 OS Dashboards

InAddrMasks

Address Mask Request messages received

OutAddrMaskReps

Address Mask Reply messages sent

OutAddrMasks

Address Mask Request messages sent

InTimestampReps

Timestamp Reply messages received

InTimestamps

Timestamp Request messages received

OutTimestampReps

Timestamp Reply messages sent

OutTimestamps

Timestamp Request messages sent

203 of 359 Percona LLC, © 2021

 5.4.4 OS Dashboards

Memory Details

MEMORY USAGE

No description

204 of 359 Percona LLC, © 2021

 5.4.4 OS Dashboards

Node Temperature Details

The Node Temperature Details dashboard exposes hardware monitoring and sensor data obtained through
the sysfs virtual file system of the node.

Hardware monitoring devices attached to the CPU and/or other chips on the motherboard let you monitor
the hardware health of a system. Most modern systems include several of such devices. The actual list can
include temperature sensors, voltage sensors, fan speed sensors, and various additional features, such as
the ability to control the rotation speed of the fans.

CPU CORES TEMPERATURES

Presents data taken from the temperature sensors of the CPU

CHIPS TEMPERATURES

Presents data taken from the temperature sensors connected to other system controllers

FAN ROTATION SPEEDS

Fan rotation speeds reported in RPM (rotations per minute).

FAN POWER USAGE

Describes the pulse width modulation of the PWN-equipped fans. PWM operates like a switch that constantly
cycles on and off, thereby regulating the amount of power the fan gains: 100% makes it rotate at full
speed, while lower percentage slows rotation down proportionally.

205 of 359 Percona LLC, © 2021

 5.4.4 OS Dashboards

Nodes Compare

This dashboard lets you compare a wide range of parameters. Parameters of the same type are shown side
by side for all servers, grouped into the following sections:

• System Information

• CPU

• Memory

• Disk Partitions

• Disk Performance

• Network

The System Information section shows the System Info summary of each server, as well as System Uptime,
CPU Cores, RAM, Saturation Metrics, and Load Average gauges.

The CPU section offers the CPU Usage, Interrupts, and Context Switches metrics.

In the Memory section, you can find the Memory Usage, Swap Usage, and Swap Activity metrics.

The Disk Partitions section encapsulates two metrics, Mountpoint Usage and Free Space.

The Disk Performance section contains the I/O Activity, Disk Operations, Disk Bandwidth, Disk IO Utilization,
Disk Latency, and Disk Load metrics.

Finally, Network section shows Network Traffic, and Network Utilization Hourly metrics.

206 of 359 Percona LLC, © 2021

 5.4.4 OS Dashboards

Nodes Overview

The Nodes Overview dashboard provides details about the efficiency of work of the following components.
Each component is represented as a section in the dashboard.

• CPU

• Memory & Swap

• Disk

• Network

The CPU section offers the CPU Usage, CPU Saturation and Max Core Usage, Interrupts and Context
Switches, and Processes metrics.

In the Memory section, you can find the Memory Utilization, Virtual Memory Utilization, Swap Space, and
Swap Activity metrics.

The Disk section contains the I/O Activity, Global File Descriptors Usage, Disk IO Latency, and Disk IO Load
metrics.

In the Network section, you can find the Network Traffic, Network Utilization Hourly, Local Network Errors,
and TCP Retransmission metrics.

207 of 359 Percona LLC, © 2021

 5.4.4 OS Dashboards

Node Summary

SYSTEM SUMMARY

The output from pt-summary , one of the Percona Toolkit utilities.

CPU USAGE

The CPU time is measured in clock ticks or seconds. It is useful to measure CPU time as a percentage of the
CPU’s capacity, which is called the CPU usage.

CPU SATURATION AND MAX CORE USAGE

When a system is running with maximum CPU utilization, the transmitting and receiving threads must all
share the available CPU. This will cause data to be queued more frequently to cope with the lack of CPU.
CPU Saturation may be measured as the length of a wait queue, or the time spent waiting on the queue.

INTERRUPTS AND CONTEXT SWITCHES

Interrupt is an input signal to the processor indicating an event that needs immediate attention. An interrupt
signal alerts the processor and serves as a request for the processor to interrupt the currently executing
code, so that the event can be processed in a timely manner.

Context switch is the process of storing the state of a process or thread, so that it can be restored and
resume execution at a later point. This allows multiple processes to share a single CPU, and is an essential
feature of a multitasking operating system.

PROCESSES

No description

MEMORY UTILIZATION

No description

VIRTUAL MEMORY UTILIZATION

No description

208 of 359 Percona LLC, © 2021

 5.4.4 OS Dashboards

SWAP SPACE

No description

SWAP ACTIVITY

Swap Activity is memory management that involves swapping sections of memory to and from physical
storage.

I/O ACTIVITY

Disk I/O includes read or write or input/output operations involving a physical disk. It is the speed with
which the data transfer takes place between the hard disk drive and RAM.

GLOBAL FILE DESCRIPTORS USAGE

No description

DISK IO LATENCY

Shows average latency for Reads and Writes IO Devices. Higher than typical latency for highly loaded
storage indicates saturation (overload) and is frequent cause of performance problems. Higher than normal
latency also can indicate internal storage problems.

DISK IO LOAD

Shows how much disk was loaded for reads or writes as average number of outstanding requests at
different period of time. High disk load is a good measure of actual storage utilization. Different storage
types handle load differently - some will show latency increases on low loads others can handle higher load
with no problems.

NETWORK TRAFFIC

Network traffic refers to the amount of data moving across a network at a given point in time.

NETWORK UTILIZATION HOURLY

No description

LOCAL NETWORK ERRORS

Total Number of Local Network Interface Transmit Errors, Receive Errors and Drops. Should be Zero

TCP RETRANSMISSION

Retransmission, essentially identical with Automatic repeat request (ARQ), is the resending of packets which
have been either damaged or lost. Retransmission is one of the basic mechanisms used by protocols
operating over a packet switched computer network to provide reliable communication (such as that
provided by a reliable byte stream, for example TCP).

209 of 359 Percona LLC, © 2021

 5.4.4 OS Dashboards

NUMA Details

For each node, this dashboard shows metrics related to Non-uniform memory access (NUMA).

MEMORY USAGE

Remotes over time the total, used, and free memory.

FREE MEMORY PERCENT

Shows the free memory as the ratio to the total available memory.

NUMA MEMORY USAGE TYPES

Dirty

Memory waiting to be written back to disk

Bounce

Memory used for block device bounce buffers

Mapped

Files which have been mapped, such as libraries

KernelStack

The memory the kernel stack uses. This is not reclaimable.

NUMA ALLOCATION HITS

Memory successfully allocated on this node as intended.

NUMA ALLOCATION MISSED

Memory missed is allocated on a node despite the process preferring some different node.

Memory foreign is intended for a node, but actually allocated on some different node.

210 of 359 Percona LLC, © 2021

 5.4.4 OS Dashboards

ANONYMOUS MEMORY

Active

Anonymous memory that has been used more recently and usually not swapped out.

Inactive

Anonymous memory that has not been used recently and can be swapped out.

NUMA FILE (PAGECACHE)

Active(file) Pagecache memory that has been used more recently and usually not reclaimed until needed.

Inactive(file) Pagecache memory that can be reclaimed without huge performance impact.

SHARED MEMORY

Shmem Total used shared memory (shared between several processes, thus including RAM disks, SYS-V-
IPC and BSD like SHMEM).

HUGEPAGES STATISTICS

Total

Number of hugepages being allocated by the kernel (Defined with vm.nr_hugepages ).

Free

The number of hugepages not being allocated by a process

Surp

The number of hugepages in the pool above the value in vm.nr_hugepages . The maximum number of
surplus hugepages is controlled by vm.nr_overcommit_hugepages .

LOCAL PROCESSES

Memory allocated on a node while a process was running on it.

REMOTE PROCESSES

Memory allocated on a node while a process was running on some other node.

SLAB MEMORY

Slab

Allocation is a memory management mechanism intended for the efficient memory allocation of kernel
objects.

SReclaimable

The part of the Slab that might be reclaimed (such as caches).

SUnreclaim

The part of the Slab that can’t be reclaimed under memory pressure

211 of 359 Percona LLC, © 2021

 5.4.4 OS Dashboards

Processes Details

The Processes Details dashboard displays Linux process information - PIDs, Threads, and Processes. The
dashboard shows how many processes/threads are either in the kernel run queue (runnable state) or in the
blocked queue (waiting for I/O). When the number of process in the runnable state is constantly higher
than the number of CPU cores available, the load is CPU bound. When the number of process blocked
waiting for I/O is large, the load is disk bound. The running average of the sum of these two quantities is
the basis of the loadavg metric.

The dashboard consists of two parts: the first section describes metrics for all hosts, and the second part
provides charts for each host.

Charts for all hosts, available in the first section, are the following ones:

• States of Processes

• Number of PIDs

• Percentage of Max PIDs Limit

• Number of Threads

• Percentage of Max Threads Limit

• Runnable Processes

• Blocked Processes Waiting for I/O

• Sleeping Processes

• Running Processes

• Disk Sleep Processes

• Stopped Processes

• Zombie Processes

• Dead Processes

212 of 359 Percona LLC, © 2021

 5.4.4 OS Dashboards

The following charts are present in the second part, available for each host:

• Processes

• States of Processes

• Number of PIDs

• Percentage of Max PIDs Limit

• Number of Threads

• Percentage of Max Threads Limit

NUMBER OF PIDS

No description

PERCENTAGE OF MAX PIDS LIMIT

No description

NUMBER OF THREADS

No description

PERCENTAGE OF MAX THREADS LIMIT

No description

RUNNABLE PROCESSES

Processes

The Processes graph shows how many processes/threads are either in the kernel run queue (runnable
state) or in the blocked queue (waiting for I/O). When the number of process in the runnable state is
constantly higher than the number of CPU cores available, the load is CPU bound. When the number of
process blocked waiting for I/O is large, the load is disk bound. The running average of the sum of these
two quantities is the basis of the loadavg metric.

BLOCKED PROCESSES WAITING FOR I/O

Processes

The Processes graph shows how many processes/threads are either in the kernel run queue (runnable
state) or in the blocked queue (waiting for I/O). When the number of process in the runnable state is
constantly higher than the number of CPU cores available, the load is CPU bound. When the number of
process blocked waiting for I/O is large, the load is disk bound. The running average of the sum of these
two quantities is the basis of the loadavg metric.

SLEEPING PROCESSES

No description

RUNNING PROCESSES

No description

DISK SLEEP PROCESSES

No description

STOPPED PROCESSES

No description

213 of 359 Percona LLC, © 2021

 5.4.4 OS Dashboards

ZOMBIE PROCESSES

No description

DEAD PROCESSES

No description

214 of 359 Percona LLC, © 2021

 5.4.5 Prometheus Dashboards

5.4.5 Prometheus Dashboards

Prometheus Exporter Status

The Prometheus Exporter Status dashboard reports the consumption of resources by the Prometheus
exporters used by PMM. For each exporter, this dashboard reveals the following information:

• CPU usage

• Memory usage

• File descriptors used

• Exporter uptime

215 of 359 Percona LLC, © 2021

 5.4.5 Prometheus Dashboards

Prometheus Exporters Overview

PROMETHEUS EXPORTERS SUMMARY

This section provides a summary of how exporters are used across the selected hosts. It includes the
average usage of CPU and memory as well as the number of hosts being monitored and the total number
of running exporters.

Avg CPU Usage per Host

Shows the average CPU usage in percent per host for all exporters.

Avg Memory Usage per Host

Shows the Exporters average Memory usage per host.

Monitored Hosts

Shows the number of monitored hosts that are running Exporters.

Exporters Running

Shows the total number of Exporters running with this PMM Server instance.

The CPU usage and memory usage do not include the additional CPU and memory usage required to
produce metrics by the application or operating system.

PROMETHEUS EXPORTERS RESOURCE USAGE BY NODE

This section shows how resources, such as CPU and memory, are being used by the exporters for the
selected hosts.

CPU Usage

Plots the Exporters’ CPU usage across each monitored host (by default, All hosts).

216 of 359 Percona LLC, © 2021

 5.4.5 Prometheus Dashboards

Memory Usage

Plots the Exporters’ Memory usage across each monitored host (by default, All hosts).

PROMETHEUS EXPORTERS RESOURCE USAGE BY TYPE

This section shows how resources, such as CPU and memory, are being used by the exporters for host
types: MySQL, MongoDB, ProxySQL, and the system.

CPU Cores Used

Shows the Exporters’ CPU Cores used for each type of Exporter.

Memory Usage

Shows the Exporters’ memory used for each type of Exporter.

LIST OF HOSTS

At the bottom, this dashboard shows details for each running host.

CPU Used

Show the CPU usage as a percentage for all Exporters.

Mem Used

Shows total Memory Used by Exporters.

Exporters Running

Shows the number of Exporters running.

RAM

Shows the total amount of RAM of the host.

Virtual CPUs

Shows the total number of virtual CPUs on the host.

You can click the value of the CPU Used, Memory Used, or Exporters Running columns to open the
Prometheus Exporter Status dashboard for further analysis.

Percona blog: Understand Your Prometheus Exporters with Percona Monitoring and Management (PMM)

217 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

5.4.6 MySQL Dashboards

MySQL Amazon Aurora Details

AMAZON AURORA TRANSACTION COMMITS

This graph shows the number of Commits which Amazon Aurora engine performed as well as average
commit latency. Graph Latency does not always correlate with the number of performed commits and can
be quite high in certain situations.

• Number of Amazon Aurora Commits: The average number of commit operations per second.

• Amazon Aurora Commit avg Latency: The average amount of latency for commit operations

218 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

AMAZON AURORA LOAD

This graph shows us what statements contribute most load on the system as well as what load corresponds
to Amazon Aurora transaction commit.

• Write Transaction Commit Load: Load in Average Active Sessions per second for COMMIT operations

• UPDATE load: Load in Average Active Sessions per second for UPDATE queries

• SELECT load: Load in Average Active Sessions per second for SELECT queries

• DELETE load: Load in Average Active Sessions per second for DELETE queries

• INSERT load: Load in Average Active Sessions per second for INSERT queries

An active session is a connection that has submitted work to the database engine and is waiting for a
response from it. For example, if you submit an SQL query to the database engine, the database session is
active while the database engine is processing that query.

AURORA MEMORY USED

This graph shows how much memory is used by Amazon Aurora lock manager as well as amount of
memory used by Amazon Aurora to store Data Dictionary.

• Aurora Lock Manager Memory: the amount of memory used by the Lock Manager, the module
responsible for handling row lock requests for concurrent transactions.

• Aurora Dictionary Memory: the amount of memory used by the Dictionary, the space that contains
metadata used to keep track of database objects, such as tables and indexes.

AMAZON AURORA STATEMENT LATENCY

This graph shows average latency for the most important types of statements. Latency spikes are often
indicative of the instance overload.

• DDL Latency: Average time to execute DDL queries

• DELETE Latency: Average time to execute DELETE queries

• UPDATE Latency: Average time to execute UPDATE queries

• SELECT Latency: Average time to execute SELECT queries

• INSERT Latency: Average time to execute INSERT queries

AMAZON AURORA SPECIAL COMMAND COUNTERS

Amazon Aurora MySQL allows a number of commands which are not available in standard MySQL. This
graph shows usage of such commands. Regular unit_test calls can be seen in default Amazon Aurora
install, the rest will depend on your workload.

• show_volume_status : The number of executions per second of the command SHOW VOLUME STATUS.
The SHOW VOLUME STATUS query returns two server status variables, Disks and Nodes. These
variables represent the total number of logical blocks of data and storage nodes, respectively, for the
DB cluster volume.

• awslambda : The number of AWS Lambda calls per second. AWS Lambda is an event-drive, server-less
computing platform provided by AWS. It is a compute service that run codes in response to an event.
You can run any kind of code from Aurora invoking Lambda from a stored procedure or a trigger.

• alter_system : The number of executions per second of the special query ALTER SYSTEM, that is a
special query to simulate an instance crash, a disk failure, a disk congestion or a replica failure. It’s a
useful query for testing the system.

219 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

AMAZON AURORA PROBLEMS

This graph shows different kinds of Internal Amazon Aurora MySQL Problems which general should be zero
in normal operation.

Anything non-zero is worth examining in greater depth.

220 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

MySQL Command/Handler Counters Compare

This dashboard shows server status variables. On this dashboard, you may select multiple servers and
compare their counters simultaneously.

Server status variables appear in two sections: Commands and Handlers. Choose one or more variables in
the Command and Handler fields in the top menu to select the variables which will appear in the
COMMANDS or HANDLERS section for each host. Your comparison may include from one up to three hosts.

By default or if no item is selected in the menu, PMM displays each command or handler respectively.

221 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

MySQL InnoDB Compression Details

This dashboard helps you analyze the efficiency of InnoDB compression.

COMPRESSION LEVEL AND FAILURE RATE THRESHOLD

InnoDB Compression Level

The level of zlib compression to use for InnoDB compressed tables and indexes.

InnoDB Compression Failure Threshold

The compression failure rate threshold for a table.

Compression Failure Rate Threshold

The maximum percentage that can be reserved as free space within each compressed page, allowing
room to reorganize the data and modification log within the page when a compressed table or index is
updated and the data might be recompressed.

Write Pages to the Redo Log

Specifies whether images of re-compressed pages are written to the redo log. Re-compression may occur
when changes are made to compressed data.

STATISTIC OF COMPRESSION OPERATIONS

Compress Attempts

Number of compression operations attempted. Pages are compressed whenever an empty page is created
or the space for the uncompressed modification log runs out.

Uncompressed Attempts

Number of uncompression operations performed. Compressed InnoDB pages are uncompressed whenever
compression fails, or the first time a compressed page is accessed in the buffer pool and the
uncompressed page does not exist.

222 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

CPU CORE USAGE

CPU Core Usage for Compression

Shows the time in seconds spent by InnoDB Compression operations.

CPU Core Usage for Uncompression

Shows the time in seconds spent by InnoDB Uncompression operations.

BUFFER POOL TOTAL

Total Used Pages

Shows the total amount of used compressed pages into the InnoDB Buffer Pool split by page size.

Total Free Pages

Shows the total amount of free compressed pages into the InnoDB Buffer Pool split by page size.

223 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

MySQL InnoDB Details

INNODB ACTIVITY

Writes (Rows)

Writes (Transactions)

Row Writes per Trx

Rows Written Per Transactions which modify rows. This is better indicator of transaction write size than
looking at all transactions which did not do any writes as well.

Rows Read Per Trx

Log Space per Trx

Rollbacks

Percent of Transaction Rollbacks (as portion of read-write transactions).

BP Reqs Per Row

Number of Buffer Pool requests per Row Access. High numbers here indicate going through long undo
chains, deep trees and other inefficient data access. It can be less than zero due to several rows being read
from single page.

224 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

Log Fsync Per Trx

Log Fsync Per Transaction.

InnoDB Row Reads

InnoDB Row Operations

This graph allows you to see which operations occur and the number of rows affected per operation. A
graph like Queries Per Second will give you an idea of queries, but one query could effect millions of rows.

InnoDB Row Writes

InnoDB Row Operations

This graph allows you to see which operations occur and the number of rows affected per operation. A
graph like Queries Per Second will give you an idea of queries, but one query could effect millions of rows.

InnoDB Read-Only Transactions

InnoDB Read-Write Transactions

InnoDB Transactions Information (RW)

The InnoDB Transactions Information graph shows details about the recent transactions. Transaction IDs
Assigned represents the total number of transactions initiated by InnoDB. RW Transaction Commits are the
number of transactions not read-only. Insert-Update Transactions Commits are transactions on the Undo
entries. Non Locking RO Transaction Commits are transactions commit from select statement in auto-
commit mode or transactions explicitly started with “start transaction read only”.

If you do not see any metric, try running: SET GLOBAL innodb_monitor_enable=all; in the MySQL client.

Misc InnoDB Transactions Information

Additional InnoDB Transaction Information

INNODB STORAGE SUMMARY

InnoDB Tables

Current Number of InnoDB Tables in database

Data Buffer Pool Fit

Buffer Pool Size as Portion of the Data

Avg Row Size

Amount of Data Per Row

Index Size Per Row

Index Size Per Row shows how much space we’re using for indexes on per row basics

InnoDB Data Summary

Space Allocated

Total Amount of Space Allocated. May not exactly match amount of space used on file system but provided
great guidance.

Space Used

Space used in All InnoDB Tables. Reported Allocated Space Less Free Space.

Data Length

Space Used by Data (Including Primary Key).

225 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

Index Length

Space Used by Secondary Indexes.

Estimated Rows

Estimated number of Rows in InnoDB Storage Engine. It is not exact value and it can change abruptly as
information is updated.

Indexing Overhead

How Much Indexes Take Compared to Data.

Free Space Percent

How Much Space is Free. Too high value wastes space on disk.

Free

Allocated Space not currently used by Data or Indexes.

InnoDB File Per Table

If Enabled, By Default every Table will have its own Tablespace represented as its own .idb file rather than
all tables stored in single system tablespace.

INNODB DISK IO

InnoDB Page Size

Avg Data Read Rq Size

Avg Data Write Rq Size

Avg Log Write Rq Size

Data Written Per Fsync

Log Written Per Fsync

Data Read Per Row Read

Data Written Per Row Written

Due to difference in timing of Row Write and Data Write the value may be misleading on short intervals.

InnoDB Data I/O

InnoDB I/O

• Data Writes - The total number of InnoDB data writes.

• Data Reads - The total number of InnoDB data reads (OS file reads).

• Log Writes - The number of physical writes to the InnoDB redo log file.

• Data Fsyncs - The number of fsync() operations. The frequency of fsync() calls is influenced by the
setting of the innodb_flush_method configuration option.

226 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

InnoDB Data Bandwidth

InnoDB Log IO

InnoDB I/O

• Data Writes - The total number of InnoDB data writes.

• Data Reads - The total number of InnoDB data reads (OS file reads).

• Log Writes - The number of physical writes to the InnoDB redo log file.

• Data Fsyncs - The number of fsync() operations. The frequency of fsync() calls is influenced by the
setting of the innodb_flush_method configuration option.

InnoDB FSyncs

InnoDB Pending IO

InnoDB Pending Fsyncs

InnoDB Auto Extend Increment

When Growing InnoDB System Tablespace extend it by this size at the time.

InnoDB Double Write

Whether InnoDB Double Write Buffer is enabled. Doing so doubles amount of writes InnoDB has to do to
storage but is required to avoid potential data corruption during the crash on most storage subsystems.

InnoDB Fast Shutdown

Fast Shutdown means InnoDB will not perform complete Undo Space and Change Buffer cleanup on
shutdown, which is faster but may interfere with certain major upgrade operations.

InnoDB Open Files

Maximum Number of Files InnoDB is Allowed to use.

InnoDB File Use

Portion of Allowed InnoDB Open Files Use.

INNODB IO OBJECTS

InnoDB IO Targets Write Load

Write Load Includes both Write and fsync (referred as misc).

INNODB BUFFER POOL

Buffer Pool Size

InnoDB Buffer Pool Size

InnoDB maintains a storage area called the buffer pool for caching data and indexes in memory. Knowing
how the InnoDB buffer pool works, and taking advantage of it to keep frequently accessed data in memory,
is one of the most important aspects of MySQL tuning. The goal is to keep the working set in memory. In
most cases, this should be between 60%-90% of available memory on a dedicated database host, but
depends on many factors.

Buffer Pool Size of Total RAM

InnoDB Buffer Pool Size % of Total RAM

InnoDB maintains a storage area called the buffer pool for caching data and indexes in memory. Knowing
how the InnoDB buffer pool works, and taking advantage of it to keep frequently accessed data in memory,
is one of the most important aspects of MySQL tuning. The goal is to keep the working set in memory. In

227 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

most cases, this should be between 60%-90% of available memory on a dedicated database host, but
depends on many factors.

NUMA Interleave

Interleave Buffer Pool between NUMA zones to better support NUMA systems.

Buffer Pool Activity

Combined value of Buffer Pool Read and Write requests.

BP Data

Percent of Buffer Pool Occupied by Cached Data.

BP Data Dirty

Percent of Data which is Dirty.

BP Miss Ratio

How often buffer pool read requests have to do read from the disk. Keep this percent low for good
performance.

BP Write Buffering

Number of Logical Writes to Buffer Pool Per logical Write.

InnoDB Buffer Pool LRU Sub-Chain Churn

Buffer Pool Chunk Size

Size of the “Chunk” for buffer pool allocation. Allocation of buffer pool will be rounded by this number. It
also affects the performance impact of online buffer pool resize.

Buffer Pool Instances

Number of Buffer Pool Instances. Higher values allow to reduce contention but also increase overhead.

Read Ahead IO Percent

Percent of Reads Caused by InnoDB Read Ahead.

Read Ahead Wasted

Percent of Pages Fetched by Read Ahead Evicted Without Access.

Dump Buffer Pool on Shutdown

Load Buffer Pool at Startup

Portion of Buffer Pool To Dump/Load

Larger Portion increases dump/load time but get more of original buffer pool content and hence may
reduce warmup time.

Include Buffer Pool in Core Dump

Whenever to Include Buffer Pool in Crash Core Dumps. Doing so may dramatically increase core dump file
slow down restart. Only makes a difference if core dumping on crash is enabled.

InnoDB Old Blocks

Percent of The Buffer Pool To be Reserved for “Old Blocks” - which has been touched repeatedly over period
of time.

228 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

InnoDB Old Blocks Time

The Time which has to pass between multiple touches for the block for it to qualify as old block.

InnoDB Random Read Ahead

Is InnoDB Random ReadAhead Enabled.

InnoDB Random Read Ahead

The Threshold (in Pages) to trigger Linear Read Ahead.

InnoDB Read IO Threads

Number of Threads used to Schedule Reads.

InnoDB Write IO Threads

Number of Threads used to Schedule Writes.

InnoDB Native AIO Enabled

Whether Native Asynchronous IO is enabled. Strongly recommended for optimal performance.

INNODB BUFFER POOL - REPLACEMENT MANAGEMENT

LRU Scan Depth

InnoDB LRU Scan Depth

This variable defines InnoDB Free Page Target per buffer pool. When number of free pages falls below this
number this number page cleaner will make required amount of pages free, flushing or evicting pages from
the tail of LRU as needed.

LRU Clean Page Searches

When Page is being read (or created) the Page need to be allocated in Buffer Pool.

Free List Miss Rate

The most efficient way to get a clean page is to grab one from free list. However if no pages are available in
Free List the LRU scan needs to be performed.

LRU Get Free Loops

If Free List was empty LRU Get Free Loop will be performed. It may perform LRU scan or may use some
other heuristics and shortcuts to get free page.

LRU Scans

If Page could not be find any Free list and other shortcuts did not work, free page will be searched by
scanning LRU chain which is not efficient.

Pages Scanned in LRU Scans

Pages Scanned Per Second while doing LRU scans. If this value is large (thousands) it means a lot of
resources are wasted.

Pages scanned per LRU Scan

Number of pages scanned per LRU scan in Average. Large number of scans can consume a lot of resources
and also introduce significant addition latency to queries.

LRU Get Free Waits

If InnoDB could not find a free page in LRU list and had to sleep. Should be zero.

229 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

INNODB CHECKPOINTING AND FLUSHING

Pages Flushed from Flush List

Number of Pages Flushed from “Flush List” This combines Pages Flushed through Adaptive Flush and
Background Flush.

Page Flush Batches Executed

InnoDB Flush Cycle typically Runs on 1 second intervals. If too far off from this number it can indicate an
issue.

Pages Flushed Per Batch

How many pages are flushed per Batch. Large Batches can “choke” IO subsystem and starve other IO
which needs to happen.

Neighbor Flushing Enabled

Neighbor Flushing is Optimized for Rotational Media and unless you’re Running spinning disks you should
disable it.

InnoDB Checkpoint Age

InnoDB Checkpoint Age

The maximum checkpoint age is determined by the total length of all transaction log files
( innodb_log_file_size ).

When the checkpoint age reaches the maximum checkpoint age, blocks are flushed synchronously. The
rules of the thumb is to keep one hour of traffic in those logs and let the check-pointing perform its work as
smooth as possible. If you don’t do this, InnoDB will do synchronous flushing at the worst possible time,
i.e., when you are busiest.

Pages Flushed (Adaptive)

Adaptive Flush Flushes pages from Flush List based on the need to advance Checkpoint (driven by Redo
Generation Rate) and by maintaining number of dirty pages within set limit.

Adaptive Flush Batches Executed

Pages Per Batch (Adaptive)

Pages Flushed Per Adaptive Batch.

Neighbor Flushing

To optimize IO for rotational Media InnoDB may flush neighbor pages. It can cause significant wasted IO
for flash storage. Generally for flash you should run with innodb_flush_neighbors=0 but otherwise this
shows how much IO you’re wasting.

Pages Flushed (LRU)

Flushing from the tail of the LRU list is needed to keep readily-available free pages for new data to be read
when data does not fit in the buffer pool.

LRU Flush Batches Executed

Pages Per Batch (LRU)

Pages Flushed Per Neighbor.

LSN Age Flush Batch Target

Target for Pages to Flush due to LSN Age.

230 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

Pages Flushed (Neighbor)

Number of Neighbor pages flushed (If neighbor flushing is enabled) from Flush List and LRU List Combined.

Neighbor Flush Batches Executed

Pages Per Batch (Neighbor)

Pages Flushed Per Neighbor.

Sync Flush Waits

If InnoDB could not keep up with Checkpoint Flushing and had to trigger Sync flush. This should never
happen.

Pages Flushed (Background)

Pages Flushed by Background Flush which is activated when server is considered to be idle.

Background Flush Batches Executed

Pages Per Batch (Background)

Pages Flushed Per Background Batch.

Redo Generation Rate

Rate at which LSN (Redo) is Created. It may not match how much data is written to log files due to block
size rounding.

InnoDB Flushing by Type

Pages Evicted (LRU)

This correspond to number of clean pages which were evicted (made free) from the tail of LRU buffer.

Page Eviction Batches

Pages Evicted per Batch

Max Log Space Used

Single Page Flushes

Single Page flushes happen in rare case, then clean page could not be found in LRU list. It should be zero
for most workloads.

Single Page Flush Pages Scanned

Pages Scanned Per Single Page Flush

InnoDB IO Capacity

Estimated number of IOPS storage system can provide. Is used to scale background activities. Do not set it
to actual storage capacity.

InnoDB IO Capacity Max

InnoDB IO Capacity to use when falling behind and need to catch up with Flushing.

INNODB LOGGING

Total Log Space

Number of InnoDB Log Files Multiplied by Their Size.

Log Buffer Size

InnoDB Log Buffer Size

231 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

The size of buffer InnoDB uses for buffering writes to log files.

At Transaction Commit

What to do with Log file At Transaction Commit. Do nothing and wait for timeout to flush the data from Log
Buffer, Flush it to OS Cache but not FSYNC or Flush only.

Flush Transaction Log Every

Every Specified Number of Seconds Flush Transaction Log.

InnoDB Write Ahead Block Size

This variable can be seen as minimum IO alignment InnoDB will use for Redo log file. High Values cause
waste, low values can make IO less efficient.

Log Write Amplification

How much Writes to Log Are Amplified compared to how much Redo is Generated.

Log Fsync Rate

Redo Generated per Trx

Amount of Redo Generated Per Write Transaction. This is a good indicator of transaction size.

InnoDB Log File Usage Hourly

InnoDB Log File Usage Hourly

Along with the buffer pool size, innodb_log_file_size is the most important setting when we are working
with InnoDB. This graph shows how much data was written to InnoDB’s redo logs over each hour. When
the InnoDB log files are full, InnoDB needs to flush the modified pages from memory to disk.

The rules of the thumb is to keep one hour of traffic in those logs and let the checkpointing perform its work
as smooth as possible. If you don’t do this, InnoDB will do synchronous flushing at the worst possible time,
i.e., when you are busiest.

This graph can help guide you in setting the correct innodb_log_file_size .

Log Padding Written

Amount of Log Padding Written.

InnoDB Log File Size

InnoDB Log Files

Number of InnoDB Redo Log Files.

Log Bandwidth

Redo Generation Rate

Rate at which LSN (Redo) is Created. It may not match how much data is written to log files due to block
size rounding.

InnoDB Group Commit Batch Size

The InnoDB Group Commit Batch Size graph shows how many bytes were written to the InnoDB log files per
attempt to write. If many threads are committing at the same time, one of them will write the log entries of
all the waiting threads and flush the file. Such process reduces the number of disk operations needed and
enlarge the batch size.

232 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

INNODB LOCKING

Lock Wait Timeout

InnoDB Lock Wait Timeout

How long to wait for row lock before timing out.

InnoDB Deadlock Detection

If Disabled InnoDB Will not detect deadlocks but rely on timeouts.

InnoDB Auto Increment Lock Mode

Will Define How much locking will come from working with Auto Increment Columns.

Rollback on Timeout

Whenever to rollback all transaction on timeout or just last statement.

Row Lock Blocking

Percent of Active Sections which are blocked due to waiting on InnoDB Row Locks.

Row Writes per Trx

Rows Written Per Transactions which modify rows. This is better indicator of transaction write size than
looking at all transactions which did not do any writes as well.

Rollbacks

Percent of Transaction Rollbacks (as portion of read-write transactions).

InnoDB Row Lock Wait Activity

InnoDB Row Lock Wait Time

InnoDB Row Lock Wait Load

Average Number of Sessions blocked from proceeding due to waiting on row level lock.

InnoDB Row Locks Activity

InnoDB Table Lock Activity

Current Locks

INNODB UNDO SPACE AND PURGING

Undo Tablespaces

Max Undo Log Size

InnoDB Undo Log Truncate

Purge Threads

Max Purge Lag

Maximum number of Unpurged Transactions, if this number exceeded delay will be introduced to incoming
DDL statements.

Max Purge Lag Delay

Current Purge Delay

The Delay Injected due to Purge Thread(s) unable to keep up with purge progress.

233 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

Rollback Segments

InnoDB Purge Activity

The InnoDB Purge Performance graph shows metrics about the page purging process. The purge process
removed the undo entries from the history list and cleanup the pages of the old versions of modified rows
and effectively remove deleted rows.

If you do not see any metric, try running: SET GLOBAL innodb_monitor_enable=all; in the MySQL client.

Transactions and Undo Records

InnoDB Undo Space Usage

The InnoDB Undo Space Usage graph shows the amount of space used by the Undo segment. If the amount
of space grows too much, look for long running transactions holding read views opened in the InnoDB
status.

If you do not see any metric, try running: SET GLOBAL innodb_monitor_enable=all; in the MySQL client.

Transaction History

InnoDB Purge Throttling

Records Per Undo Log Page

How Many Undo Operations Are Handled Per Each Undo Log Page.

Purge Invoked

How Frequently Purge Operation is Invoked.

Ops Per Purge

Home Many Purge Actions are done Per invocation.

Undo Slots Used

Number of Undo Slots Used.

Max Transaction History Length

Purge Batch Size

Rseg Truncate Frequency

INNODB PAGE OPERATIONS

InnoDB Page Splits and Merges

The InnoDB Page Splits graph shows the InnoDB page maintenance activity related to splitting and merging
pages. When an InnoDB page, other than the top most leaf page, has too much data to accept a row
update or a row insert, it has to be split in two. Similarly, if an InnoDB page, after a row update or delete
operation, ends up being less than half full, an attempt is made to merge the page with a neighbor page. If
the resulting page size is larger than the InnoDB page size, the operation fails. If your workload causes a
large number of page splits, try lowering the innodb_fill_factor variable (5.7+).

If you do not see any metric, try running: SET GLOBAL innodb_monitor_enable=all; in the MySQL client.

Page Merge Success Ratio

InnoDB Page Reorg Attempts

The InnoDB Page Reorgs graph shows information about the page reorganization operations. When a page
receives an update or an insert that affect the offset of other rows in the page, a reorganization is needed.

234 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

If the reorganization process finds out there is not enough room in the page, the page will be split. Page
reorganization can only fail for compressed pages.

If you do not see any metric, try running: SET GLOBAL innodb_monitor_enable=all; in the MySQL client.

InnoDB Page Reorgs Failures

The InnoDB Page Reorgs graph shows information about the page reorganization operations. When a page
receives an update or an insert that affect the offset of other rows in the page, a reorganization is needed.
If the reorganization process finds out there is not enough room in the page, the page will be split. Page
reorganization can only fail for compressed pages.

If you do not see any metric, try running: SET GLOBAL innodb_monitor_enable=all; in the MySQL client.

InnoDB Fill Factor

The portion of the page to fill then doing sorted Index Build. Lowering this value will worsen space
utilization but will reduce need to split pages when new data is inserted in the index.

INNODB ADAPTIVE HASH INDEX

Adaptive Hash Index Enabled

Adaptive Hash Index helps to optimize index Look-ups but can be severe hotspot for some workloads.

Adaptive Hash Index Partitions

How many Partitions Used for Adaptive Hash Index (to reduce contention).

Percent of Pages Hashed

Number of Pages Added to AHI vs Number of Pages Added to Buffer Pool.

AHI Miss Ratio

Percent of Searches which could not be resolved through AHI.

Rows Added Per Page

Number of Rows “Hashed” Per Each Page which needs to be added to AHI.

AHI ROI

How Many Successful Searches using AHI are performed per each row maintenance operation.

InnoDB AHI Usage

The InnoDB AHI Usage graph shows the search operations on the InnoDB adaptive hash index and its
efficiency. The adaptive hash index is a search hash designed to speed access to InnoDB pages in memory.
If the Hit Ratio is small, the working data set is larger than the buffer pool, the AHI should likely be
disabled.

If you do not see any metric, try running: SET GLOBAL innodb_monitor_enable=all; in the MySQL client.

InnoDB AHI Miss Ratio

InnoDB AHI Churn - Rows

InnoDB AHI Churn - Pages

INNODB CHANGE BUFFER

Change Buffer Max Size

The Maximum Size of Change Buffer (as Percent of Buffer Pool Size).

235 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

Change Buffer Max Size

The Maximum Size of Change Buffer (Bytes).

InnoDB Change Buffer Merge Load

Number of Average of Active Merge Buffer Operations in Process.

INNODB CONTENTION

InnoDB Thread Concurrency

If Enabled limits number of Threads allowed inside InnoDB Kernel at the same time.

InnoDB Commit Concurrency

If Enabled limits number of Threads allowed inside InnoDB Kernel at the same time during Commit Stage.

InnoDB Thread Sleep Delay

The Time the thread will Sleep before Re-Entering InnoDB Kernel if high contention.

InnoDB Adaptive Max Sleep Delay

If Set to Non-Zero Value InnoDB Thread Sleep Delay will be adjusted automatically depending on the load
up to the value specified by this variable.

InnoDB Concurrency Tickets

Number of low level operations InnoDB can do after it entered InnoDB kernel before it is forced to exit and
yield to another thread waiting.

InnoDB Spin Wait Delay

InnoDB Spin Wait Pause Multiplier

InnoDB Sync Spin Loops

InnoDB Contention - OS Waits

The InnoDB Contention - OS Waits graph shows the number of time an OS wait operation was required
while waiting to get the lock. This happens once the spin rounds are exhausted.

If you do not see any metric, try running: SET GLOBAL innodb_monitor_enable=all; in the MySQL client.

InnoDB Contention - Spin Rounds

The InnoDB Contention - Spin Rounds graph shows the number of spin rounds executed to get a lock. A
spin round is a fast retry to get the lock in a loop.

If you do not see any metric, try running: SET GLOBAL innodb_monitor_enable=all; in the MySQL client.

INNODB MISC

InnoDB Main Thread Utilization

The InnoDB Main Thread Utilization graph shows the portion of time the InnoDB main thread spent at
various task.

If you do not see any metric, try running: SET GLOBAL innodb_monitor_enable=all; in the MySQL client.

InnoDB Activity

The InnoDB Activity graph shows a measure of the activity of the InnoDB threads.

If you do not see any metric, try running: SET GLOBAL innodb_monitor_enable=all; in the MySQL client.

236 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

InnoDB Dedicated Server

InnoDB automatically optimized for Dedicated Server Environment (auto scaling cache and some other
variables).

InnoDB Sort Buffer Size

This Buffer is used for Building InnoDB Indexes using Sort algorithm.

InnoDB Stats Auto Recalc

Update Stats when Metadata Queried

Refresh InnoDB Statistics when meta-data queries by SHOW TABLE STATUS or INFORMATION_SCHEMA queries. If
Enabled can cause severe performance issues.

Index Condition Pushdown (ICP)

Index Condition Pushdown (ICP) is an optimization for the case where MySQL retrieves rows from a table
using an index. Without ICP, the storage engine traverses the index to locate rows in the base table and
returns them to the MySQL server which evaluates the WHERE condition for the rows. With ICP enabled, and
if parts of the WHERE condition can be evaluated by using only columns from the index, the MySQL server
pushes this part of the WHERE condition down to the storage engine. The storage engine then evaluates the
pushed index condition by using the index entry and only if this is satisfied is the row read from the table.
ICP can reduce the number of times the storage engine must access the base table and the number of
times the MySQL server must access the storage engine.

InnoDB Persistent Statistics

InnoDB Persistent Sample Pages

Number of Pages To Sample if Persistent Statistics are Enabled.

InnoDB Transient Sample Pages

Number of Pages To Sample if Persistent Statistics are Disabled.

INNODB ONLINE OPERATIONS (MARIADB)

InnoDB Defragmentation

The InnoDB Defragmentation graph shows the status information related to the InnoDB online
defragmentation feature of MariaDB for the optimize table command. To enable this feature, the variable
innodb-defragment must be set to 1 in the configuration file.

Currently available only on a MariaDB server.

InnoDB Online DDL

The InnoDB Online DDL graph shows the state of the online DDL (alter table) operations in InnoDB. The
progress metric is estimate of the percentage of the rows processed by the online DDL.

Currently available only on a MariaDB server.

MYSQL SUMMARY

MySQL Uptime

MySQL Uptime

The amount of time since the last restart of the MySQL server process.

Current QPS

Current QPS

237 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

Based on the queries reported by MySQL’s SHOW STATUS command, it is the number of statements executed
by the server within the last second. This variable includes statements executed within stored programs,
unlike the Questions variable. It does not count COM_PING or COM_STATISTICS commands.

File Handlers Used

Table Open Cache Miss Ratio

Table Open Cache Size

Table Definition Cache Size

MySQL Connections

Max Connections

Max Connections is the maximum permitted number of simultaneous client connections. By default, this is
151. Increasing this value increases the number of file descriptors that mysqld requires. If the required
number of descriptors are not available, the server reduces the value of Max Connections.

mysqld actually permits Max Connections + 1 clients to connect. The extra connection is reserved for use by
accounts that have the SUPER privilege, such as root.

Max Used Connections is the maximum number of connections that have been in use simultaneously since
the server started.

Connections is the number of connection attempts (successful or not) to the MySQL server.

MySQL Client Thread Activity

MySQL Active Threads

Threads Connected is the number of open connections, while Threads Running is the number of threads not
sleeping.

MySQL Handlers

MySQL Handlers

Handler statistics are internal statistics on how MySQL is selecting, updating, inserting, and modifying rows,
tables, and indexes.

This is in fact the layer between the Storage Engine and MySQL.

• read_rnd_next is incremented when the server performs a full table scan and this is a counter you
don’t really want to see with a high value.

• read_key is incremented when a read is done with an index.

• read_next is incremented when the storage engine is asked to ‘read the next index entry’. A high value
means a lot of index scans are being done.

Top Command Counters

Top Command Counters

The Com_ statement counter variables indicate the number of times each xxx statement has been executed.
There is one status variable for each type of statement. For example, Com_delete and Com_update count
DELETE and UPDATE statements, respectively. Com_delete_multi and Com_update_multi are similar but
apply to DELETE and UPDATE statements that use multiple-table syntax.

MySQL Network Traffic

MySQL Network Traffic

238 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

Here we can see how much network traffic is generated by MySQL. Outbound is network traffic sent from
MySQL and Inbound is network traffic MySQL has received.

NODE SUMMARY

System Uptime

The parameter shows how long a system has been up and running without a shut down or restart.

Load Average

The system load is a measurement of the computational work the system is performing. Each running
process either using or waiting for CPU resources adds 1 to the load.

RAM

RAM (Random Access Memory) is the hardware in a computing device where the operating system,
application programs and data in current use are kept so they can be quickly reached by the device’s
processor.

Memory Available

Percent of Memory Available

on Modern Linux Kernels amount of Memory Available for application is not the same as
Free+Cached+Buffers.

Virtual Memory

RAM + SWAP

Disk Space

Sum of disk space on all partitions.

It can be significantly over-reported in some installations.

Min Space Available

Lowest percent of the disk space available.

CPU Usage

The CPU time is measured in clock ticks or seconds. It is useful to measure CPU time as a percentage of the
CPU’s capacity, which is called the CPU usage.

CPU Saturation and Max Core Usage

When a system is running with maximum CPU utilization, the transmitting and receiving threads must all
share the available CPU. This will cause data to be queued more frequently to cope with the lack of CPU.
CPU Saturation may be measured as the length of a wait queue, or the time spent waiting on the queue.

Disk I/O and Swap Activity

Disk I/O includes read or write or input/output operations involving a physical disk. It is the speed with
which the data transfer takes place between the hard disk drive and RAM.

Swap Activity is memory management that involves swapping sections of memory to and from physical
storage.

Network Traffic

Network traffic refers to the amount of data moving across a network at a given point in time.

239 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

MySQL MyISAM/Aria Details

MYISAM KEY BUFFER PERFORMANCE

The Key Read Ratio ( Key_reads / Key_read_requests ) ratio should normally be less than 0.01.

The Key Write Ratio ( Key_writes / Key_write_requests ) ratio is usually near 1 if you are using mostly
updates and deletes, but might be much smaller if you tend to do updates that affect many rows at the
same time or if you are using the DELAY_KEY_WRITE table option.

ARIA PAGECACHE READS/WRITES

This graph is similar to InnoDB buffer pool reads/writes. aria-pagecache-buffer-size is the main cache for
the Aria storage engine. If you see high reads/writes (physical IO), i.e. reads are close to read requests
and/or writes are close to write requests you may need to increase the aria-pagecache-buffer-size (may
need to decrease other buffers: key_buffer_size , innodb_buffer_pool_size , etc.)

ARIA TRANSACTION LOG SYNCS

This is similar to InnoDB log file syncs. If you see lots of log syncs and want to relax the durability settings
you can change aria_checkpoint_interval (in seconds) from 30 (default) to a higher number. It is good to
look at the disk IO dashboard as well.

ARIA PAGECACHE BLOCKS

This graph shows the utilization for the Aria pagecache. This is similar to InnoDB buffer pool graph. If you
see all blocks are used you may consider increasing aria-pagecache-buffer-size (may need to decrease
other buffers: key_buffer_size , innodb_buffer_pool_size , etc.)

240 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

MySQL MyRocks Details

The MyRocks storage engine developed by Facebook based on the RocksDB storage engine is applicable to
systems which primarily interact with the database by writing data to it rather than reading from it.
RocksDB also features a good level of compression, higher than that of the InnoDB storage engine, which
makes it especially valuable when optimizing the usage of hard drives.

PMM collects statistics on the MyRocks storage engine for MySQL in the Metrics Monitor information for this
dashboard comes from the Information Schema tables.

241 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

Metrics

• MyRocks cache

• MyRocks cache data bytes R/W

• MyRocks cache index hit rate

• MyRocks cache index

• MyRocks cache filter hit rate

• MyRocks cache filter

• MyRocks cache data bytes inserted

• MyRocks bloom filter

• MyRocks memtable

• MyRocks memtable size

• MyRocks number of keys

• MyRocks cache L0/L1

• MyRocks number of DB ops

• MyRocks R/W

• MyRocks bytes read by iterations

• MyRocks write ops

• MyRocks WAL

• MyRocks number reseeks in iterations

• RocksDB row operations

• MyRocks file operations

• RocksDB stalls

• RocksDB stops/slowdowns

242 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

MySQL Instance Summary

MYSQL CONNECTIONS

Max Connections

Max Connections is the maximum permitted number of simultaneous client connections. By default, this is
151. Increasing this value increases the number of file descriptors that mysqld requires. If the required
number of descriptors are not available, the server reduces the value of Max Connections.

mysqld actually permits Max Connections + 1 clients to connect. The extra connection is reserved for use by
accounts that have the SUPER privilege, such as root.

Max Used Connections is the maximum number of connections that have been in use simultaneously since
the server started.

Connections is the number of connection attempts (successful or not) to the MySQL server.

MYSQL ABORTED CONNECTIONS

Aborted Connections

When a given host connects to MySQL and the connection is interrupted in the middle (for example due to
bad credentials), MySQL keeps that info in a system table (since 5.6 this table is exposed in
performance_schema ).

If the amount of failed requests without a successful connection reaches the value of max_connect_errors ,
mysqld assumes that something is wrong and blocks the host from further connection.

To allow connections from that host again, you need to issue the FLUSH HOSTS statement.

MYSQL CLIENT THREAD ACTIVITY

MySQL Active Threads

Threads Connected is the number of open connections, while Threads Running is the number of threads not
sleeping.

243 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

MYSQL THREAD CACHE

MySQL Thread Cache

The thread_cache_size variable sets how many threads the server should cache to reuse. When a client
disconnects, the client’s threads are put in the cache if the cache is not full. It is auto-sized in MySQL 5.6.8
and above (capped to 100). Requests for threads are satisfied by reusing threads taken from the cache if
possible, and only when the cache is empty is a new thread created.

• threads_created : The number of threads created to handle connections.

• threads_cached : The number of threads in the thread cache.

MYSQL SLOW QUERIES

MySQL Slow Queries

Slow queries are defined as queries being slower than the long_query_time setting. For example, if you
have long_query_time set to 3, all queries that take longer than 3 seconds to complete will show on this
graph.

MYSQL SELECT TYPES

MySQL Select Types

As with most relational databases, selecting based on indexes is more efficient than scanning an entire
table’s data. Here we see the counters for selects not done with indexes.

• Select Scan is how many queries caused full table scans, in which all the data in the table had to be
read and either discarded or returned.

• Select Range is how many queries used a range scan, which means MySQL scanned all rows in a
given range.

• Select Full Join is the number of joins that are not joined on an index, this is usually a huge
performance hit.

MYSQL SORTS

MySQL Sorts

Due to a query’s structure, order, or other requirements, MySQL sorts the rows before returning them. For
example, if a table is ordered 1 to 10 but you want the results reversed, MySQL then has to sort the rows to
return 10 to 1.

This graph also shows when sorts had to scan a whole table or a given range of a table to return the
results and which could not have been sorted via an index.

MYSQL TABLE LOCKS

Table Locks

MySQL takes a number of different locks for varying reasons. In this graph we see how many Table level
locks MySQL has requested from the storage engine. In the case of InnoDB, many times the locks could
actually be row locks as it only takes table level locks in a few specific cases.

It is most useful to compare Locks Immediate and Locks Waited. If Locks waited is rising, it means you
have lock contention. Otherwise, Locks Immediate rising and falling is normal activity.

MYSQL QUESTIONS

MySQL Questions

244 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

The number of statements executed by the server. This includes only statements sent to the server by clients
and not statements executed within stored programs, unlike the Queries used in the QPS calculation.

This variable does not count the following commands:

• COM_PING

• COM_STATISTICS

• COM_STMT_PREPARE

• COM_STMT_CLOSE

• COM_STMT_RESET

MYSQL NETWORK TRAFFIC

MySQL Network Traffic

Here we can see how much network traffic is generated by MySQL. Outbound is network traffic sent from
MySQL and Inbound is network traffic MySQL has received.

MYSQL NETWORK USAGE HOURLY

MySQL Network Usage Hourly

Here we can see how much network traffic is generated by MySQL per hour. You can use the bar graph to
compare data sent by MySQL and data received by MySQL.

MYSQL INTERNAL MEMORY OVERVIEW

System Memory: Total Memory for the system.

InnoDB Buffer Pool Data: InnoDB maintains a storage area called the buffer pool for caching data and
indexes in memory.

TokuDB Cache Size: Similar in function to the InnoDB Buffer Pool, TokuDB will allocate 50% of the
installed RAM for its own cache.

Key Buffer Size: Index blocks for MyISAM tables are buffered and are shared by all threads.
key_buffer_size is the size of the buffer used for index blocks.

Adaptive Hash Index Size: When InnoDB notices that some index values are being accessed very
frequently, it builds a hash index for them in memory on top of B-Tree indexes.

Query Cache Size: The query cache stores the text of a SELECT statement together with the corresponding
result that was sent to the client. The query cache has huge scalability problems in that only one thread can
do an operation in the query cache at the same time.

InnoDB Dictionary Size: The data dictionary is InnoDB’s internal catalog of tables. InnoDB stores the data
dictionary on disk, and loads entries into memory while the server is running.

InnoDB Log Buffer Size: The MySQL InnoDB log buffer allows transactions to run without having to write
the log to disk before the transactions commit.

TOP COMMAND COUNTERS

Top Command Counters

The Com_xxx statement counter variables indicate the number of times each xxx statement has been
executed. There is one status variable for each type of statement. For example, Com_delete and Com_update
count DELETE and UPDATE statements, respectively. Com_delete_multi and Com_update_multi are similar
but apply to DELETE and UPDATE statements that use multiple-table syntax.

245 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

TOP COMMAND COUNTERS HOURLY

Top Command Counters Hourly

The Com_xxx statement counter variables indicate the number of times each xxx statement has been
executed. There is one status variable for each type of statement. For example, Com_delete and Com_update
count DELETE and UPDATE statements, respectively. Com_delete_multi and Com_update_multi are similar
but apply to DELETE and UPDATE statements that use multiple-table syntax.

MYSQL HANDLERS

MySQL Handlers

Handler statistics are internal statistics on how MySQL is selecting, updating, inserting, and modifying rows,
tables, and indexes.

This is in fact the layer between the Storage Engine and MySQL.

• read_rnd_next is incremented when the server performs a full table scan and this is a counter you
don’t really want to see with a high value.

• read_key is incremented when a read is done with an index.

• read_next is incremented when the storage engine is asked to ‘read the next index entry’. A high value
means a lot of index scans are being done.

MYSQL QUERY CACHE MEMORY

MySQL Query Cache Memory

The query cache has huge scalability problems in that only one thread can do an operation in the query
cache at the same time. This serialization is true not only for SELECTs, but also for INSERT/UPDATE/
DELETE.

This also means that the larger the query_cache_size is set to, the slower those operations become. In
concurrent environments, the MySQL Query Cache quickly becomes a contention point, decreasing
performance. MariaDB and AWS Aurora have done work to try and eliminate the query cache contention in
their flavors of MySQL, while MySQL 8.0 has eliminated the query cache feature.

The recommended settings for most environments is to set:

• query_cache_type=0

• query_cache_size=0

While you can dynamically change these values, to completely remove the contention point you have to
restart the database.

MYSQL QUERY CACHE ACTIVITY

MySQL Query Cache Activity

The query cache has huge scalability problems in that only one thread can do an operation in the query
cache at the same time. This serialization is true not only for SELECTs, but also for INSERT/UPDATE/
DELETE.

This also means that the larger the query_cache_size is set to, the slower those operations become. In
concurrent environments, the MySQL Query Cache quickly becomes a contention point, decreasing
performance. MariaDB and AWS Aurora have done work to try and eliminate the query cache contention in
their flavors of MySQL, while MySQL 8.0 has eliminated the query cache feature.

246 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

The recommended settings for most environments is to set:

• query_cache_type=0

• query_cache_size=0

While you can dynamically change these values, to completely remove the contention point you have to
restart the database.

MYSQL TABLE OPEN CACHE STATUS

MySQL Table Open Cache Status

The recommendation is to set the table_open_cache_instances to a loose correlation to virtual CPUs,

keeping in mind that more instances means the cache is split more times. If you have a cache set to 500
but it has 10 instances, each cache will only have 50 cached.

The table_definition_cache and table_open_cache can be left as default as they are auto-sized MySQL 5.6
and above (i.e., do not set them to any value).

MYSQL OPEN TABLES

MySQL Open Tables

The recommendation is to set the table_open_cache_instances to a loose correlation to virtual CPUs,

keeping in mind that more instances means the cache is split more times. If you have a cache set to 500
but it has 10 instances, each cache will only have 50 cached.

The table_definition_cache and table_open_cache can be left as default as they are auto-sized MySQL 5.6
and above (i.e., do not set them to any value).

MYSQL TABLE DEFINITION CACHE

MySQL Table Definition Cache

The recommendation is to set the table_open_cache_instances to a loose correlation to virtual CPUs,

keeping in mind that more instances means the cache is split more times. If you have a cache set to 500
but it has 10 instances, each cache will only have 50 cached.

The table_definition_cache and table_open_cache can be left as default as they are auto-sized MySQL 5.6
and above (i.e., do not set them to any value).

247 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

MySQL Instances Compare

No description

248 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

MySQL Instances Overview

No description

249 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

MySQL Wait Event Analyses Details

This dashboard helps to analyze Performance Schema wait events. It plots the following metrics for the
chosen (one or more) wait events:

• Count - Performance Schema Waits

• Load - Performance Schema Waits

• Avg Wait Time - Performance Schema Waits

250 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

MySQL Performance Schema Details

The MySQL Performance Schema dashboard helps determine the efficiency of communicating with
Performance Schema. This dashboard contains the following metrics:

• Performance Schema file IO (events)

• Performance Schema file IO (load)

• Performance Schema file IO (Bytes)

• Performance Schema waits (events)

• Performance Schema waits (load)

• Index access operations (load)

• Table access operations (load)

• Performance Schema SQL and external locks (events)

• Performance Schema SQL and external locks (seconds)

251 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

MySQL Query Response Time Details

AVERAGE QUERY RESPONSE TIME

The Average Query Response Time graph shows information collected using the Response Time Distribution
plugin sourced from table INFORMATION_SCHEMA.QUERY_RESPONSE_TIME . It computes this value across all
queries by taking the sum of seconds divided by the count of queries.

QUERY RESPONSE TIME DISTRIBUTION

Query response time counts (operations) are grouped into three buckets:

• 100 ms - 1 s

• 1 s - 10 s

• > 10 s

AVERAGE QUERY RESPONSE TIME

Available only in Percona Server for MySQL, provides visibility of the split of READ vs WRITE query response
time.

READ QUERY RESPONSE TIME DISTRIBUTION

Available only in Percona Server for MySQL, illustrates READ query response time counts (operations)
grouped into three buckets:

• 100 ms - 1 s

• 1 s - 10 s

• > 10 s

252 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

WRITE QUERY RESPONSE TIME DISTRIBUTION

Available only in Percona Server for MySQL, illustrates WRITE query response time counts (operations)
grouped into three buckets:

• 100 ms - 1 s

• 1 s - 10 s

• > 10 s

253 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

MySQL Replication Summary

IO THREAD RUNNING

This metric shows if the IO Thread is running or not. It only applies to a secondary host.

SQL Thread is a process that runs on a secondary host in the replication environment. It reads the events
from the local relay log file and applies them to the secondary server.

Depending on the format of the binary log it can read query statements in plain text and re-execute them or
it can read raw data and apply them to the local host.

Possible values

Yes

The thread is running and is connected to a replication primary

No

The thread is not running because it is not launched yet or because an error has occurred connecting to
the primary host

Connecting

The thread is running but is not connected to a replication primary

No value

The host is not configured to be a replication secondary

IO Thread Running is one of the parameters that the command SHOW SLAVE STATUS returns.

SQL THREAD RUNNING

This metric shows if the SQL thread is running or not. It only applies to a secondary host.

Possible values

254 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

Yes

SQL Thread is running and is applying events from the relay log to the local secondary host

No

SQL Thread is not running because it is not launched yet or because of an error occurred while applying
an event to the local secondary host

REPLICATION ERROR NO

This metric shows the number of the last error in the SQL Thread encountered which caused replication to
stop.

One of the more common errors is Error: 1022 Duplicate Key Entry. In such a case replication is
attempting to update a row that already exists on the secondary. The SQL Thread will stop replication to
avoid data corruption.

READ ONLY

This metric indicates whether the host is configured to be in Read Only mode or not.

Possible values

Yes

The secondary host permits no client updates except from users who have the SUPER privilege or the
REPLICATION SLAVE privilege.

This kind of configuration is typically used for secondary hosts in a replication environment to avoid a
user can inadvertently or voluntarily modify data causing inconsistencies and stopping the replication
process.

No

The secondary host is not configured in Read Only mode.

MYSQL REPLICATION DELAY

This metric shows the number of seconds the secondary host is delayed in replication applying events
compared to when the primary host applied them, denoted by the Seconds_Behind_Master value, and only
applies to a secondary host.

Since the replication process applies the data modifications on the secondary asynchronously, it could
happen that the secondary replicates events after some time. The main reasons are:

• Network round trip time - high latency links will lead to non-zero replication lag values.

• Single threaded nature of replication channels - primary servers have the advantage of applying
changes in parallel, whereas secondary ones are only able to apply changes in serial, thus limiting their
throughput. In some cases Group Commit can help but is not always applicable.

• High number of changed rows or computationally expensive SQL - depending on the replication
format ( ROW vs STATEMENT ), significant changes to the database through high volume of rows
modified, or expensive CPU will all contribute to secondary servers lagging behind the primary.

Generally adding more CPU or Disk resources can alleviate replication lag issues, up to a point.

BINLOG SIZE

This metric shows the overall size of the binary log files, which can exist on both primary and secondary
servers.

255 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

The binary log (also known as the binlog) contains events that describe database changes: CREATE TABLE ,
ALTER TABLE , updates, inserts, deletes and other statements or database changes.

The binlog file is read by secondaries via their IO Thread process to replicate database changes
modification on the data and on the table structures. There can be more than one binlog file depending on
the binlog rotation policy (for example using the configuration variables max_binlog_size and
expire_logs_days ) or because of server reboots.

When planning the disk space, take care of the overall dimension of binlog files and adopt a good rotation
policy or think about having a separate mount point or disk to store the binlog data.

BINLOG DATA WRITTEN HOURLY

This metric shows the amount of data written hourly to the binlog files during the last 24 hours. This metric
can give you an idea of how big is your application in terms of data writes (creation, modification,
deletion).

BINLOG COUNT

This metric shows the overall count of binary log files, on both primary and secondary servers.

BINLOGS CREATED HOURLY

This metric shows the number of binlog files created hourly during the last 24 hours.

RELAY LOG SPACE

This metric shows the overall size of the relay log files. It only applies to a secondary host.

The relay log consists of a set of numbered files containing the events to be executed on the secondary host
to replicate database changes.

The relay log has the same format as the binlog.

There can be multiple relay log files depending on the rotation policy adopted (using the configuration
variable max_relay_log_size ).

As soon as the SQL thread completes to execute all events in the relay log file, the file is deleted.

If this metric contains a high value, the variable max_relay_log_file is high too. Generally, this not a
serious issue. If the value of this metric is constantly increased, the secondary is delaying too much in
applying the events.

Treat this metric in the same way as the MySQL Replication Delay metric.

RELAY LOG WRITTEN HOURLY

This metric shows the amount of data written hourly into relay log files during the last 24 hours.

256 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

MySQL Group Replication Summary

OVERVIEW

• PRIMARY Service

• Group Replication Service States

• Replication Group Members

• Replication Lag

• Replication Delay

• Transport Time

TRANSACTIONS

• Transaction Details

257 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

• Applied Transactions

• Sent Transactions

• Checked Transactions

• Rolled Back Transactions

• Transactions Row Validating

• Transactions in the Queue for Checking

• Received Transactions Queue

CONFLICTS

• Detected Conflicts

258 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

MySQL Table Details

LARGEST TABLES

Largest Tables by Row Count

The estimated number of rows in the table from information_schema.tables .

Largest Tables by Size

The size of the table components from information_schema.tables .

PIE

Total Database Size

The total size of the database: as data + index size, so freeable one.

Most Fragmented Tables by Freeable Size

The list of 5 most fragmented tables ordered by their freeable size

TABLE ACTIVITY

The next two graphs are available only for Percona Server and MariaDB and require userstat variable
turned on.

ROWS READ

The number of rows read from the table, shown for the top 5 tables.

ROWS CHANGED

The number of rows changed in the table, shown for the top 5 tables.

AUTO INCREMENT USAGE

The current value of an auto_increment column from information_schema , shown for the top 10 tables.

259 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

MySQL User Details

This dashboard requires Percona Server for MySQL 5.1+ or MariaDB 10.1/10.2 with XtraDB. Also
userstat should be enabled, for example with the SET GLOBAL userstat=1 statement. See Setting up
MySQL.

Data is displayed for the 5 top users.

Top Users by Connections Created

The number of times user’s connections connected using SSL to the server.

Top Users by Traffic

The number of bytes sent to the user’s connections.

Top Users by Rows Fetched/Read

The number of rows fetched by the user’s connections.

Top Users by Rows Updated

The number of rows updated by the user’s connections.

Top Users by Busy Time

The cumulative number of seconds there was activity on connections from the user.

Top Users by CPU Time

The cumulative CPU time elapsed, in seconds, while servicing connections of the user.

260 of 359 Percona LLC, © 2021

 5.4.6 MySQL Dashboards

MySQL TokuDB Details

No description

261 of 359 Percona LLC, © 2021

 5.4.7 MongoDB Dashboards

5.4.7 MongoDB Dashboards

MongoDB Cluster Summary

CURRENT CONNECTIONS PER SHARD

TCP connections (Incoming) in mongod processes.

TOTAL CONNECTIONS

Incoming connections to mongos nodes.

CURSORS PER SHARD

The Cursor is a MongoDB Collection of the document which is returned upon the find method execution.

MONGOS CURSORS

The Cursor is a MongoDB Collection of the document which is returned upon the find method execution.

OPERATIONS PER SHARD

Ops/sec, classified by legacy wire protocol type ( query , insert , update , delete , getmore ).

TOTAL MONGOS OPERATIONS

Ops/sec, classified by legacy wire protocol type ( query , insert , update , delete , getmore ).

CHANGE LOG EVENTS

Count, over last 10 minutes, of all types of configuration db changelog events.

OPLOG RANGE BY SET

Timespan ‘window’ between oldest and newest ops in the Oplog collection.

262 of 359 Percona LLC, © 2021

 5.4.7 MongoDB Dashboards

MongoDB Instance Summary

COMMAND OPERATIONS

Ops or Replicated Ops/sec classified by legacy wire protocol type ( query , insert , update , delete ,
getmore ). And (from the internal TTL threads) the docs deletes/sec by TTL indexes.

LATENCY DETAIL

Average latency of operations (classified by read, write, or (other) command)

CONNECTIONS

TCP connections (Incoming)

CURSORS

Open cursors. Includes idle cursors.

DOCUMENT OPERATIONS

Docs per second inserted, updated, deleted or returned. (not 1-to-1 with operation counts.)

QUEUED OPERATIONS

Operations queued due to a lock.

QUERY EFFICIENCY

Ratio of Documents returned or Index entries scanned / full documents scanned

SCANNED AND MOVED OBJECTS

This panel shows the number of objects (both data ( scanned_objects ) and index ( scanned )) as well as the
number of documents that were moved to a new location due to the size of the document growing. Moved
documents only apply to the MMAPv1 storage engine.

GETLASTERROR WRITE TIME

Legacy driver operation: Number of, and Sum of time spent, per second executing getLastError
commands to confirm write concern.

263 of 359 Percona LLC, © 2021

 5.4.7 MongoDB Dashboards

GETLASTERROR WRITE OPERATIONS

Legacy driver operation: Number of getLastError commands that timed out trying to confirm write
concern.

ASSERT EVENTS

This panel shows the number of assert events per second on average over the given time period. In most
cases assertions are trivial, but you would want to check your log files if this counter spikes or is
consistently high.

PAGE FAULTS

Unix or Window memory page faults. Not necessarily from MongoDB.

264 of 359 Percona LLC, © 2021

 5.4.7 MongoDB Dashboards

MongoDB Instances Overview

This dashboard provides basic information about MongoDB instances.

COMMAND OPERATIONS

Shows how many times a command is executed per second on average during the selected interval.

Look for peaks and drops and correlate them with other graphs.

CONNECTIONS

Keep in mind the hard limit on the maximum number of connections set by your distribution.

Anything over 5,000 should be a concern, because the application may not close connections correctly.

CURSORS

Helps identify why connections are increasing. Shows active cursors compared to cursors being
automatically killed after 10 minutes due to an application not closing the connection.

DOCUMENT OPERATIONS

When used in combination with Command Operations, this graph can help identify write amplification. For
example, when one insert or update command actually inserts or updates hundreds, thousands, or even
millions of documents.

265 of 359 Percona LLC, © 2021

 5.4.7 MongoDB Dashboards

QUEUED OPERATIONS

Any number of queued operations for long periods of time is an indication of possible issues. Find the cause
and fix it before requests get stuck in the queue.

GETLASTERROR WRITE TIME, GETLASTERROR WRITE OPERATIONS

This is useful for write-heavy workloads to understand how long it takes to verify writes and how many
concurrent writes are occurring.

ASSERTS

Asserts are not important by themselves, but you can correlate spikes with other graphs.

MEMORY FAULTS

Memory faults indicate that requests are processed from disk either because an index is missing or there is
not enough memory for the data set. Consider increasing memory or sharding out.

266 of 359 Percona LLC, © 2021

 5.4.7 MongoDB Dashboards

MongoDB Instances Compare

CONNECTIONS

No description

CURSORS

No description

LATENCY

Average latency of operations (classified by read, write, or (other) command)

SCAN RATIOS

Ratio of index entries scanned or whole docs scanned / number of documents returned

INDEX FILTERING EFFECTIVENESS

No description

REQUESTS

Ops/sec (classified by (legacy) wire protocol request type)

DOCUMENT OPERATIONS

Documents inserted/updated/deleted or returned per sec

QUEUED OPERATIONS

The number of operations that are currently queued and waiting for a lock

USED MEMORY

No description

267 of 359 Percona LLC, © 2021

 5.4.7 MongoDB Dashboards

MongoDB ReplSet Summary

REPLICATION LAG

MongoDB replication lag occurs when the secondary node cannot replicate data fast enough to keep up with
the rate that data is being written to the primary node. It could be caused by something as simple as
network latency, packet loss within your network, or a routing issue.

OPERATIONS - BY SERVICE NAME

Operations are classified by legacy wire protocol type (insert, update, and delete only).

MAX MEMBER PING TIME - BY SERVICE NAME

This metric can show a correlation with the replication lag value.

MAX HEARTBEAT TIME

Time span between now and last heartbeat from replicaset members.

ELECTIONS

Count of elections. Usually zero; 1 count by each healthy node will appear in each election. Happens when
the primary role changes due to either normal maintenance or trouble events.

OPLOG RECOVERY WINDOW - BY SERVICE NAME

Timespan ‘window’ between newest and the oldest op in the Oplog collection.

268 of 359 Percona LLC, © 2021

 5.4.7 MongoDB Dashboards

MongoDB InMemory Details

INMEMORY TRANSACTIONS

WiredTiger internal transactions

INMEMORY CAPACITY

Configured max and current size of the WiredTiger cache.

INMEMORY SESSIONS

Internal WiredTiger storage engine cursors and sessions currently open.

INMEMORY PAGES

Pages in the WiredTiger cache

INMEMORY CONCURRENCY TICKETS

A WT ‘ticket’ is assigned out for every operation running simultaneously in the WT storage engine. “Tickets
available” = hard coded high value - “Tickets Out”.

QUEUED OPERATIONS

Operations queued due to a lock

DOCUMENT CHANGES

Mixed metrics: Docs per second inserted, updated, deleted or returned on any type of node (primary or
secondary); + replicated write Ops/sec; + TTL deletes per second.

INMEMORY CACHE EVICTION

This panel shows the number of pages that have been evicted from the WiredTiger cache for the given time
period. The InMemory storage engine only evicts modified pages which signals a compaction of the data
and removal of the dirty pages.

269 of 359 Percona LLC, © 2021

 5.4.7 MongoDB Dashboards

SCANNED AND MOVED OBJECTS

This panel shows the number of objects (both data ( scanned_objects ) and index ( scanned )) as well as the
number of documents that were moved to a new location due to the size of the document growing. Moved
documents only apply to the MMAPv1 storage engine.

PAGE FAULTS

Unix or Window memory page faults. Not necessarily from MongoDB.

270 of 359 Percona LLC, © 2021

 5.4.7 MongoDB Dashboards

MongoDB MMAPv1 Details

DOCUMENT ACTIVITY

Docs per second inserted, updated, deleted or returned. Also showing replicated write ops and internal TTL
index deletes.

MMAPV1 LOCK WAIT TIME

Time spent per second waiting to acquire locks.

MMAPV1 PAGE FAULTS

Unix or Window memory page faults. Not necessarily from MongoDB.

MMAPV1 JOURNAL WRITE ACTIVITY

MB processed through the journal in memory.

MMAPV1 JOURNAL COMMIT ACTIVITY

MB committed to disk for the journal.

MMAPV1 BACKGROUND FLUSHING TIME

Average time in ms, over full uptime of mongod process, the MMAP background flushes have taken.

QUEUED OPERATIONS

Queue size of ops waiting to be submitted to storage engine layer. (see WiredTiger concurrency tickets for
number of ops being processed simultaneously in storage engine layer.)

CLIENT OPERATIONS

Ops and Replicated Ops/sec, classified by legacy wire protocol type ( query , insert , update , delete ,
getmore ).

271 of 359 Percona LLC, © 2021

 5.4.7 MongoDB Dashboards

SCANNED AND MOVED OBJECTS

This panel shows the number of objects (both data ( scanned_objects ) and index ( scanned )) as well as the
number of documents that were moved to a new location due to the size of the document growing. Moved
documents only apply to the MMAPv1 storage engine.

272 of 359 Percona LLC, © 2021

 5.4.7 MongoDB Dashboards

MongoDB WiredTiger Details

WIREDTIGER TRANSACTIONS

WiredTiger internal transactions

WIREDTIGER CACHE ACTIVITY

Data volume transferred per second between the WT cache and data files. Writes out always imply disk;
Reads are often from OS file buffer cache already in RAM, but disk if not.

WIREDTIGER BLOCK ACTIVITY

Data volume handled by the WT block manager per second

WIREDTIGER SESSIONS

Internal WT storage engine cursors and sessions currently open

WIREDTIGER CONCURRENCY TICKETS AVAILABLE

A WT ‘ticket’ is assigned out for every operation running simultaneously in the WT storage engine.
“Available” = hard-coded high value - “Out”.

QUEUED OPERATIONS

Operations queued due to a lock.

WIREDTIGER CHECKPOINT TIME

The time spent in WT checkpoint phase. Warning: This calculation averages the cyclical event (default: 1
min) execution to a per-second value.

WIREDTIGER CACHE EVICTION

Least-recently used pages being evicted due to WT cache becoming full.

WIREDTIGER CACHE CAPACITY

Configured max and current size of the WT cache.

273 of 359 Percona LLC, © 2021

 5.4.7 MongoDB Dashboards

WIREDTIGER CACHE PAGES

WIREDTIGER LOG OPERATIONS

WT internal write-ahead log operations.

WIREDTIGER LOG ACTIVITY

Data volume moved per second in WT internal write-ahead log.

WIREDTIGER LOG RECORDS

Number of records appended per second in WT internal log.

DOCUMENT CHANGES

Mixed metrics: Docs per second inserted, updated, deleted or returned on any type of node (primary or
secondary); + replicated write Ops/sec; + TTL deletes per second.

SCANNED AND MOVED OBJECTS

This panel shows the number of objects (both data ( scanned_objects ) and index ( scanned )) as well as the
number of documents that were moved to a new location due to the size of the document growing. Moved
documents only apply to the MMAPv1 storage engine.

PAGE FAULTS

Unix or Window memory page faults. Not necessarily from MongoDB.

274 of 359 Percona LLC, © 2021

 5.4.8 PostgreSQL Dashboards

5.4.8 PostgreSQL Dashboards

PostgreSQL Instances Overview

CONNECTED

Reports whether PMM Server can connect to the PostgreSQL instance.

VERSION

The version of the PostgreSQL instance.

SHARED BUFFERS

Defines the amount of memory the database server uses for shared memory buffers. Default is 128MB .
Guidance on tuning is 25% of RAM, but generally doesn’t exceed 40% .

275 of 359 Percona LLC, © 2021

 5.4.8 PostgreSQL Dashboards

DISK-PAGE BUFFERS

The setting wal_buffers defines how much memory is used for caching the write-ahead log entries.
Generally this value is small ( 3% of shared_buffers value), but it may need to be modified for heavily
loaded servers.

MEMORY SIZE FOR EACH SORT

The parameter work_mem defines the amount of memory assigned for internal sort operations and hash
tables before writing to temporary disk files. The default is 4MB .

DISK CACHE SIZE

PostgreSQL’s effective_cache_size variable tunes how much RAM you expect to be available for disk
caching. Generally adding Linux free+cached will give you a good idea. This value is used by the query
planner whether plans will fit in memory, and when defined too low, can lead to some plans rejecting
certain indexes.

AUTOVACUUM

Whether autovacuum process is enabled or not. Generally the solution is to vacuum more often, not less.

POSTGRESQL CONNECTIONS

Max Connections

The maximum number of client connections allowed. Change this value with care as there are some
memory resources that are allocated on a per-client basis, so setting max_connections higher will
generally increase overall PostgreSQL memory usage.

Connections

The number of connection attempts (successful or not) to the PostgreSQL server.

Active Connections

The number of open connections to the PostgreSQL server.

POSTGRESQL TUPLES

Tuples

The total number of rows processed by PostgreSQL server: fetched, returned, inserted, updated, and
deleted.

Read Tuple Activity

The number of rows read from the database: as returned so fetched ones.

Tuples Changed per 5 min

The number of rows changed in the last 5 minutes: inserted, updated, and deleted ones.

POSTGRESQL TRANSACTIONS

Transactions

The total number of transactions that have been either been committed or rolled back.

Duration of Transactions

Maximum duration in seconds any active transaction has been running.

276 of 359 Percona LLC, © 2021

 5.4.8 PostgreSQL Dashboards

TEMP FILES

Number of Temp Files

The number of temporary files created by queries.

Size of Temp files

The total amount of data written to temporary files by queries in bytes.

All temporary files are taken into account by these two gauges, regardless of why the temporary file was
created (e.g., sorting or hashing), and regardless of the log_temp_files setting.

CONFLICTS AND LOCKS

Conflicts/Deadlocks

The number of queries canceled due to conflicts with recovery in the database (due to dropped
tablespaces, lock timeouts, old snapshots, pinned buffers, or deadlocks).

Number of Locks

The number of deadlocks detected by PostgreSQL.

BUFFERS AND BLOCKS OPERATIONS

Operations with Blocks

The time spent reading and writing data file blocks by back ends, in milliseconds.

Capturing read and write time statistics is possible only if track_io_timing setting is enabled. This can be
done either in configuration file or with the following query executed on the running system:

ALTER SYSTEM SET track_io_timing=ON;

SELECT pg_reload_conf();

Buffers

The number of buffers allocated by PostgreSQL.

CANCELED QUERIES

The number of queries that have been canceled due to dropped tablespaces, lock timeouts, old snapshots,
pinned buffers, and deadlocks.

Data shown by this gauge are based on the pg_stat_database_conflicts view.

CACHE HIT RATIO

The number of times disk blocks were found already in the buffer cache, so that a read was not necessary.

This only includes hits in the PostgreSQL buffer cache, not the operating system’s file system cache.

CHECKPOINT STATS

The total amount of time that has been spent in the portion of checkpoint processing where files are either
written or synchronized to disk, in milliseconds.

POSTGRESQL SETTINGS

The list of all settings of the PostgreSQL server.

277 of 359 Percona LLC, © 2021

 5.4.8 PostgreSQL Dashboards

SYSTEM SUMMARY

This section contains the following system parameters of the PostgreSQL server: CPU Usage, CPU
Saturation and Max Core Usage, Disk I/O Activity, and Network Traffic.

278 of 359 Percona LLC, © 2021

 5.4.8 PostgreSQL Dashboards

PostgreSQL Instance Summary

NUMBER OF TEMP FILES

Cumulative number of temporary files created by queries in this database since service start. All temporary
files are counted, regardless of why the temporary file was created (e.g., sorting or hashing), and
regardless of the log_temp_files setting.

SIZE OF TEMP FILES

Cumulative amount of data written to temporary files by queries in this database since service start. All
temporary files are counted, regardless of why the temporary file was created, and regardless of the
log_temp_files setting.

TEMP FILES ACTIVITY

Number of temporary files created by queries in this database. All temporary files are counted, regardless
of why the temporary file was created (e.g., sorting or hashing), and regardless of the log_temp_files
setting.

TEMP FILES UTILIZATION

Total amount of data written to temporary files by queries in this database. All temporary files are counted,
regardless of why the temporary file was created, and regardless of the log_temp_files setting.

CANCELED QUERIES

Based on pg_stat_database_conflicts view

279 of 359 Percona LLC, © 2021

 5.4.8 PostgreSQL Dashboards

PostgreSQL Instances Compare

No description

280 of 359 Percona LLC, © 2021

 5.4.9 ProxySQL Dashboards

5.4.9 ProxySQL Dashboards

ProxySQL Instance Summary

NETWORK TRAFFIC

Network traffic refers to the amount of data moving across a network at a given point in time.

281 of 359 Percona LLC, © 2021

 5.4.10 HA Dashboards

5.4.10 HA Dashboards

PXC/Galera Node Summary

GALERA REPLICATION LATENCY

Shows figures for the replication latency on group communication. It measures latency from the time point
when a message is sent out to the time point when a message is received. As replication is a group
operation, this essentially gives you the slowest ACK and longest RTT in the cluster.

GALERA REPLICATION QUEUES

Shows the length of receive and send queues.

GALERA CLUSTER SIZE

Shows the number of members currently connected to the cluster.

GALERA FLOW CONTROL

Shows the number of FC_PAUSE events sent/received. They are sent by a node when its replication queue
gets too full. If a node is sending out FC messages it indicates a problem.

GALERA PARALLELIZATION EFFICIENCY

Shows the average distances between highest and lowest seqno that are concurrently applied, committed
and can be possibly applied in parallel (potential degree of parallelization).

GALERA WRITING CONFLICTS

Shows the number of local transactions being committed on this node that failed certification (some other
node had a commit that conflicted with ours) – client received deadlock error on commit and also the
number of local transactions in flight on this node that were aborted because they locked something an
applier thread needed – deadlock error anywhere in an open transaction. Spikes in the graph may indicate
writing to the same table potentially the same rows from 2 nodes.

282 of 359 Percona LLC, © 2021

 5.4.10 HA Dashboards

AVAILABLE DOWNTIME BEFORE SST REQUIRED

Shows for how long the node can be taken out of the cluster before SST is required. SST is a full state
transfer method.

GALERA WRITESET COUNT

Shows the count of transactions received from the cluster (any other node) and replicated to the cluster
(from this node).

GALERA WRITESET SIZE

Shows the average transaction size received/replicated.

GALERA WRITESET TRAFFIC

Shows the bytes of data received from the cluster (any other node) and replicated to the cluster (from this
node).

GALERA NETWORK USAGE HOURLY

Shows the bytes of data received from the cluster (any other node) and replicated to the cluster (from this
node).

283 of 359 Percona LLC, © 2021

 5.4.10 HA Dashboards

PXC/Galera Cluster Summary

No description

284 of 359 Percona LLC, © 2021

 5.4.10 HA Dashboards

PXC/Galera Nodes Compare

$CLUSTER - GALERA CLUSTER SIZE

Shows the number of members currently connected to the cluster.

285 of 359 Percona LLC, © 2021

 5.4.10 HA Dashboards

HAProxy Instance Summary

286 of 359 Percona LLC, © 2021

 5.5 Commands

5.5 Commands

5.5.1 Commands

• pmm-admin – Command line tool for configuring and administering PMM

• pmm-agent – Daemon process, communicating between PMM Client and PMM Server

287 of 359 Percona LLC, © 2021

 5.5.2 pmm-admin - PMM Administration Tool

5.5.2 pmm-admin - PMM Administration Tool

NAME

pmm-admin - Administer PMM

SYNOPSIS

pmm-admin [FLAGS]

pmm-admin config [FLAGS] --server-url=server-url

pmm-admin add DATABASE [FLAGS] [NAME] [ADDRESS]

DATABASE:= [MongoDB | MySQL | PostgreSQL | ProxySQL]

pmm-admin add external [FLAGS] [NAME] [ADDRESS] (CAUTION: Technical preview feature)

pmm-admin add haproxy [FLAGS] [NAME]

pmm-admin add external [FLAGS] [NAME] [ADDRESS]

pmm-admin add external-serverless [FLAGS] [NAME] [ADDRESS]

pmm-admin remove [FLAGS] service-type [service-name]

pmm-admin register [FLAGS] [node-address] [node-type] [node-name]

pmm-admin list [FLAGS] [node-address]

pmm-admin status [FLAGS] [node-address]

pmm-admin summary [FLAGS] [node-address]

pmm-admin annotate [--node|--service] [--tags <tags>] [node-name|service-name]

pmm-admin help [COMMAND]

DESCRIPTION

pmm-admin is a command-line tool for administering PMM using a set of COMMAND keywords and
associated FLAGS.

PMM communicates with the PMM Server via a PMM agent process.

COMMON FLAGS

-h , --help

Show help and exit.

--help-long

Show extended help and exit.

--help-man

Generate man page. (Use pmm-admin --help-man | man -l - to view.)

288 of 359 Percona LLC, © 2021

 5.5.2 pmm-admin - PMM Administration Tool

--debug

Enable debug logging.

--trace

Enable trace logging (implies debug).

--json

Enable JSON output.

--version

Show the application version and exit.

--server-url=server-url

PMM Server URL in https://username:password@pmm-server-host/ format.

--server-insecure-tls

Skip PMM Server TLS certificate validation.

--group=<group-name>

Group name for external services. Default: external

COMMANDS

GENERAL COMMANDS

pmm-admin help [COMMAND]

Show help for COMMAND .

INFORMATION COMMANDS

pmm-admin list --server-url=server-url [FLAGS]

Show Services and Agents running on this Node, and the agent mode (push/pull).

pmm-admin status --server-url=server-url [FLAGS]

Show the following information about a local pmm-agent, and its connected server and clients:

• Agent: Agent ID, Node ID.

• PMM Server: URL and version.

• PMM Client: connection status, time drift, latency, vmagent status, pmm-admin version.

• Agents: Agent ID path and client name.

FLAGS:

--wait=<period><unit>

Time to wait for a successful response from pmm-agent. period is an integer. unit is one of ms for
milliseconds, s for seconds, m for minutes, h for hours.

289 of 359 Percona LLC, © 2021

 5.5.2 pmm-admin - PMM Administration Tool

pmm-admin summary --server-url=server-url [FLAGS]

Creates an archive file in the current directory with default file name
summary_<hostname>_<year>_<month>_<date>_<hour>_<minute>_<second>.zip . The contents are two
directories, client and server containing diagnostic text files.

FLAGS:

--filename="filename"

The Summary Archive filename.

--skip-server

Skip fetching logs.zip from PMM Server.

--pprof

Include performance profiling data in the summary.

CONFIGURATION COMMANDS

pmm-admin config

pmm-admin config [FLAGS] [node-address] [node-type] [node-name]

Configure a local pmm-agent .

FLAGS:

--node-id=node-id

Node ID (default is auto-detected).

--node-model=node-model

Node model

--region=region

Node region

--az=availability-zone

Node availability zone

--metrics-mode=mode

Metrics flow mode for agents node-exporter. Allowed values: - auto : chosen by server (default) -
push : agent will push metrics - pull : server scrapes metrics from agent

pmm-admin register

pmm-admin register [FLAGS] [node-address] [node-type] [node-name]

Register the current Node with the PMM Server.

--server-url=server-url

PMM Server URL in https://username:password@pmm-server-host/ format.

290 of 359 Percona LLC, © 2021

 5.5.2 pmm-admin - PMM Administration Tool

--machine-id="/machine_id/9812826a1c45454a98ba45c56cc4f5b0"

Node machine-id (default is auto-detected).

--distro="linux"

Node OS distribution (default is auto-detected).

--container-id=container-id

Container ID.

--container-name=container-name

Container name.

--node-model=node-model

Node model.

--region=region

Node region.

--az=availability-zone

Node availability zone.

--custom-labels=labels

Custom user-assigned labels.

pmm-admin remove

pmm-admin remove [FLAGS] service-type [service-name]

Remove Service from monitoring.

--service-id=service-id

Service ID.

--force

Remove service with that name or ID and all dependent services and agents.

When you remove a service, collected data remains on PMM Server for the specified retention period.

pmm-admin annotate

pmm-admin annotate [--node|--service] <annotation> [--tags <tags>] [--node-name=<node>] [--service-

name=<service>]

Annotate an event. (Read more)

<annotation>

The annotation string. If it contains spaces, it should be quoted.

291 of 359 Percona LLC, © 2021

 5.5.2 pmm-admin - PMM Administration Tool

--node

Annotate the current node or that specified by --node-name .

--service

Annotate all services running on the current node, or that specified by --service-name .

--tags

A quoted string that defines one or more comma-separated tags for the annotation. Example: "tag
1,tag 2" .

--node-name

The node name being annotated.

--service-name

The service name being annotated.

Combining flags

Flags may be combined as shown in the following examples.

--node

current node

--node-name

node with name

--node --node-name=NODE_NAME

node with name

--node --service-name

current node and service with name

--node --node-name --service-name

node with name and service with name

--node --service

current node and all services of current node

-node --node-name --service --service-name

service with name and node with name

--service

all services of the current node

--service-name

service with name

292 of 359 Percona LLC, © 2021

 5.5.2 pmm-admin - PMM Administration Tool

--service --service-name

service with name

--service --node-name

all services of current node and node with name

--service-name --node-name

service with name and node with name

--service --service-name -node-name

service with name and node with name

If node or service name is specified, they are used instead of other parameters.

DATABASE COMMANDS

MongoDB

pmm-admin add mongodb [FLAGS] [node-name] [node-address]

Add MongoDB to monitoring.

FLAGS:

--node-id=node-id

Node ID (default is auto-detected).

--pmm-agent-id=pmm-agent-id

The pmm-agent identifier which runs this instance (default is auto-detected).

--username=username

MongoDB username.

--password=password

MongoDB password.

--query-source=profiler

Source of queries, one of: profiler , none (default: profiler ).

--environment=environment

Environment name.

--cluster=cluster

Cluster name.

--replication-set=replication-set

Replication set name.

293 of 359 Percona LLC, © 2021

 5.5.2 pmm-admin - PMM Administration Tool

--custom-labels=custom-labels

Custom user-assigned labels.

--skip-connection-check

Skip connection check.

--tls

Use TLS to connect to the database.

--tls-skip-verify

Skip TLS certificates validation.

--tls-certificate-key-file=PATHTOCERT

Path to TLS certificate file.

--tls-certificate-key-file-password=IFPASSWORDTOCERTISSET

Password for TLS certificate file.

--tls-ca-file=PATHTOCACERT

Path to certificate authority file.

--metrics-mode=mode

Metrics flow mode for agents node-exporter. Allowed values: - auto : chosen by server (default) -
push : agent will push metrics - pull : server scrapes metrics from agent

MySQL

pmm-admin add mysql [FLAGS] node-name node-address | [--name=service-name] --address=address[:port] |

--socket

Add MySQL to monitoring.

FLAGS:

--address

MySQL address and port (default: 127.0.0.1:3306).

--socket=socket

Path to MySQL socket. (Find the socket path with mysql -u root -p -e "select @@socket" .)

--node-id=node-id

Node ID (default is auto-detected).

--pmm-agent-id=pmm-agent-id

The pmm-agent identifier which runs this instance (default is auto-detected).

294 of 359 Percona LLC, © 2021

 5.5.2 pmm-admin - PMM Administration Tool

--username=username

MySQL username.

--password=password

MySQL password.

--query-source=slowlog

Source of SQL queries, one of: slowlog , perfschema , none (default: slowlog ).

--size-slow-logs=N

Rotate slow log file at this size. If 0 , use server-defined default. Negative values disable log rotation. A
unit suffix must be appended to the number and can be one of:

• KB , kB , MB , mB , GB , gB , TB , tB for base 10 units (1000, 1000000, etc);

• KiB , MiB , GiB , TiB for base 2 units (1024, 1048576, etc).

--disable-queryexamples

Disable collection of query examples.

--disable-tablestats

Disable table statistics collection.

Excluded collectors for low-resolution time intervals:

• --collect.auto_increment.columns

• --collect.info_schema.tables

• --collect.info_schema.tablestats

• --collect.perf_schema.indexiowaits

• --collect.perf_schema.tableiowaits

• --collect.perf_schema.file_instances

Excluded collectors for medium-resolution time intervals:

• --collect.perf_schema.tablelocks

--disable-tablestats-limit=disable-tablestats-limit

Table statistics collection will be disabled if there are more than specified number of tables (default:
server-defined). 0=no limit. Negative value disables collection.

--environment=environment

Environment name.

--cluster=cluster

Cluster name.

--replication-set=replication-set

Replication set name.

295 of 359 Percona LLC, © 2021

 5.5.2 pmm-admin - PMM Administration Tool

--custom-labels=custom-labels

Custom user-assigned labels.

--skip-connection-check

Skip connection check.

--tls

Use TLS to connect to the database.

--tls-skip-verify

Skip TLS certificates validation.

--tls-cert-file=PATHTOCERT

Path to TLS client certificate file.

--tls-key=PATHTOCERTKEY

Key for TLS client certificate file.

--tls-ca-file=PATHTOCACERT

Path to certificate authority file.

--ssl-ca=PATHTOCACERT

The path name of the Certificate Authority (CA) certificate file. If used must specify the same certificate
used by the server. (-ssl-capath is similar but specifies the path name of a directory of CA certificate
files.)

--ssl-cert=PATHTOCERTKEY

The path name of the client public key certificate file.

--ssl-key

The path name of the client private key file.

--ssl-skip-verify

Skip SSL certificate verification.

--metrics-mode=mode

Metrics flow mode for agents node-exporter. Allowed values: - auto : chosen by server (default) -
push : agent will push metrics - pull : server scrapes metrics from agent

PostgreSQL

pmm-admin add postgresql [FLAGS] [node-name] [node-address]

Add PostgreSQL to monitoring.

FLAGS:

296 of 359 Percona LLC, © 2021

 5.5.2 pmm-admin - PMM Administration Tool

--node-id=<node id>

Node ID (default is auto-detected).

--pmm-agent-id=<pmm agent id>

The pmm-agent identifier which runs this instance (default is auto-detected).

--username=<username>

PostgreSQL username.

--password=<password>

PostgreSQL password.

--query-source=<query source>

Source of SQL queries, one of: pgstatements , pgstatmonitor , none (default: pgstatements ).

--environment=<environment>

Environment name.

--cluster=<cluster>

Cluster name.

--replication-set=<replication set>

Replication set name

--custom-labels=<custom labels>

Custom user-assigned labels.

--skip-connection-check

Skip connection check.

--tls

Use TLS to connect to the database.

--tls-skip-verify

Skip TLS certificates validation.

--metrics-mode=mode

Metrics flow mode for agents node-exporter. Allowed values: - auto : chosen by server (default) -
push : agent will push metrics - pull : server scrapes metrics from agent

ProxySQL

pmm-admin add proxysql [FLAGS] [node-name] [node-address]

Add ProxySQL to monitoring.

FLAGS:

297 of 359 Percona LLC, © 2021

 5.5.2 pmm-admin - PMM Administration Tool

--node-id=node-id

Node ID (default is auto-detected).

--pmm-agent-id=pmm-agent-id

The pmm-agent identifier which runs this instance (default is auto-detected).

--username=username

ProxySQL username.

--password=password

ProxySQL password.

--environment=environment

Environment name.

--cluster=cluster

Cluster name.

--replication-set=replication-set

Replication set name.

--custom-labels=custom-labels

Custom user-assigned labels.

--skip-connection-check

Skip connection check.

--tls

Use TLS to connect to the database.

--tls-skip-verify

Skip TLS certificates validation.

--metrics-mode=mode

Metrics flow mode for agents node-exporter. Allowed values: - auto : chosen by server (default) -
push : agent will push metrics - pull : server scrapes metrics from agent

--disable-collectors

Comma-separated list of collector names to exclude from exporter

HAProxy

pmm-admin add haproxy [FLAGS] [NAME]

Add HAProxy to monitoring.

FLAGS:

298 of 359 Percona LLC, © 2021

 5.5.2 pmm-admin - PMM Administration Tool

--server-url=SERVER-URL

PMM Server URL in https://username:password@pmm-server-host/ format

--server-insecure-tls

Skip PMM Server TLS certificate validation.

--username=USERNAME

HAProxy username.

--password=PASSWORD

HAProxy password.

--scheme=SCHEME

Scheme to generate URI to exporter metrics endpoints (http or https).

--metrics-path=METRICS-PATH

Path under which metrics are exposed, used to generate URI (default: /metrics).

--listen-port=LISTEN-PORT

Listen port of haproxy exposing the metrics for scraping metrics (Required).

--service-node-id=SERVICE-NODE-ID

Node ID where service runs (default is auto-detected).

--environment=ENVIRONMENT

Environment name like ‘production’ or ‘qa’.

--cluster=CLUSTER

Cluster name.

--replication-set=REPLICATION-SET

Replication set name.

--custom-labels=CUSTOM-LABELS

Custom user-assigned labels. Example: region=east,app=app1.

--metrics-mode=MODE

Metrics flow mode for agents node-exporter. Allowed values: - auto : chosen by server (default) -
push : agent will push metrics - pull : server scrapes metrics from agent

--skip-connection-check

Skip connection check.

299 of 359 Percona LLC, © 2021

 5.5.2 pmm-admin - PMM Administration Tool

OTHER COMMANDS

pmm-admin add external [FLAGS]

Add External source of data (like a custom exporter running on a port) to the monitoring

FLAGS:

--service-name="current-hostname"

Service name (autodetected defaults to the hostname where pmm-admin is running)

--agent-node-id=AGENT-NODE-ID

Node ID where agent runs (default is autodetected)

--username=USERNAME

External username

--password=PASSWORD

External password

--scheme=http or https

Scheme to generate URI to exporter metrics endpoints

--metrics-path=/metrics

Path under which metrics are exposed, used to generate URI.

--listen-port=LISTEN-PORT

Listen port of external exporter for scraping metrics. (Required)

--service-node-id=SERVICE-NODE-ID

Node ID where service runs (default is autodetected)

--environment=prod

Environment name like ‘production’ or ‘qa’

--cluster=east-cluster

Cluster name

--replication-set=rs1

Replication set name

--custom-labels=CUSTOM-LABELS

Custom user-assigned labels. Example: region=east,app=app1

--metrics-mode=auto

Metrics flow mode, can be push : agent will push metrics, pull : server scrape metrics from agent or
auto : chosen by server.

300 of 359 Percona LLC, © 2021

 5.5.2 pmm-admin - PMM Administration Tool

--group="external"

Group name of external service (default: external)

pmm-admin add external-serverless [FLAGS]

Add External Service on Remote node to monitoring.

Usage example: sudo pmm-admin add external-serverless --url=http://1.2.3.4:9093/metrics

Also, individual parameters can be set instead of --url like: sudo pmm-admin add external-serverless --
scheme=http --host=1.2.3.4 --listen-port=9093 --metrics-path=/metrics --container-name=ddd --
external-name=e125

Notice that some parameters are mandatory depending on the context. For example, if you specify --url ,
--schema and other related parameters are not mandatory but, if you specify --host you must provide
all other parameters needed to build the destination URL or even you can specify --address instead of
host and port as individual parameters.

FLAGS:

--url=URL

Full URL to exporter metrics endpoints

--scheme=https

Scheme to generate URL to exporter metrics endpoints

--username=USERNAME

External username

--password=PASSWORD

External password

--address=1.2.3.4:9000

External exporter address and port

--host=1.2.3.4

External exporters hostname or IP address

--listen-port=9999

Listen port of external exporter for scraping metrics.

--metrics-path=/metrics

Path under which metrics are exposed, used to generate URL.

--environment=testing

Environment name

--cluster=CLUSTER

Cluster name

301 of 359 Percona LLC, © 2021

 5.5.2 pmm-admin - PMM Administration Tool

--replication-set=rs1

Replication set name

--custom-labels='app=myapp,region=s1'

Custom user-assigned labels

--group="external"

Group name of external service (default: external)

--machine-id=MACHINE-ID

Node machine-id

--distro=DISTRO

Node OS distribution

--container-id=CONTAINER-ID

Container ID

--container-name=CONTAINER-NAME

Container name

--node-model=NODE-MODEL

Node model

--region=REGION

Node region

--az=AZ

Node availability zone

EXAMPLES

pmm-admin add mysql --query-source=slowlog --username=pmm --password=pmm sl-mysql 127.0.0.1:3306

MySQL Service added.

Service ID : /service_id/a89191d4-7d75-44a9-b37f-a528e2c4550f
Service name: sl-mysql

pmm-admin add mysql --username=pmm --password=pmm --service-name=ps-mysql --host=127.0.0.1 --

port=3306

pmm-admin status
pmm-admin status --wait=30s

Agent ID: /agent_id/c2a55ac6-a12f-4172-8850-4101237a4236

Node ID : /node_id/29b2cc24-3b90-4892-8d7e-4b44258d9309
PMM Server:

302 of 359 Percona LLC, © 2021

 5.5.2 pmm-admin - PMM Administration Tool

URL : https://x.x.x.x:443/
Version: 2.5.0
PMM Client:
Connected : true
Time drift: 2.152715ms
Latency : 465.658µs
pmm-admin version: 2.5.0
pmm-agent version: 2.5.0
Agents:
/agent_id/aeb42475-486c-4f48-a906-9546fc7859e8 mysql_slowlog_agent Running

DISABLE COLLECTORS

pmm-admin add mysql --disable-collectors='heartbeat,global_status,info_schema.innodb_cmp' --

username=pmm --password=pmm --service-name=db1-mysql --host=127.0.0.1 --port=3306

For other collectors that you can disable with the --disable-collectors option, please visit the official
repositories for each exporter:

• node_exporter

• mysqld_exporter

• mongodb_exporter

• postgres_exporter

• proxysql_exporter

303 of 359 Percona LLC, © 2021

 5.5.3 pmm-agent - PMM Client agent

5.5.3 pmm-agent - PMM Client agent

NAME

pmm-agent - The PMM Client daemon program

SYNOPSIS

pmm-agent [command] [options]

DESCRIPTION

pmm-agent, part of the PMM Client package, runs as a daemon process on all monitored hosts.

COMMANDS

pmm-agent run

Run pmm-agent (default)

pmm-agent setup [node-address] [node-type] [node-name]

Configure local pmm-agent

pmm-agent help [command]

Show help (for command) and exit

304 of 359 Percona LLC, © 2021

 5.5.3 pmm-agent - PMM Client agent

OPTIONS AND ENVIRONMENT

Most options can be set via environment variables (shown in parentheses).

305 of 359 Percona LLC, © 2021

 5.5.3 pmm-agent - PMM Client agent

Option Environment variable Description

--server- PMM_AGENT_SERVER_PASSWORD Password to connect to PMM Server.

password=SERVER-
PASSWORD

--server- PMM_AGENT_SERVER_USERNAME Username to connect to PMM Server.

username=SERVER-
USERNAME

--server- PMM_AGENT_SERVER_ADDRESS PMM Server address and port

address=host:port number.

--server-insecure-tls PMM_AGENT_SERVER_INSECURE_TLS Skip PMM Server TLS certificate

validation.

--az=AZ PMM_AGENT_SETUP_AZ Node availability zone.

--config-file=path_to/ PMM_AGENT_CONFIG_FILE Configuration file path and name.

pmm-agent.yaml

--container- PMM_AGENT_SETUP_CONTAINER_ID Container ID.

id=CONTAINER-ID

--container- PMM_AGENT_SETUP_CONTAINER_NAME Container name.

name=CONTAINER-NAME

--debug PMM_AGENT_DEBUG Enable debug output.

--distro=distro PMM_AGENT_SETUP_DISTRO Node OS distribution (default is auto-

detected).

--force PMM_AGENT_SETUP_FORCE Remove Node with that name and all

dependent Services and Agents (if
existing).

--id=/agent_id/... PMM_AGENT_ID ID of this pmm-agent.

--listen-address=LISTEN- PMM_AGENT_LISTEN_ADDRESS Agent local API address.

ADDRESS

--listen-port=LISTEN- PMM_AGENT_LISTEN_PORT Agent local API port.

PORT

--machine-id=machine-id PMM_AGENT_SETUP_MACHINE_ID Node machine ID (default is auto-

detected).

--metrics-mode=auto PMM_AGENT_SETUP_METRICS_MODE Metrics flow mode for agents node-

exporter. Can be push (agent will
push metrics), pull (server scrapes
metrics from agent) or auto (chosen
by server).

--node-model=NODE-MODEL PMM_AGENT_SETUP_NODE_MODEL Node model.

--paths- PMM_AGENT_PATHS_EXPORTERS_BASE Base path for exporters to use.

exporters_base=PATH

--paths- PMM_AGENT_PATHS_MONGODB_EXPORTER Path to mongodb_exporter .

mongodb_exporter=PATH

--paths- PMM_AGENT_PATHS_MYSQLD_EXPORTER Path to mysqld_exporter .

mysqld_exporter=PATH

306 of 359 Percona LLC, © 2021

 5.5.3 pmm-agent - PMM Client agent

Option Environment variable Description

--paths- PMM_AGENT_PATHS_NODE_EXPORTER Path to node_exporter .

node_exporter=PATH

--paths- PMM_AGENT_PATHS_POSTGRES_EXPORTER Path to postgres_exporter .

postgres_exporter=PATH

--paths- PMM_AGENT_PATHS_PROXYSQL_EXPORTER Path to proxysql_exporter .

proxysql_exporter=PATH

--paths-pt-summary=PATH PMM_AGENT_PATHS_PT_SUMMARY Path to pt-summary .

--paths-pt-mysql- PMM_AGENT_PATHS_PT_MYSQL_SUMMARY Path to pt-mysql-summary .

summary=PATH

--paths-pt-pg- PMM_AGENT_PATHS_PT_PG_SUMMARY Path to pt-pg-summary .

summary=PATH

--paths-tempdir=PATH PMM_AGENT_PATHS_TEMPDIR Temporary directory for exporters.

--ports-max=PORTS-MAX PMM_AGENT_PORTS_MAX Highest allowed port number for

listening sockets.

--ports-min=PORTS-MIN PMM_AGENT_PORTS_MIN Lowest allowed port number for

listening sockets.

--region=REGION PMM_AGENT_SETUP_REGION Node region.

--skip-registration PMM_AGENT_SETUP_SKIP_REGISTRATION Skip registration on PMM Server.

--trace PMM_AGENT_TRACE Enable trace output (implies --debug

).

-h , --help Show help (synonym for pmm-agent

help ).

--version Show application version, PMM

version, time-stamp, git commit hash
and branch.

LOGGING

By default, pmm-agent sends messages to stderr and to the system log ( syslogd or journald on Linux).

To get a separate log file, edit the pmm-agent start-up script.

systemd -based systems

• Script file: /usr/lib/systemd/system/pmm-agent.service

• Parameter: StandardError

• Default value: file:/var/log/pmm-agent.log

Example:

StandardError=file:/var/log/pmm-agent.log

307 of 359 Percona LLC, © 2021

 5.5.3 pmm-agent - PMM Client agent

initd -based systems

• Script file: /etc/init.d/pmm-agent

• Parameter: pmm_log

• Default value: /var/log/pmm-agent.log

Example:

pmm_log="/var/log/pmm-agent.log"

If you change the default log file name, reflect the change in the log rotation rules file /etc/logrotate.d/
pmm-agent-logrotate .

308 of 359 Percona LLC, © 2021

 5.6 API

5.6 API
PMM Server lets you visually interact with API resources representing all objects within PMM. You can
browse the API using the Swagger UI, accessible at the /swagger/ endpoint URL:

Clicking an object lets you examine objects and execute requests on them:

309 of 359 Percona LLC, © 2021

 5.6 API

The objects visible are nodes, services, and agents:

• A Node represents a bare metal server, a virtual machine, a Docker container, or a more specific type
such as an Amazon RDS Node. A node runs zero or more Services and Agents, and has zero or more
Agents providing insights for it.

• A Service represents something useful running on the Node: Amazon Aurora MySQL, MySQL,
MongoDB, etc. It runs on zero (Amazon Aurora Serverless), single (MySQL), or several (Percona XtraDB
Cluster) Nodes. It also has zero or more Agents providing insights for it.

• An Agent represents something that runs on the Node which is not useful in itself but instead provides
insights (metrics, query performance data, etc) about Nodes and/or Services. An agent always runs on
the single Node (except External Exporters), and provides insights for zero or more Services and
Nodes.

Nodes, Services, and Agents have Types which define specific their properties, and the specific logic they
implement.

Nodes and Services are external by nature – we do not manage them (create, destroy), but merely maintain
a list of them (add to inventory, remove from inventory) in pmm-managed . Most Agents, however, are started
and stopped by pmm-agent . The only exception is the External Exporter Type which is started externally.

310 of 359 Percona LLC, © 2021

 5.7 VictoriaMetrics

5.7 VictoriaMetrics
VictoriaMetrics is a third-party monitoring solution and time-series database that replaced Prometheus in
PMM 2.12.0.

5.7.1 Push/Pull modes

VictoriaMetrics allows metrics data to be ‘pushed’ to the server in addition to it being ‘pulled’ by the server.
When setting up services, you can decide which mode to use.

For PMM 2.12.0 the default mode is ‘pull’. Later releases will use the ‘push’ mode by default for newly-
added services.

The mode (push/pull) is controlled by the --metrics-mode flag for the pmm-admin config and
pmm-admin add commands.

If you need to change the metrics mode for an existing Service, you must remove it and re-add it with the
same name and the required flags. (There is currently no ability to “update” a service.)

5.7.2 Remapped targets for direct Prometheus paths

Direct Prometheus paths return structured information directly from Prometheus, bypassing the PMM
application.

They are accessed by requesting a URL of the form <PMM SERVER URL>/prometheus/<PATH> .

As a result of the move to VictoriaMetrics some direct Prometheus paths are no longer available.

Here are their equivalents.

• /prometheus/alerts → No change.

• /prometheus/config → No equivalent. However, some information is at /prometheus/targets .

• /prometheus/flags → The flag metrics at /prometheus/metrics .

• /prometheus/graph → /graph/explore (Grafana) or graph/d/prometheus-advanced/advanced-data-

exploration (PMM dashboard).

• /prometheus/rules → No change.

• /prometheus/service-discovery → No equivalent.

• /prometheus/status → Some information at /prometheus/metrics . High cardinality metrics information

at /prometheus/api/v1/status/tsdb .

• /prometheus/targets → /victoriametrics/targets .

5.7.3 Troubleshooting

To troubleshoot issues, see the VictoriaMetrics troubleshooting documentation.

You can also contact the VictoriaMetrics team via:

• Google Groups

• Slack

• Reddit

• Telegram

311 of 359 Percona LLC, © 2021

 5.8 Glossary

5.8 Glossary
5.8.1 Annotation

A way of showing a mark on dashboards signifying an important point in time.

5.8.2 Dimension

In the Query Analytics dashboard, to help focus on the possible source of performance issues, you can
group queries by dimension, one of: Query, Service Name, Database, Schema, User Name, Client Host

5.8.3 EBS

Amazon’s Elastic Block Store.

5.8.4 Fingerprint

A normalized statement digest—a query string with values removed that acts as a template or typical
example for a query.

5.8.5 IAM

Identity and Access Management (for Amazon AWS).

5.8.6 MM

Metrics Monitor.

5.8.7 NUMA

Non-Uniform Memory Access.

5.8.8 PEM

Privacy Enhanced Mail.

5.8.9 QPS

Queries Per Second. A measure of the rate of queries being monitored.

5.8.10 Query Analytics

Component of PMM Server that enables you to analyze MySQL query performance over periods of time.

5.8.11 STT

Security Threat Tool.

312 of 359 Percona LLC, © 2021

 5.8.12 Technical Preview

5.8.12 Technical Preview

Releases intended for public preview and feedback but with no support or service level agreement (SLA).
Should not be used on production or business-critical systems. May contain breaking changes to UI, API,
CLI. (Read more.)

5.8.13 VG

Volume Group.

313 of 359 Percona LLC, © 2021

 6. FAQ

6. FAQ

6.1 How can I contact the developers?

• Community forum.

• Discord channel.

• PMM project in JIRA.

6.2 What are the minimum system requirements for PMM?

See:

• PMM Server

• PMM Client

6.3 How can I upgrade from PMM version 1?

See Upgrade from PMM1.

6.4 How to control data retention for PMM?

See How to configure Data retention.

6.5 How often are NGINX logs in PMM Server rotated?

PMM Server runs logrotate on a daily basis to rotate NGINX logs and keeps up to ten of the most recent
log files.

6.6 What privileges are required to monitor a MySQL instance?

See Setting Up PMM Client.

6.7 Can I monitor multiple service instances?

You can add multiple instances of MySQL or some other service to be monitored from one PMM Client. In
this case, you must provide a unique port and IP address, or a socket for each instance, and specify a
unique name for each. (If a name is not provided, PMM uses the name of the PMM Client host.)

For example, to add complete MySQL monitoring for two local MySQL servers, the commands would be:

sudo pmm-admin add mysql --username root --password root instance-01 127.0.0.1:3001
sudo pmm-admin add mysql --username root --password root instance-02 127.0.0.1:3002

For more information, run:

pmm-admin add mysql --help

314 of 359 Percona LLC, © 2021

 6.8 Can I rename instances?

6.8 Can I rename instances?

You can remove any monitoring instance and then add it back with a different name.

When you remove a monitoring service, previously collected data remains available in Grafana. However,
the metrics are tied to the instance name. So if you add the same instance back with a different name, it
will be considered a new instance with a new set of metrics. So if you are re-adding an instance and want
to keep its previous data, add it with the same name.

6.9 Can I add an AWS RDS MySQL or Aurora MySQL instance from a non-default
AWS partition?
By default, the RDS discovery works with the default aws partition. But you can switch to special regions,
like the GovCloud one, with the alternative AWS partitions (e.g. aws-us-gov ) adding them to the Settings via
the PMM Server API.

To specify other than the default value, or to use several, use the JSON Array syntax: ["aws", "aws-cn"] .

6.10 How do I troubleshoot communication issues between PMM Client and PMM
Server?
See Troubleshoot PMM Server/PMM Client connection.

6.11 What resolution is used for metrics?

See Metrics resolution.

315 of 359 Percona LLC, © 2021

 6.12 How do I set up Alerting in PMM?

6.12 How do I set up Alerting in PMM?

When a monitored service metric reaches a defined threshold, PMM Server can trigger alerts for it either
using the Grafana Alerting feature or by using an external alert manager.

With these methods you must configure alerting rules that define conditions under which an alert should be
triggered, and the channel used to send the alert (e.g. email).

Alerting in Grafana allows attaching rules to your dashboard panels. Grafana Alerts are already integrated
into PMM Server and may be simpler to get set up.

Alertmanager allows the creation of more sophisticated alerting rules and can be easier to manage
installations with a large number of hosts. This additional flexibility comes at the expense of simplicity.

We only offer support for creating custom rules to our customers, so you should already have a working
Alertmanager instance prior to using this feature.

See also PMM Alerting with Grafana: Working with Templated Dashboards

6.13 How do I use a custom Prometheus configuration file inside PMM Server?
Normally, PMM Server fully manages the Prometheus configuration file.

However, some users may want to change the generated configuration to add additional scrape jobs,
configure remote storage, etc.

From version 2.4.0, when pmm-managed starts the Prometheus file generation process, it tries to load the /
srv/prometheus/prometheus.base.yml file first, to use it as a base for the prometheus.yml file.

The prometheus.yml file can be regenerated by restarting the PMM Server container, or by using the
SetSettings API call with an empty body.

See also

• API

• Percona blog: Extending PMM’s Prometheus Configuration

6.14 How to troubleshoot an Update?

See Troubleshoot update.

6.15 What are my login credentials when I try to connect to a Prometheus

Exporter?
PMM protects an exporter’s output from unauthorized access by adding an authorization layer. To access
an exporter you can use “ pmm ” as a user name and the Agent ID as a password. You can find the Agent ID
corresponding to a given exporter by running pmm-admin list .

6.16 How to provision PMM Server with non-default admin password?

Currently there is no API available to change the admin password. If you’re deploying through Docker you
can use the following code snippet to change the password after starting the Docker container:

316 of 359 Percona LLC, © 2021

 6.16 How to provision PMM Server with non-default admin password?

PMMPASSWORD="mypassword"
echo "Waiting for PMM to initialize to set password..."
until [ "`docker inspect -f {{.State.Health.Status}} pmm2-server`" = "healthy" ]; do sleep 1;
done
docker exec -t pmm2-server bash -c  "ln -s /srv/grafana /usr/share/grafana/data; grafana-cli --
homepath /usr/share/grafana admin reset-admin-password $PMMPASSWORD"

317 of 359 Percona LLC, © 2021

 7. Release Notes

7. Release Notes

7.1 Release Notes

• Percona Monitoring and Management 2.17.0

• Percona Monitoring and Management 2.16.0

• Percona Monitoring and Management 2.15.1

• Percona Monitoring and Management 2.15.0

• Percona Monitoring and Management 2.14.0

• Percona Monitoring and Management 2.13.0

• Percona Monitoring and Management 2.12.0

• Percona Monitoring and Management 2.11.1

• Percona Monitoring and Management 2.11.0

• Percona Monitoring and Management 2.10.1

• Percona Monitoring and Management 2.10.0

• Percona Monitoring and Management 2.9.1

• Percona Monitoring and Management 2.9.0

• Percona Monitoring and Management 2.8.0

• Percona Monitoring and Management 2.7.0

• Percona Monitoring and Management 2.6.1

• Percona Monitoring and Management 2.6.0

• Percona Monitoring and Management 2.5.0

• Percona Monitoring and Management 2.4.0

• Percona Monitoring and Management 2.3.0

• Percona Monitoring and Management 2.2.2

• Percona Monitoring and Management 2.2.1

• Percona Monitoring and Management 2.2.0

• Percona Monitoring and Management 2.1.0

• Percona Monitoring and Management 2.0.1

• Percona Monitoring and Management 2.0.0

318 of 359 Percona LLC, © 2021

 7.2 Percona Monitoring and Management 2.17.0

7.2 Percona Monitoring and Management 2.17.0

Date: May 11, 2021
Installation: Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance.

7.2.1 Release Highlights

• Custom certificates help define proper security levels for remotely monitored MySQL instances,
including Google Cloud SQL.

• Usability improvements to the External Monitoring UI. When filling parameters, you can enter the parts
of an endpoint (scheme, host, path) or let PMM automatically extract them from a URL.

• pg_stat_monitor 0.9.0 support. This change will give you compatibility with the latest version. Support
for new features will be in an upcoming release.

• Single-line install of PMM Server on supported Linux distributions (this feature is in Technical Preview).

• DBaaS Changes: (this feature is in Technical Preview)

• It is easier to experience DBaaS functionality; you can quickly turn it ON/OFF in Advanced settings
on the Settings page. (Read more)

• Database components management will enable PMM administrators to limit users in your
organization to specific (admin-approved) database versions in their DBaaS DB Clusters.

• For PXC clusters created using DBaaS, HAProxy will now be used by default. Please note: Monitoring
of the HAProxy in DBaaS will be enabled in an upcoming release.

• Changes to Sign in to Percona Platform. From this release, Registration of the Percona account will be
more secure and require additional confirmation.

7.2.2 New Features

• PMM-7863: DBaaS: Ability to specify in K8s configuration the version of HAProxy to be used for DB
creation

• PMM-7848, PMM-7847, PMM-7421: Add support for using SSL certificates between pmm-admin and
monitored MySQL databases

• PMM-7883: Single-line install of PMM Server on supported Linux distributions - [Technical Preview]

• PMM-7013, PMM-7819: DBaaS: Use HAProxy by default instead of ProxySQL for MySQL DB clusters

• PMM-7356, PMM-7581: DBaaS: Management of available versions of DB components

• PMM-7358, PMM-7576: DBaaS: Management of default versions of DB components

7.2.3 Improvements

• PMM-7572: Add TLS options to mysqld_exporter

• PMM-7783: Support of pg_stat_monitor 0.9.0

• PMM-7064: Integrated Alerting: Presenting severity of the Alert Rule using different colors

• PMM-7946: Better error message on PMM client if server doesn’t support HAProxy

• PMM-7932: Usability improvements on UI for adding External Services

• PMM-7641, PMM-7820: Add DBaaS to Technical Preview section and allow user to Enable/Disable via UI

319 of 359 Percona LLC, © 2021

 7.2.4 Bugs Fixed

• PMM-7966: Telemetry: Collect enabled/disabled status for Integrated Alerting and Security Threat Tool
features

7.2.4 Bugs Fixed

• PMM-7911: DBaaS: Invalid Number of Nodes results in an annoying error message pop-up

• PMM-7884: DBaaS: Fix DB Cluster tab loading

• PMM-7917: PostgreSQL exporter has high CPU usage during Restart

• PMM-8037: User can create a Percona Platform account without proper confirmation

• PMM-7702: DBaaS: Cannot edit already-created PSMDB clusters

• PMM-7991: MySQL Summary panel doesn’t exist on MySQL Summary dashboard

• PMM-7939: Inconsistent format of version reporting in pmm-admin

• PMM-7920: PostgreSQL Exporter has increased memory usage with pmm-client 2.15.1 & pmm-server
2.16.0

• PMM-7700: Integrated Alerting: Rule API crashing with more than two parameters or invalid values

• PMM-7616: Integrated Alerting: Incorrect title of the page in a browser

• PMM-7396: Integrated Alerting: Alerts tab error if user deletes Alert Rule which has Firing alerts

320 of 359 Percona LLC, © 2021

 7.3 Percona Monitoring and Management 2.16.0

7.3 Percona Monitoring and Management 2.16.0

Date: April 15, 2021
Installation: Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance.

7.3.1 Release Highlights

• Amazon RDS PostgreSQL monitoring

AWS monitoring in PMM now covers PostgreSQL RDS and PostgreSQL Aurora types. PMM will include
them in a Discovery UI where they can be added which will result in node related metrics as well as
PostgreSQL database performance metrics. Before this release, this was available only to MySQL-
related instances from Amazon RDS.

• Azure Discovery and Node metrics extraction

Technical Preview: PMM will have the same level of support for Microsoft Azure Database as a Service
(DBaaS) as we have for AWS’s DBaaS (RDS/Aurora on MySQL or PostgreSQL). You will be able to easily
discover and add Azure databases for monitoring by PMM complete with node-level monitoring. This
feature is available only if you explicitly activate it on the PMM Settings page. Deactivating it will not
remove added services from monitoring, but will just hide the ability to discover and add new Microsoft
Azure Services.

(This is a feature technical preview because we want to release it as soon as possible to get feedback
from users. We are expecting to do more work on this feature to make it more API and resource
efficient.)

• Security Threat Tool Scheduling - Manage execution and execution intervals

Security Threat Tool users are now able to control the Security Check execution time intervals for
groups of checks, move checks between groups, and disable individual checks if necessary.

• Support for pg_stat_monitor 0.8

Added compatibility with pg_stat_monitor plugin v 0.8.0. This is not exposing the new features for the
plugin in PMM yet, but ensures Query Analytics metrics are collected to the same degree it was with
version 0.6.0 of the plugin.

• Consistent support of Technical Preview Features

Reworked the PMM Settings page to make it clear what features are in Technical Preview vs General
Availability (GA) and to simplify activation/deactivation of technical preview features. We also provide a
better definition of what a Technical Preview is.

• Migration of Settings and other service pages in PMM from Grafana dashboards

The PMM Settings page and several others (including Add Instance and Inventory) are being converted
to Grafana pages and will no longer be presented as dashboards. Additionally, we’re moving the menu
to the sidebar navigation for consistency and more flexibility compared to the older menu structure.

• Integrated Alerting improvements

We released the next stage of improvements in Integrated Alerting functionality of PMM to simplify the
usage of the feature. Together with improvements, we continue fixing known bugs in this feature.

• [DBaaS] Resource planning and prediction (Resource calculator)

321 of 359 Percona LLC, © 2021

 7.3.2 New Features

Technical preview: While creating a DB cluster a user can see a prediction of the resources this
cluster will consume with all components as well as the current total and available resources in the
K8s. Users will be warned that if they attempt to create a DB cluster it may be unsuccessful because of
available resources in the K8s.

• [DBaaS] PSMDB 1.7.0 operator support

DBaaS in PMM will be using the recently-released Percona Kubernetes Operator for Percona Server for
MongoDB 1.7.0 to create MongoDB clusters.

7.3.2 New Features

• PMM-7313, PMM-7610: Ability to discover and monitor Amazon RDS PostgreSQL instances with
collecting PostgreSQL and RDS node metrics (Thanks to Daniel Guzman Burgos for reporting this
issue).

• PMM-7345: Expose metrics for all available databases on a PMM monitored PostgreSQL server.

• PMM-7344: Update postgres_exporter version from 0.4.6 to 0.8.0. (See the full list of improvements in
the changelog.)

• PMM-7767, PMM-7696: Implement feature flag to enable Microsoft Azure monitoring. Users can use the
UI or set an environment variable ( ENABLE_AZUREDISCOVER=1 ) during container creation.

• PMM-7684, PMM-7498: Ability to discover running and supported Microsoft Azure Databases instances
in a provided account.

• PMM-7678, PMM-7679, PMM-7676, PMM-7499, PMM-7691: Prepare, modify and use azure_exporter to
collect Node related metrics.

• PMM-7681: Use Microsoft Azure metrics on Node/OS-related dashboards to show the metrics on
panels.

• PMM-7339: Security Threat Tool: Ability to execute security checks individually and on-demand.

• PMM-7451, PMM-7337: Security Threat Tool: Ability to change intervals for security checks on the PMM
Settings page.

• PMM-7772, PMM-7338: Security Threat Tool: Ability to change default execution interval per check.

• PMM-7336: Security Threat Tool: Execute checks based on execution interval they belong to.

• PMM-7335: Security Threat Tool: Ship security check files with predefined execution interval.

• PMM-7748: Add an additional experimental menu for Dashboards on the left side panel.

• PMM-7688: Unify UX and layout of all PMM specific pages like Settings, Add Instance etc.

• PMM-7687: Modify links in menus to ensure both menus are working as expected after dashboard URL
change.

• PMM-7705: Simplify display of features in technical preview to easily identify them and their current
state.

• PMM-7522, PMM-7511: Integrated Alerting: Improve Notification Channels UX by Pagination for the
Notification list.

• PMM-7521, PMM-7510: Integrated Alerting: Improve Alert Rule Templates UX by Pagination on Rule
Templates list.

• PMM-7652, PMM-7674, PMM-7503, PMM-7486: DBaaS: While creating the DB cluster see all and
available resources in the K8s cluster, such as Disk, CPU & Memory.

• PMM-7508, PMM-7488: DBaaS: See predicted resource usage for selected DB Cluster configuration.

• PMM-7364: DBaaS: Show warning before starting creating the cluster if there are not enough resources
in the K8s cluster to create DB Cluster with requested configuration.

322 of 359 Percona LLC, © 2021

 7.3.3 Improvements

• PMM-7580, PMM-7359: DBaaS: Users can select the database version to use during DB Cluster
creation.

7.3.3 Improvements

• PMM-7506: Security Threat Tool: Reduce False Positives due to Roles automatically created in PXC with
no password but cannot be used to login.

• PMM-7569: Make PMM compatible with pg_stat_monitor 0.8 release.

• PMM-7571: Modified Percona Platform Account registration flow from PMM server UI.

• PMM-7513: Integrated Alerting: Ability to see default values and Threshold values during the Alert Rule
creation.

• PMM-7461: Integrated Alerting: Improve UX of tables presentation and loading on UI.

• PMM-7375: Integrated Alerting: Inform users about the template that they are editing and warn them
about the limitations.

• PMM-7260: Integrated Alerting: Make it clearer what rule is being edited.

7.3.4 Bugs Fixed

• PMM-7131, PMM-7555: QAN for PostgreSQL attempts to connect to a database with the same name as
the username. (Thanks to Daniel Guzman Burgos for reporting this issue)

• PMM-7481: Query Analytics is not showing “Query with Errors” in the Profile section.

• PMM-7464: NGINX misconfiguration leads to log storm in push mode

• PMM-7434: Integrated Alerting: Unknown parameters [threshold] error during Add/Update Alert Rule.

• PMM-7231: Integrated Alerting: Disabling channels does nothing.

• PMM-7379: Integrated Alerting: Can not edit Alert Rule Name through API.

• PMM-7232: Integrated Alerting: Disabling IA does not disable rules evaluation and notifications
sending.

• PMM-7119: Integrated Alerting: Update error notification for adding/update Alert rule Template – There
was inconsistent behavior if you tried to add a new Rule Template with an already-used name.

• PMM-7543: Integrated Alerting: selected section disappears from a breadcrumb after clicking the tab
for a second time.

• PMM-7766: DBaaS: PMM Upgrade breaks DBaaS get credentials method.

• PMM-7351: DBaaS: Safari does not accept float numbers as a custom option in the “Create Cluster”
dialogue.

• PMM-7701: DBaaS: PSMDB clusters stuck in initializing due to special characters in secrets.

323 of 359 Percona LLC, © 2021

 7.4 Percona Monitoring and Management 2.15.1

7.4 Percona Monitoring and Management 2.15.1

Date: March 18, 2021
Installation: Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance.

7.4.1 Release Highlights

This patch release fixes performance issues discovered in systems, together with other small fixes.

7.4.2 Bugs Fixed

• PMM-7635: Fix high CPU consumption by Grafana server after upgrade by docker replacement to
2.15.0 with large numbers of services in ‘push’ mode.

• PMM-7713: Fix high CPU and Memory consumption by Victoria Metrics after upgrade by docker
replacement to 2.15.0 with large numbers of services in ‘pull’ mode.

• PMM-7470: MongoDB exporter IndexStatsCollections is assigned values from wrong flag (intended
for 2.15.0, omitted due to missing merge cutoff) (Thanks to Tim for reporting this issue).

• PMM-1531: Metrics not being collected due to rename of MySQL 8 information schema tables.

324 of 359 Percona LLC, © 2021

 7.5 Percona Monitoring and Management 2.15.0

7.5 Percona Monitoring and Management 2.15.0

Date: March 01, 2021
Installation: Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance.

7.5.1 Release Highlights

• PMM 1 vs. 2 Parity

Disable collectors during adding node/service to monitoring

With this feature users can disable any collector used by PMM to get metrics. When metrics cannot be
collected or are no longer needed, disabling the collector(s) prevents PMM from flooding logs and saves
infrastructure resources.

Our vision for PMM collectors is to provide “stop from collecting” functionality to prevent possible harm
to the user environment. This “disable” feature is an initial step towards the ideal functionality. The full
and flexible management for “What metrics to collect and in what resolution” is slated for future
releases.

External services monitoring

Since PMM 1.4.0, users had the ability to monitor external services Percona didn’t currently support
(e.g., Redis). This blog article from 2018 nicely described external services monitoring at that time. (At
that time Percona was not natively supporting a PostgreSQL monitoring service and so this was listed
as an external service. Today, PostgreSQL is natively supported by PMM.)

Until now, PMM 2.x didn’t support external services monitoring. With this release, any non-natively
supported by PMM service will now become supported with external services monitoring. You can see
the list of possible exporters to be used in https://prometheus.io/docs/instrumenting/exporters/.
Natively-supported services will continue to deliver an expanded set of metrics and insights.

Provide summary information for systems (pt-*-summary actions)

With the addition of pt-*-summary in PMM 2, users can now view summary information about services
and nodes on PMM’s dashboard. This summary information is in the industry common format of
pt-*-summary tools output to simplify portability of this data. This format will also be preserved in the
snapshot of the dashboard shared with Percona Support to simplify investigations of issues.

Note: pt-*-summary includes formats for:

• pt-mysql-summary

• pt-mongodb-summary

• pt-pg-summary

• pt-summary

• HAProxy support by PMM

Users are able to add HAProxy Services for monitoring in PMM2. The support level of them in PMM will
be the same we have for ProxySQL, so they will be presented in Inventory and on Dashboard. This will
allow users who use HAProxy in their HA configuration to also have this component monitored by PMM.
In future releases PMM will start use HAProxy by default for the DBaaS feature and will also use this
functionality to monitor HAProxy.

• DBaaS Preview improvements (Technical Preview)

325 of 359 Percona LLC, © 2021

 7.5.2 New Features

From now you will be able to see the progress of internal steps the system makes when executing some
operations with DBaaS. The Progress Bar will not be time-related and will present only steps. The
Progress Bar component will also reflect the K8s/Operator-related errors to the user, so in the case of
errors, you will have the error text on the UI, and no need to use K8s tools to see the error. With the
same UI, you will be able to see the latest logs from K8s so they will have even more information about
why the error happened.

Known Limitations: The progress bar will not provide valuable information for the Delete operation (will
be in a later version when we’ll change the API with Operators Team), Operation of DB Cluster
Modification will have “strange” behavior and will start changes from non-zero values of steps. (This
will be modified after API changes.)

7.5.2 New Features

• PMM-4172, PMM-4306, PMM-5784, PMM-7177: Services and Nodes Summary presentation. Present
information about DB’s and Node status using pt-mysql-summary , pt-mongodb-summary , pt-pg-summary
outputs (in API and on Dashboards).

• PMM-7123: Ability to add External Services via the UI in PMM server.

• PMM-6711: Add external-group flag for pmm-admin inventory commands for simpler work with
External services.

• PMM-7405: Check connection response format when adding External Service to monitoring.

• PMM-6797: HAProxy monitoring: Ability to add HAProxy services with pmm-admin [inventory] add
[service] haproxy command.

• PMM-7487: HAProxy monitoring: Check connection to HAProxy services when adding them for
monitoring.

• PMM-7496: HAProxy monitoring: New HAProxy PXC dashboards.

• PMM-6943: HAProxy monitoring: Show HAProxy type services in PMM Inventory.

• PMM-6924: Integrated Alerting: Show ‘breadcrumbs’ navigation aid on non-dashboard pages as well
as Grafana dashboard pages.

• PMM-7294: Integrated Alerting: Pagination for viewing large numbers of Alert Rules.

• PMM-7417: Security Threat Tool: Show list of all available security checks.

• PMM-7418: Security Threat Tool: Ability to disable specific security checks.

• PMM-7419: DBaaS: Ability to see DB Cluster creation/modification logs.

• PMM-7266: DBaaS: Cluster creation progress bar – You can now see the progress of DBaaS DB cluster
creation. (The progress bar is based on the number of back-end technical steps, not the time required
to perform the tasks.)

7.5.3 Improvements

• PMM-4679: Docker: :latest tag for pmm-server and pmm-client images has been moved from v1
latest release to v2 latest release. Note: use of the latest tag is not recommended in production
environments, instead use :2 tag.

• PMM-7472: Remove Prometheus data source – If you were using custom dashboards with a specified
data source (not using empty to use default one) you may need to edit your dashboards to use the
proper data source. PMM is no longer using Prometheus but uses compatible storage for metrics from
VictoriaMetrics. We renamed the data source to be more technology-agnostic.

• PMM-6695: Software update: Grafana 7.1.3 to 7.3.7 (See What’s new in Grafana 7.2 and What’s new
in Grafana 7.3.)

326 of 359 Percona LLC, © 2021

 7.5.4 Bugs Fixed

• PMM-7471: Software update: VictoriaMetrics 1.52.0 to 1.53.1 (See VictoriaMetrics 1.53.0 and
VictoriaMetrics 1.53.1.)

• PMM-6693: API keys usage – PMM users can now use API keys (generated in Grafana UI) for
interaction with PMM server instead of username/password pairs. The API key should have the same
level of access (Admin or Viewer) as is required for username/password pairs.

• PMM-7240: DBaaS: Change from Dashboard to Grafana Page – We changed the DBaaS page from a
Grafana Dashboard to a Grafana Page to be better aligned with the DBaaS enable/disable status and
avoid confusion when DBaaS is disabled.

• PMM-7328: Security Threat Tool: Download and run checks when activated, immediately, repeating
every 24 hours thereafter (Previously, downloading and running new checks happened every 24 hours
but the cycle didn’t begin when STT was activated.)

• PMM-7329: Security Threat Tool: Hide check results tab if STT is disabled.

• PMM-7331: Security Threat Tool: Failed checks have ‘Read more’ links with helpful content.

• PMM-7422: Security Threat Tool: View all active and silenced alerts.

• PMM-7257,PMM-7433: Integrated Alerting: Easier-to-read rule details in Alert Rules list (API and UI
presentation).

• PMM-7259: Integrated Alerting: Better UI error reporting for disabled Integrated Alerting. (Hint to users
how to enable it.)

• PMM-5533: Better indentation of columns in pmm-admin list output.

• PMM-5888: Improve pmm-admin --help descriptions for external services.

7.5.4 Bugs Fixed

• PMM-5837: pmm-agent reports “Malformed DSN” error when adding PostgreSQL instance with a PMM
user password containing = (equals sign) (Thanks to Alexandre Barth for reporting this issue).

• PMM-5969: Removing Services or Nodes with pmm-admin ... --force mode does not stop running
agents, VictoriaMetrics continues collecting data from exporters.

• PMM-6685: In low screen resolutions Services submenu wraps, becomes obscured, and can’t be
accessed.

• PMM-6681: Not all PMM admin users can download diagnostic logs, only those with Grafana admin
rights.

• PMM-7227: Table stats metrics not being collected in instances with millions of tables.

• PMM-7426: vmagent continually restarts, blocking comms between pmm-agent & pmm-managed – Users
running multiple services on the same PMM agent in ‘push’ mode could face this issue when restarting
the agent after bulk-adding services.

• PMM-6636: Dashboards: MySQL Replication Summary: ‘Binlog Size’, ‘Binlog Data Written Hourly’,
‘Node’ not being charted when the instance is RDS.

• PMM-7325: Dashboards: MySQL User Details: user labels unreadable with high number (>20) of users
(Thanks to Andrei Fedorov for reporting this issue).

• PMM-7416: Dashboards: PostgreSQL Instance Summary: Some panels (e.g. Tuple) not using selected
database.

• PMM-7235: Integrated Alerting: Filtered out alerts are shown in the UI as firing.

• PMM-7324: Integrated Alerting: Add Pager Duty Notification Channel: after user pastes copied key Add
button is not enabled.

• PMM-7346: Integrated Alerting: It is possible to create Alert Rule with negative duration time.

• PMM-7366: Integrated Alerting: Entities (e.g. templates, channels, rules) are in inconsistent states.

327 of 359 Percona LLC, © 2021

 7.5.4 Bugs Fixed

• PMM-7467: Integrated Alerting: < (less-than symbol) wrongly interpreted by Alert templates (as
&lt; ).

• PMM-7591: Integrated Alerting: User can not receive notifications on email after password update.

• PMM-7343: Security Threat Tool: Check results show previously failed checks after STT re-enabled.

• PMM-7250: DBaaS: Confusing error “Cannot get PSMDB/PXC cluster” appears after removing DB
cluster.

• PMM-7193: DBaaS: Number of Nodes can be set as float.

• PMM-7349: DBaaS: Host and Password occasionally disappearing from Connection column.

328 of 359 Percona LLC, © 2021

 7.6 Percona Monitoring and Management 2.14.0

7.6 Percona Monitoring and Management 2.14.0

Date: January 28, 2021
Installation: Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance.

7.6.1 Release Highlights

• Switch to push metrics by default

In PMM 2.12.0, Percona replaced its metrics collection engine (formerly Prometheus) with
VictoriaMetrics. Historically, PMM used a pull method with Prometheus while VictoriaMetrics can operate
in either a pull or push method. When PMM 2.12.0 was released, Percona kept the default method as
pull. Now with PMM 2.14.0, Percona is shifting the default to push for all newly-added instances. This
blog post describes the two methods and why push benefits users. Also, here is a post by Peter Zaitzev
of FAQs relating to the move to VictoriaMetrics and the push model. Documentation on the push
method is here.

Note: Installing the 2.14.0 or newer PMM server will change the default behavior on 2.12.0 and 2.13.0
clients from “pull” method to “push” for any newly added services. Existing services will remain in
whatever mode they were prior to upgrade.

• DBaaS Preview phase 1.0 (Technical Preview)

In 2.13.0 we introduced Percona’s Database as a Service (DBaaS) which enables non-DBAs (software
architects, developers, site reliability engineers, etc.) to perform typical DBA tasks to manage an
organization’s database environment via user interfaces and automation orchestration. This release
contains several enhancements and fixes, many directly from user feedback.

Note: This capability is feature-flagged and turned off by default. Users require a variable to be passed
to PMM to expose this functionality.

• External services presentation on node summary dashboard

Improvements to the user experience for adding and viewing external services (any data that can be
monitored by a Prometheus exporter such as: non-Percona supported databases like Redis,
ElasticSearch, Cassandra, etc. or an organization’s external application) on the Node Summary
dashboard of PMM.

7.6.2 New Features

• PMM-5765: Ability to monitor External Services for situations where PMM Client can’t be installed – Uses
a new command pmm-admin add external-serverless . (See pmm-admin.) (This is a Technical
Preview feature)

• PMM-7015: DBaaS Preview: Create DB cluster with randomly-generated password

• PMM-7007: Integrated Alerting: Ability to copy (duplicate) alert rules

• PMM-7006: Integrated Alerting: Ability to delete alert rules

• PMM-6941: Integrated Alerting: Ability to delete alert rule templates

7.6.3 Improvements

• PMM-6985: DBaaS: Ability to force unregister Kubernetes cluster

329 of 359 Percona LLC, © 2021

 7.6.4 Bugs Fixed

• PMM-7145: ‘Push’ metrics mode is default when adding services and nodes (All agents collecting data
from Services and Nodes will now use PUSH model if not specified explicitly. You will still be able to use
--metrics-mode flag to use Pull metrics if needed. All previously set up agents will keep their existing
mode. To change these you need to remove and re-add them.)

• PMM-7282: Integrated Alerting: Ability to create rule without channels and filters

• PMM-7226: Integrated Alerting: Validate parameters during rule creation/update

• PMM-7082: Integrated Alerting: Severity levels are color-coded

• PMM-7065: Integrated Alerting: Show rule details for items in Alert Rules list

• PMM-7048: DBaaS: Simplify Cluster creation by moving Create Cluster button to earlier steps

• PMM-6993: Protect against possible problems with EXPLAIN of stored functions in MySQL – We are
fixing possible problems caused by an attempt to analyze queries covered in https://bugs.mysql.com/
bug.php?id=67632.

7.6.4 Bugs Fixed

• PMM-7312: Error when accessing Metrics data on Dashboards for large installations

• PMM-7310: VictoriaMetrics consuming 100’s Gb’s of disk in /tmp/searchResults in PMM 2.13.0

• PMM-5137: Swagger page redirect isn’t working

• PMM-7144: DBaaS: Creating DB cluster with same name (Thanks to Beata Handzelova for reporting
this issue)

• PMM-7323: DBaaS: ‘Remove DB Cluster from Kubernetes Cluster’ removes wrong one

• PMM-7251: Integrated Alerting: Error Rule with ID "mysql_version" not found if both Security Threat
Tool and Integrated Alerting enabled

• PMM-7247: DBaaS: Disk size is always 0 for Percona XtraDB cluster

• PMM-7178: pg_stat_monitor integration is broken with version 0.6.0 of the plugin

• PMM-7169: Old data (from Prometheus) not deleted when Retention period expires

• PMM-7105: Query Analytics: no ‘Example’ or ‘Explain’ data for MariaDB

• PMM-7239: Integrated Alerting: Validate Slack channel names in Notification Channels

• PMM-7213: MySQL InnoDB Details dashboard: remove color-coding on ‘Data Buffer Pool Fit’ element

• PMM-7167: Some panels not visible when using long time intervals (e.g. 30 days)

• PMM-7133: Incorrect descriptions for data links in dashboards

• PMM-7103: VictoriaMetrics build logs not deleted from PMM Server Docker image

• PMM-6904: pmm-admin annotate command crashes for non-generic node types

• PMM-6902: No query Examples on PostgreSQL 12 with pg_stat_monitor

• PMM-6838: ProxySQL Instance Summary dashboard: Incorrect “Hostgroup Size” formula

• PMM-6490: rds_exporter crashes when more than 100 AWS RDS instances added (Thanks to https://
github.com/vlinevych for fixing this)

• PMM-6096: pmm-agent connection checker does not check authentication for MongoDB

• PMM-7303: Disk Details, Nodes Compare dashboards: ‘Disk Utilization’ description is confusing

330 of 359 Percona LLC, © 2021

 7.7 Percona Monitoring and Management 2.13.0

7.7 Percona Monitoring and Management 2.13.0

Date: December 29, 2020
Installation: Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance.

7.7.1 Release Highlights

1. Ability to monitor SSL-enabled MongoDB

Allows PMM administrators to set up configured SSL certificate “keys” to authenticate the connection to
PMM, specifically for setting up MongoDB. This is a critical security requirement especially in large
enterprise infrastructure environments.

2. Technical Previews

Note: We do not recommend the use of technical preview features in enterprise or production
environments until the functionality is released as general availability (GA). While in Technical Preview
status, these features are not supported by Percona Support SLA, except by Product/Engineering on a
best-efforts basis.

a. Integrated Alerting MVP

A new feature in PMM to set up parameters and revive alerts about the Services and Nodes monitored
by PMM.

Read more on our blog and in our documentation.

b. Node Summary/Nodes Overview dashboards: Show External services on dashboards

Improves the user experience for adding and viewing external services on the Node Summary
dashboard of PMM. External services means any data that can be monitored by a Prometheus
exporter, for example, non-Percona supported databases like Redis, ElasticSearch, Cassandra, etc. or
an organization’s external application.

c. DBaaS Preview phase 1.0

We are also releasing the first preview of DBaaS functionality; when combined with a compatible
Kubernetes environment and Percona Operators, you can create Percona XtraDB or MongoDB clusters
with just a few clicks. (Read more about configuration and usage.)

7.7.2 Improvements

• PMM-5364: Ability to monitor SSL-enabled MongoDB by passing certificate parameters in pmm-admin

add command (Thanks to Hubertus Krogmann for reporting this issue)

• PMM-7086: Re-mapped /prometheus/<end-point> to /victoriametrics/<end-point> but created

aliases for users that still rely on the /prometheus/<end-point> in bookmarks and scripts (Thanks to
Daniel Guzman Burgos for reporting this issue)

• PMM-6713: Node Summary/Nodes Overview dashboards: External exporters can now be added to
dashboard and shown as part of grouping of a broader service

• PMM-7173: VictoriaMetrics updated to 1.50.2: Includes HTML pages vs JSON output and new functions
available for alerting rules (see all tags)

7.7.3 Bugs Fixed

• PMM-7054: ProxySQL Instance Summary dashboard: no Node Metrics

331 of 359 Percona LLC, © 2021

 7.7.3 Bugs Fixed

• PMM-7092: PMM Server Docker update from 2.11.1 to 2.12.0 leaves container in unhealthy state
(Thanks to Hubertus Krogmann for reporting this issue)

• PMM-7208: Confusing “Access denied” message for ‘Viewer’ users on many dashboards

• PMM-6987: No IP address shown in log file of OVF appliance running in headless mode

• PMM-7146: MongoDB Instance Summary dashboard: ReplSet element showing metric name instead
of replication set

• PMM-6992: Administrators can’t see user’s actual IP address in Grafana profile-Preferences-Sessions

• PMM-6865: Rendered dashboard images partly obscured by error message

332 of 359 Percona LLC, © 2021

 7.8 Percona Monitoring and Management 2.12.0

7.8 Percona Monitoring and Management 2.12.0

Date: December 1, 2020
Installation: Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance.

7.8.1 Release Highlights

• VictoriaMetrics replaces Prometheus and is now the default data source. VictoriaMetrics supports both
PUSH (client to server) and PULL metrics collection modes. (Read more.)

• PMM Client can be run as a Docker image.

• The ‘Add Instance’ page and forms have been redesigned and look much better.

7.8.2 New Features

• PMM-5799: PMM Client now available as docker image in addition to RPM, DEB and .tgz

• PMM-6968: Integrated Alerting: Basic notification channels actions API Create, Read, Update, Delete

• PMM-6842: VictoriaMetrics: Grafana dashboards to monitor VictoriaMetricsDB as replacement for

dashboards that used to monitor Prometheus DB

• PMM-6395: Replace Prometheus with VictoriaMetrics in PMM for better performance and additional
functionality

7.8.3 Improvements

• PMM-6744: Prevent timeout of low resolution metrics in MySQL instances with many tables (~1000’s)

• PMM-6504: MySQL Replication Summary: MySQL Replication Delay graph not factoring in value of
intentionally set SQL_Delay thus inflating time displayed

• PMM-6820: pmm-admin status --wait option added to allow for configurable delay in checking status
of pmm-agent

• PMM-6710: pmm-admin : Allow user-specified custom ‘group’ name when adding external services

• PMM-6825: Allow user to specify ‘listen address’ to pmm-agent otherwise default to 127.0.0.1

• PMM-6793: Improve user experience of ‘add remote instance’ workflow

• PMM-6759: Enable Kubernetes startup probes to get status of pmm-agent using ‘GET HTTP’ verb

• PMM-6736: MongoDB Instance Summary dashboard: Ensure colors for ReplSet status matches those
in MongoDB ReplSet Summary dashboard for better consistency

• PMM-6730: Node Overview/Summary Cleanup: Remove duplicate service type ‘DB Service Connections’

• PMM-6542: PMM Add Instance: Redesign page for more intuitive experience when adding various
instance types to monitoring

• PMM-6518: Update default data source name from ‘Prometheus’ to ‘Metrics’ to ensure graphs are
populated correctly after upgrade to VictoriaMetrics

• PMM-6428: Query Analytics dashboard - Ensure user-selected filter selections are always visible even if
they don’t appear in top 5 results

333 of 359 Percona LLC, © 2021

 7.8.4 Bugs Fixed

• PMM-5020: PMM Add Remote Instance: User can specify ‘Table Statistics Limit’ for MySQL and AWS
RDS MySQL to disable table stat metrics which can have an adverse impact on performance with too
many tables

7.8.4 Bugs Fixed

• PMM-6811: MongoDB Cluster Summary: when secondary optime is newer than primary optime, lag
incorrectly shows 136 years

• PMM-6650: Custom queries for MySQL 8 fail on 5.x (on update to pmm-agent 2.10) (Thanks to user
debug for reporting this issue)

• PMM-6751: PXC/Galera dashboards: Empty service name with MySQL version < 5.6.40

• PMM-5823: PMM Server: Timeout when simultaneously generating and accessing logs via download or
API

• PMM-4547: MongoDB dashboard replication lag count incorrect (Thanks to user vvol for reporting this
issue)

• PMM-7057: MySQL Instances Overview: Many monitored instances (~250+) gives ‘too long query’ error

• PMM-6883: Query Analytics: ‘Reset All’ and ‘Show Selected’ filters behaving incorrectly

• PMM-6686: Query Analytics: Filters panel blank on Microsoft Edge 44.18362.449.0

• PMM-6007: PMM Server virtual appliance’s IP address not shown in OVF console

• PMM-6754: Query Analytics: Bad alignment of percentage values in Filters panel

• PMM-6752: Query Analytics: Time interval not preserved when using filter panel dashboard shortcuts

• PMM-6664: Query Analytics: No horizontal scroll bar on Explain tab

• PMM-6632: Node Summary - Virtual Memory Utilization chart: incorrect formulas

• PMM-6537: MySQL InnoDB Details - Logging - Group Commit Batch Size: giving incorrect description

• PMM-6055: PMM Inventory - Services: ‘Service Type’ column empty when it should be ‘External’ for
external services

7.8.5 Known Issues

• PMM-7092: Update docker pmm-server 2.11.1 to 2.12.0 results in an unhealthy container.

Workaround: A folder is not created on container upgrade and will need to be created manually for one of
the components. Before starting the new pmm-server 2.12.0, execute:  

docker exec -ti pmm-server mkdir -p /srv/victoriametrics/data

docker exec -ti pmm-server chown -R pmm:pmm /srv/victoriametrics/
docker restart pmm-server

334 of 359 Percona LLC, © 2021

 7.9 Percona Monitoring and Management 2.11.1

7.9 Percona Monitoring and Management 2.11.1

Date: October 19, 2020
Installation: Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance.

7.9.1 Bugs Fixed

• PMM-6782: High CPU usage after update to 2.11.0

335 of 359 Percona LLC, © 2021

 7.10 Percona Monitoring and Management 2.11.0

7.10 Percona Monitoring and Management 2.11.0

Date: October 14, 2020
Installation: Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance.

7.10.1 New Features

• PMM-6567: Technical preview of new PostgreSQL extension pg_stat_monitor

• PMM-6515: Link added directly to Node/Service page from Query Analytics filters, opens in new window

7.10.2 Improvements

• PMM-6727: Grafana plugin updates: grafana-polystat-panel=1.2.2 , grafana-piechart-panel=1.6.1

• PMM-6625: Default sort to “Average - descending” on all dashboards

• PMM-6609: MySQL Instances Compare & Summary dashboards: Changed metric in ‘MySQL Internal
Memory Overview’

• PMM-6598: Dashboard image sharing (Share Panel): Improved wording with link to configuration
instructions

• PMM-6557: Update Prometheus to 2.21.0

• PMM-6554: MySQL InnoDB Details dashboard: Add “sync flushing” to “InnoDB Flushing by Type”

7.10.3 Bugs Fixed

• PMM-4547: MongoDB dashboard replication lag count incorrect (Thanks to user vvol for reporting this
issue)

• PMM-6639: Integrated update does not detect all container types

• PMM-6765: Tables information tab reports ‘table not found’ with new PostgreSQL extension
pg_stat_monitor

• PMM-6764: Query Analytics: cannot filter items that are hidden - must use “Show all”

• PMM-6742: Upgrade via PMM UI stalls (on yum update pmm-update )

• PMM-6689: No PostgreSQL queries or metrics in Query Analytics with PostgreSQL 13

( postgresql_pgstatements_agent in Waiting status)

• PMM-6738: PostgreSQL examples shown despite --disable-queryexamples option

• PMM-6535: Unable to open ‘Explore’ in new window from Grafana menu

• PMM-6532: Click-through URLs lose time ranges when redirecting to other dashboards

• PMM-6531: Counter-intuitive coloring of element “Update Stats when Metadata Queried”

• PMM-6645: Clean up unnecessary errors in logs ( vertamedia-clickhouse-datasource plugin)

• PMM-6547: Hexagonal graph tooltip text overflows bounding box

336 of 359 Percona LLC, © 2021

 7.11 Percona Monitoring and Management 2.10.1

7.11 Percona Monitoring and Management 2.10.1

Date: September 22, 2020
Installation: Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance.

7.11.1 Bugs Fixed

• PMM-6643: New MongoDB exporter has higher CPU usage compared with old

337 of 359 Percona LLC, © 2021

 7.12 Percona Monitoring and Management 2.10.0

7.12 Percona Monitoring and Management 2.10.0

Date: September 15, 2020
Installation: Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance.

7.12.1 New Features

• PMM-2045: New dashboard: MySQL Group Replication Summary

• PMM-5738: Enhanced exporter: replaced original mongodb-exporter with a completely rewritten one
with improved functionality

• PMM-5126: Query Analytics Dashboard: Search by query substring or dimension (Thanks to user
debug for reporting this issue)

• PMM-6360: Grafana Upgrade to 7.1.3

• PMM-6355: Upgrade Prometheus to 2.19.3

• PMM-6597: Documentation: Updated Image rendering instructions for PMM

• PMM-6568: Reusable user interface component: Pop-up dialog. Allows for more consistent interfaces
across PMM

• PMM-6375, PMM-6373, PMM-6372: Sign in, Sign up and Sign out UI for Percona Account inside PMM
Server

• PMM-6328: Query Analytics Dashboard: Mouse-over crosshair shows value on sparklines

• PMM-3831: Node Summary Dashboard: Add pt-summary output to dashboard to provide details on
system status and configuration

7.12.2 Improvements

• PMM-6647: MongoDB dashboards: RocksDB Details removed, MMAPv1 & Cluster Summary changed

• PMM-6536: Query Analytics Dashboard: Improved filter/time search message when no results

• PMM-6467: PMM Settings: User-friendly error message

• PMM-5947: Bind services to internal address for containers

7.12.3 Bugs Fixed

• PMM-6336: Suppress sensitive data: honor pmm-admin flag --disable-queryexamples when used in
conjunction with --query-source=perfschema

• PMM-6244: MySQL InnoDB Details Dashboard: Inverted color scheme on “BP Write Buffering” panel

• PMM-6294: Query Analytics Dashboard doesn’t resize well for some screen resolutions (Thanks to user
debug for reporting this issue)

• PMM-5701: Home Dashboard: Incorrect metric for DB uptime (Thanks to user hubi_oediv for
reporting this issue)

• PMM-6427: Query Analytics dashboard: Examples broken when switching from MongoDB to MySQL
query

• PMM-5684: Use actual data from INFORMATION_SCHEMA vs relying on cached data (which can be 24 hrs
old by default)

338 of 359 Percona LLC, © 2021

 7.12.4 Known Issues

• PMM-6500: PMM Database Checks: Unwanted high-contrast styling

• PMM-6440: MongoDB ReplSet Summary Dashboard: Primary shows more lag than replicas

• PMM-6436: Query Analytics Dashboard: Styles updated to conform with upgrade to Grafana 7.x

• PMM-6415: Node Summary Dashboard: Redirection to database’s Instance Summary dashboard omits
Service Name

• PMM-6324: Query Analytics Dashboard: Showing stale data while fetching updated data for query
details section

• PMM-6316: Query Analytics Dashboard: Inconsistent scrollbar styles

• PMM-6276: PMM Inventory: Long lists unclear; poor contrast & column headings scroll out of view

• PMM-6529: Query Analytics filter input margin disappears after scrolling

7.12.4 Known Issues

• PMM-6643: High CPU usage for new MongoDB exporter (fixed in Percona Monitoring and Management
2.10.1)

339 of 359 Percona LLC, © 2021

 7.13 Percona Monitoring and Management 2.9.1

7.13 Percona Monitoring and Management 2.9.1

Date: August 4, 2020
Installation: Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance.

7.13.1 Improvements

• PMM-6230: Custom dashboards set as Home remain so after update

• PMM-6300: Query Analytics Dashboard: Column sorting arrows made easier to use (Thanks to user
debug for reporting this issue)

• PMM-6208: Security Threat Tool: Temporarily silence viewed but un-actioned alerts

• PMM-6315: Query Analytics Dashboard: Improved metrics names and descriptions

• PMM-6274: MySQL User Details Dashboard: View selected user’s queries in Query Analytics Dashboard

• PMM-6266: Query Analytics Dashboard: Pagination device menu lists 25, 50 or 100 items per page

• PMM-6262: PostgreSQL Instance Summary Dashboard: Descriptions for all ‘Temp Files’ views

• PMM-6253: Query Analytics Dashboard: Improved SQL formatting in Examples panel

• PMM-6211: Query Analytics Dashboard: Loading activity spinner added to Example, Explain and Tables
tabs

• PMM-6162: Consistent sort order in dashboard drop-down filter lists

• PMM-5132: Better message when filter search returns nothing

7.13.2 Bugs Fixed

• PMM-5783: Bulk failure of SHOW ALL SLAVES STATUS scraping on PS/MySQL distributions triggers
errors

• PMM-6294: Query Analytics Dashboard doesn’t resize well for some screen resolutions (Thanks to user
debug for reporting this issue)

• PMM-6420: Wrong version in successful update pop-up window

• PMM-6319: Query Analytics Dashboard: Query scrolls out of view when selected

• PMM-6302: Query Analytics Dashboard: Unnecessary EXPLAIN requests

• PMM-6256: Query Analytics Dashboard: InvalidNamespace EXPLAIN error with some MongoDB queries

• PMM-6329: Query Analytics Dashboard: Unclear origin of sparkline tool-tip on mouse-over

• PMM-6259: Query Analytics Dashboard: Slow appearance of query time distribution graph for some
queries

• PMM-6189: Disk Details Dashboard: Disk IO Size chart larger by factor of 512

• PMM-6269: Query Analytics Dashboard: Metrics drop-down list obscured when opened

• PMM-6247: Query Analytics Dashboard: Overview table not resizing on window size change

• PMM-6227: Home Dashboard redirection to Node Summary Dashboard not working

340 of 359 Percona LLC, © 2021

 7.14 Percona Monitoring and Management 2.9.0

7.14 Percona Monitoring and Management 2.9.0

Date: July 14, 2020
Installation: Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance.

7.14.1 Highlights

This release brings a major rework of the Query Analytics (QAN) component, completing the migration from
Angular to React, and adding new UI functionality and features.

For details, see:

• PMM-5125: Implement new version of QAN

• PMM-5516: QAN migration to React and new UI implementation

You can read more in the accompanying blog post (here).

7.14.2 New Features

• PMM-6124: New dashboards: MongoDB Replica Set Summary and MongoDB Cluster Summary

• PMM-1027: New dashboard: MySQL User Details ( INFORMATION_SCHEMA.CLIENT_STATISTICS )

• PMM-5604: User interface for MongoDB EXPLAIN

• PMM-5563: Per-Service and per-Node Annotations (This completes the work on improvements to the
Annotation functionality.)

7.14.3 Improvements

• PMM-6114: Sort Agents, Nodes, and Services alphabetically by name in Inventory page (Thanks to user
debug for reporting this issue)

• PMM-6147: Update Grafana plugins to latest versions

7.14.4 Bugs Fixed

• PMM-5800: QAN explain and tables tabs not working after removing MySQL metrics agent

• PMM-5812: Prometheus relabeling broken ( relabel_configs un-marshal errors) (Thanks to user

b4bufr1k for reporting this issue)

• PMM-6184: MongoDB Instances Compare dashboard shows MySQL metric

• PMM-5941: Stacked Incoming/Outgoing Network Traffic graphs in MySQL Instances Overview

dashboard prevents comparison

• PMM-6194: Missing UID for Advanced Data Exploration dashboard

• PMM-6191: Incorrect computation for Prometheus Process CPU Usage panel values in Prometheus
dashboard

• PMM-6175: Node Overview dashboard shows unit for unit-less value ‘Top I/O Load’

341 of 359 Percona LLC, © 2021

 7.15 Percona Monitoring and Management 2.8.0

7.15 Percona Monitoring and Management 2.8.0

Date: June 25, 2020
Installation: Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance.

7.15.1 Improvements

• PMM-544: Agents, Services and Nodes can now be removed via the ‘PMM Inventory’ page

• PMM-5706: User-installed Grafana plugins unaffected by PMM upgrade

7.15.2 Bugs Fixed

• PMM-6153: PMM 2.7.0 inoperable when no Internet connectivity

• PMM-5365: Client fails to send non-UTF-8 query analytics content to server (Thanks to user romulus
for reporting this issue)

• PMM-5920: Incorrect metric used in formula for “Top Users by Rows Fetched/Read” graph

• PMM-6084: Annotations not showing consistently on dashboards

• PMM-6011: No data in MongoDB Cluster summary, RocksDB & MMAPv1 details

• PMM-5987: Incorrect total value for virtual memory utilization

342 of 359 Percona LLC, © 2021

 7.16 Percona Monitoring and Management 2.7.0

7.16 Percona Monitoring and Management 2.7.0

Date: June 9, 2020
Installation: Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance.

In this release, we have updated Grafana to version 6.7.4 to fix CVE-2020-13379. We recommend updating
to the latest version of PMM as soon as possible.

7.16.1 New Features

• PMM-5257, PMM-5256, & PMM-5243: pmm-admin socket option ( –-socket ) to specify UNIX socket path
for connecting to MongoDB, PostgreSQL, and ProxySQL instances

7.16.2 Improvements

• PMM-2244: pmm-admin status command output shows both pmm-admin and pmm-agent versions

• PMM-5968: Disallow PMM Server node or agent removal via API

• PMM-5946: MySQL Table Details dashboard filter on Service Name prevents display of services without
data

• PMM-5926: Expose PMM agent version in pmm-admin status command

• PMM-5891: PMM Home page now includes News panel

• PMM-5906: Independent update of PMM components deactivated

7.16.3 Bugs Fixed

• PMM-6004: MySQL exporter reporting wrong values for cluster status ( wsrep_cluster_status )

• PMM-4547: MongoDB dashboard replication lag count incorrect

• PMM-5524: Prometheus alerting rule changes needs docker restart to activate

• PMM-5949: Unwanted filters applied when moving from QAN to Add Instance page

• PMM-5870: MySQL Table Details dashboard not showing separate service names for tables

• PMM-5839: PostgreSQL metrics disparity between query time and block read/write time

• PMM-5348: Inventory page has inaccessible tabs that need reload to access

• PMM-5348: Incorrect access control vulnerability fix (CVE-2020-13379) by upgrading Grafana to 6.7.4

343 of 359 Percona LLC, © 2021

 7.17 Percona Monitoring and Management 2.6.1

7.17 Percona Monitoring and Management 2.6.1

Date: May 18, 2020
Installation: Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance.

7.17.1 Improvements

• PMM-5936: Improved Summary dashboard for Security Threat Tool ‘Failed Checks’

• PMM-5937: Improved Details dashboard for Security Threat Tool ‘Failed Database Checks’

7.17.2 Bugs Fixed

• PMM-5924: Alertmanager not running after PMM Server upgrade via Docker

• PMM-5915: supervisord not restarting after restart of PMM Server virtual appliances (OVF/AMI)

• PMM-5945: ‘Updates’ dashboard not showing available updates

• PMM-5870: MySQL Table Details dashboard not showing separate service names for tables

344 of 359 Percona LLC, © 2021

 7.18 Percona Monitoring and Management 2.6.0

7.18 Percona Monitoring and Management 2.6.0

Date: May 11, 2020
Installation: Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance.

7.18.1 New Features

• PMM-5728: Technical preview of External Services monitoring feature. A new command provides
integration with hundreds of third-party systems (https://prometheus.io/docs/instrumenting/
exporters/) via the Prometheus protocol so that you can monitor external services on a node where
PMM agent is installed.

• PMM-5822: PMM now includes a Security Threat Tool to help users avoid the most common database
security issues. Read more here.

• PMM-5559: Global annotations can now be set with the pmm-admin annotate command.

• PMM-4931: PMM now checks Docker environment variables and warns about invalid ones.

7.18.2 Improvements

• PMM-1962: The PMM Server API (via /v1/readyz ) now also returns Grafana status information in
addition to that for Prometheus.

• PMM-5854: The Service Details dashboards were cleaned up and some unused selectors were removed.

• PMM-5775: It is now clearer which nodes are Primary and which are Secondary on MongoDB Instance
dashboards.

• PMM-5549: PMM’s Grafana component is now the latest, 6.7.3.

• PMM-5393: There’s a new ‘Node Summary’ row in the services Summary and Details dashboards
summarizing the system update, load average, RAM and memory.

• PMM-4778: mongodb_exporter is now the latest version, 0.11.0.

• PMM-5734: Temporary files activity and utilization charts ( rate & irate ) were added to the
PostgreSQL Instance overview.

• PMM-5695: The error message explains better when using the –-socket option incorrectly.

7.18.3 Bugs Fixed

• PMM-4829: The MongoDB Exporter wasn’t able to collect metrics from hidden nodes without either the
latest driver or using the connect-direct parameter.

• PMM-5056: The average values for Query time in the Details and Profile sections were different.

• PMM-2717: Updating MongoDB Exporter resolves an error ( Failed to execute find query on
'config.locks': not found. ) when used with shardedCluster 3.6.4.

• PMM-4541: MongoDB exporter metrics collection was including system collections from collStats and
indexStats , causing “log bloat”.

• PMM-5913: Only totals were shown in QAN when filtering on Cluster=MongoDB .

• PMM-5903: When applying a filter the QAN Overview was being refreshed twice.

• PMM-5821: The Compare button was missing from HA Dashboard main menus.

345 of 359 Percona LLC, © 2021

 7.18.3 Bugs Fixed

• PMM-5687: Cumulative charts for Disk Details were not showing any data if metrics were returning
NaN results.

• PMM-5663: The ‘version’ value was not being refreshed in various MySQL dashboards.

• PMM-5643: Advanced Data Exploration charts were showing ‘N/A’ for Metric Resolution and ‘No data to
show’ in the Metric Data Table.

• PMM-4756: Dashboards were not showing services with empty environments.

• PMM-4562: MongoDB and MySQL registered instances with empty cluster labels ( –environment=<label>
) were not visible in the dashboard despite being added instances.

• PMM-4906: The MongoDB exporter for MongoDB 4.0 and above was causing a “log bloat” condition.

Help us improve our software quality by reporting any bugs you encounter using our bug tracking system.

346 of 359 Percona LLC, © 2021

 7.19 Percona Monitoring and Management 2.5.0

7.19 Percona Monitoring and Management 2.5.0

Date: April 14, 2020
Installation: Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance.

7.19.1 New Features

• PMM-5042 and PMM-5272: PMM can now connect to MySQL instances by specifying a UNIX socket. This
can be done with a new --socket option of the pmm-admin add mysql command. (Note: Updates to
both PMM Client and PMM Server were done to allow UNIX socket connections.)

• PMM-4145: Amazon RDS instance metrics can now be independently enabled/disabled for Basic and/or
Enhanced metrics.

7.19.2 Improvements

• PMM-5581: PMM Server Grafana plugins can now be updated on the command line with the grafana-
cli command-line utility.

• PMM-5536: Three Grafana plugins were updated to the latest versions: vertamedia-clickhouse-
datasource to 1.9.5, grafana-polystat-panel to 1.1.0, and grafana-piechart-panel to 1.4.0.

• PMM-4252: The resolution of the PMM Server favicon image has been improved.

7.19.3 Bugs Fixed

• PMM-5547: PMM dashboards were failing when presenting data from more than 100 monitored
instances (error message proxy error: context canceled ).

• PMM-5624: Empty charts were being shown in some Node Temperature dashboards.

• PMM-5637: The Data retention value in Settings was incorrectly showing the value as minutes instead
of days.

• PMM-5613: Sorting data by Query Time was not working properly in Query Analytics.

• PMM-5554: Totals in charts were inconsistently plotted with different colors across charts.

• PMM-4919: The force option ( --force ) in pmm-admin config was not always working.

• PMM-5351: The documentation on MongoDB user privileges has been corrected.

Help us improve our software quality by reporting any bugs you encounter using our bug tracking system.

347 of 359 Percona LLC, © 2021

 7.20 Percona Monitoring and Management 2.4.0

7.20 Percona Monitoring and Management 2.4.0

Date: March 18, 2020
Installation: Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance.

7.20.1 New Features

• PMM-3387: Prometheus custom configuration is now supported by PMM Server. The feature is targeted
at experienced users and is done by adding the base configuration file into the PMM Server container to
be parsed and included into the managed Prometheus configuration.

• PMM-5186: Including –-pprof option in the pmm-admin summary command adds pprof debug profiles
to the diagnostics data archive

• PMM-5102: The new “Node Details” dashboard now displays data from the hardware monitoring
sensors in hwmon . The new dashboard is based on the hwmon collector data from the node_exporter .
Please note that data may be unavailable for some nodes because of the configuration or virtualization
parameters.

7.20.2 Improvements

• PMM-4915: The Query Analytics dashboard now shows Time Metrics in the Profile Section as “AVG per
query” instead of “AVG per second”

• PMM-5470: ClickHouse query optimized for Query Analytics to improve its speed and reduce the load
on the back-end

• PMM-5448: The default high and medium metrics resolutions were changed to 1-5-30 and 5-10-60 sec.
To reduce the effect of this change on existing installations, systems having the “old” high resolution
chosen on the PMM Settings page (5-5-60 sec.) will be automatically re-configured to the medium one
during an upgrade. If the resolution was changed to some custom values via API, it will not be affected

• PMM-5531: A health check indicator was implemented for the PMM Server Docker image. It is based on
the Docker HEALTHCHECK. This feature can be used as follows:

docker inspect -f {{.State.Health.Status}}

until [ "`docker inspect -f {{.State.Health.Status}} pmm-server`" == "healthy" ]; do sleep
1; done

• PMM-5489: The “Total” line in all charts is now drawn with the same red color for better consistency

• PMM-5461: Memory graphs on the node-related dashboards were adjusted to have fixed colors that
are more distinguishable from each other

• PMM-5329: Prometheus in PMM Server was updated to version 2.16.0. This update has brought several
improvements. Among them are significantly reduced memory footprint of the loaded TSDB blocks,
lower memory footprint for the compaction process (caused by the more balanced choice of what to
buffer during compaction), and improved query performance for the queries that only touch the most
recent 2 hours of data.

• PMM-5210: Data Retention is now specified in days instead of seconds on the PMM Settings page.
Please note this is a UI-only change, so the actual data retention precision is not changed

• PMM-5182: The logs.zip archive available on the PMM Settings page now includes additional self-
monitoring information in a separate client subfolder. This subfolder contains information collected

348 of 359 Percona LLC, © 2021

 7.20.3 Bugs Fixed

on the PMM Server and is equivalent to the one collected on a node by the pmm-admin summary
command.

• PMM-5112: The Inventory API List requests now can be filtered by the Node/Service/Agent type

7.20.3 Bugs Fixed

• PMM-5178: Query Detail Section of the Query Analytics dashboard didn’t show tables definitions and
indexes for the internal PostgreSQL database

• PMM-5465: MySQL Instance related dashboards had row names not always matching the actual
contents. To fix this, elements were re-ordered and additional rows were added for better matching of
the row name and the corresponding elements

• PMM-5455: Dashboards from the Insight menu were fixed to work correctly when the low resolution is
set on the PMM Settings page

• PMM-5446: A number of the Compare Dashboards were fixed to work correctly when the low resolution
is set on the PMM Settings page

• PMM-5430: MySQL Exporter section on the Prometheus Exporter Status dashboard now collapsed by
default to be consistent with other database-related sections

• PMM-5445, PMM-5439, PMM-5427, PMM-5426, PMM-5419: Labels change (which occurs e.g. when the
metrics resolution is changed on the PMM Settings page) was breaking dashboards

• PMM-5347: Selecting queries on the Query Analytics dashboard was generating errors in the browser
console

• PMM-5305: Some applied filters on the Query Analytics dashboard were not preserved after changing
the time range

• PMM-5267: The Refresh button was not working on the Query Analytics dashboard

• PMM-5003: pmm-admin list and status use different JSON naming for the same data

• PMM-5526: A typo was fixed in the Replication Dashboard description tooltip

Help us improve our software quality by reporting any bugs you encounter using our bug tracking system.

349 of 359 Percona LLC, © 2021

 7.21 Percona Monitoring and Management 2.3.0

7.21 Percona Monitoring and Management 2.3.0

Date: February 19, 2020

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance.

For PMM install instructions, see Installing PMM Server and Installing PMM Client.

Important PMM 2 is designed to be used as a new installation — please don’t try to upgrade your existing
PMM 1 environment.

7.21.1 Improvements and new features

• PMM-5064 and PMM-5065: Starting from this release, users will be able to integrate PMM with an
external Alertmanager by specifying the Alertmanager URL and the Alert Rules to be executed inside the
PMM server (This feature is for advanced users only at this point)

• PMM-4954: Query Analytics dashboard now shows units both in the list of queries in a summary table
and in the Details section to ease understanding of the presented data

• PMM-5179: Relations between metrics are now specified in the Query Analytics Details section

• PMM-5115: The CPU frequency and temperature graphs were added to the CPU Utilization dashboard

• PMM-5394: A special treatment for the node-related dashboards was implemented for the situations
when the data resolution change causes new metrics to be generated for existing nodes and services,
to make graphs show continuous lines of the same colors

7.21.2 Fixed bugs

• PMM-4620: The high CPU usage by the pmm-agent process related to MongoDB Query Analytics was
fixed

• PMM-5377: singlestats showing percentage had sparklines scaled vertically along with the graph
swing, which made it difficult to visually notice the difference between neighboring singlestats .

• PMM-5204: Changing resolution on the PMM settings page was breaking some singlestats on the
Home and MySQL Overview dashboards

• PMM-5251: Vertical scroll bars on the graph elements were not allowed to do a full scroll, making last
rows of the legend unavailable for some graphs

• PMM-5410: The “Available Downtime before SST Required” chart on the PXC/Galera Node Summary
dashboard was not showing data because it was unable to use metrics available with different scraping
intervals

350 of 359 Percona LLC, © 2021

 7.22 Percona Monitoring and Management 2.2.2

7.22 Percona Monitoring and Management 2.2.2

Date: February 4, 2020

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance.

For PMM install instructions, see Installing PMM Server and Installing PMM Client.

Important PMM 2 is designed to be used as a new installation — please don’t try to upgrade your existing
PMM 1 environment.

7.22.1 Improvements and new features

• PMM-5321: The optimization of the Query Analytics parser code for PostgreSQL queries allowed us to
reduce the memory resources consumption by 1-5%, and the parsing time of an individual query by 30
to 40%

• PMM-5184: The pmm-admin summary command have gained a new --skip-server flag which makes it
operating in a local-only mode, creating summary file without contacting the PMM Server

7.22.2 Fixed bugs

• PMM-5340: The Scraping Time Drift graph on the Prometheus dashboard was showing wrong values
because the actual metrics resolution wasn’t taken into account

• PMM-5060: Query Analytics Dashboard did not show the row with the last query of the first page, if the
number of queries to display was 11

351 of 359 Percona LLC, © 2021

 7.23 Percona Monitoring and Management 2.2.1

7.23 Percona Monitoring and Management 2.2.1

Date: January 23, 2020

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance.

For PMM install instructions, see Installing PMM Server and Installing PMM Client.

Important PMM 2 is designed to be used as a new installation — please don’t try to upgrade your existing
PMM 1 environment.

PMM Server version 2.2.0 suffered an unauthenticated denial of service vulnerability (CVE-2020-7920). Any
other PMM versions do not carry the same code logic, and are thus unaffected by this issue. Users who
have already deployed PMM Server 2.2.0 are advised to upgrade to version 2.2.1 which resolves
this issue.

7.23.1 Improvements and new features

• PMM-5229: The new RDS Exporter section added to the Prometheus Exporter Status dashboard shows
singlestats and charts related to the rds_exporter

• PMM-5228 and PMM-5238: The Prometheus dashboard and the Exporters Overview dashboard were
updated to include the rds_exporter metrics in their charts, allowing better understanding of the
impacts of monitoring RDS instances

• PMM-4830: The consistency of the applied filters between the Query Analytics and the Overview
dashboards was implemented, and now filters selected in QAN will continue to be active after the switch
to any of the Overview dashboards available in the Services menu

• PMM-5235: The DB uptime singlestats in node rows on the Home dashboard were changed to show
minimal values instead of average ones to be consistent with the top row

• PMM-5127: The “Search by” bar on the Query Analytics dashboard was renamed to “Filter by” to make
its purpose more clear

• PMM-5131: The Filter panel on the Query Analytics dashboard now shows the total number of available
Labels within the “See all” link, which appears if the Filter panel section shows only top 5 of its Labels

7.23.2 Fixed bugs

• PMM-5232: The pmm-managed component of the PMM Server 2.2.0 is vulnerable to DoS attacks, that
could be carried out by anyone who knows the PMM Server IP address (CVE-2020-7920). Versions
other than 2.2.0 are not affected.

• PMM-5226: The handlebars package was updated to version 4.5.3 because of the Prototype Pollution
vulnerability in it (CVE-2019-19919). Please note PMM versions were not affected by this vulnerability,
as handlebars package is used as a build dependency only.

• PMM-5206: Switching to the Settings dashboard was breaking the visual style of some elements on the
Home dashboard

• PMM-5139: The breadcrumb panel, which shows all dashboards visited within one session starting
from the root, was unable to fully show breadcrumb longer than one line

• PMM-5212: The explanatory text was added to the Download PMM Server Logs button in the Diagnostic
section of the PMM Settings dashboard, and a link to it was added to the Prometheus dashboard which
was the previous place to download logs

352 of 359 Percona LLC, © 2021

 7.23.2 Fixed bugs

• PMM-5215: The unneeded mariadb-libs package was removed from the PMM Server 2.2.0 OVF image,
resulting in both faster updating with the yum update command and avoiding dependency conflict
messages in the update logs

• PMM-5216: PMM Server Upgrade to 2.2.0 was showing Grafana Update Error page with the Refresh
button which had to be clicked to start using the updated version

• PMM-5211: The “Where do I get the security credentials for my Amazon RDS DB instance” link in the
Add AWS RDS MySQL or Aurora MySQL instance dialog was not targeted at the appropriate instruction

• PMM-5217: PMM 2.x OVF Image memory size was increased from 1 Gb to 4 Gb with the additional 1
Gb swap space because the previous amount was hardly housing the PMM Server, and it wasn’t enough
in some cases like performing an upgrade

• PMM-5271: LVM logical volumes were wrongly resized on AWS deployment, resulting in “no space left
on device” errors

• PMM-5295: InnoDB Transaction Rollback Rate values on the MySQL InnoDB Details dashboard were
calculated incorrectly

• PMM-5270: PXC/Galera Cluster Summary dashboard was showing empty Cluster drop-down list,
making it impossible to choose the cluster name

• PMM-4769: The wrongly named “Timeout value used for retransmitting” singlestat on the Network
Details dashboard was renamed to “The algorithm used to determine the timeout value” and updated
to show the algorithm name instead of a digital code

• PMM-5260: Extensive resource consumption by pmm-agent took place in case of Query Analytics for
PostgreSQL; it was fixed by a number of optimizations in the code, resulting in about 4 times smaller
memory usage

• PMM-5261: CPU usage charts on all dashboards which contain them have undergone colors update to
make softIRQ and Steal curves better differentiated

• PMM-5244: High memory consumption in the PMM Server with a large number of agents sending data
simultaneously was fixed by improving bulk data insertion to the ClickHouse database

353 of 359 Percona LLC, © 2021

 7.24 Percona Monitoring and Management 2.2.0

7.24 Percona Monitoring and Management 2.2.0

Date: December 24, 2019

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance. You can run PMM in your own environment for
maximum security and reliability. It provides thorough time-based analysis for MySQL, MongoDB, and
PostgreSQL servers to ensure that your data works as efficiently as possible.

Main improvements in this release are:

• Alternative installation methods available for PMM 1.x are re-implemented for PMM 2: now PMM Server
can be installed as a virtual appliance, or run using AWS Marketplace

• AWS RDS and remote instances monitoring re-added in this release include AWS RDS MySQL / Aurora
MySQL instances, and remote PostgreSQL, MySQL, MongoDB, and ProxySQL ones

• The new Settings dashboard allows configuring PMM Server via the graphical interface

For PMM install instructions, see Installing PMM Server and Installing PMM Client.

Important PMM 2 is designed to be used as a new installation — please don’t try to upgrade your existing
PMM 1 environment.

7.24.1 Improvements and new features

• PMM-4575: The new PMM Settings dashboard allows users to configure various PMM Server options:
setting metrics resolution and data retention, enabling or disabling send usage data statistics back to
Percona and checking for updates; this dashboard is now the proper place to upload your public key
for the SSH login and to download PMM Server logs for diagnostics

• PMM-4907 and PMM-4767: The user’s AMI Instance ID is now used to setup running PMM Server using
AWS Marketplace as an additional verification on the user, based on the Amazon Marketplace rules

• PMM-4950 and PMM-3094: Alternative AWS partitions are now supported when adding an AWS RDS
MySQL or Aurora MySQL Instance to PMM

• PMM-4976: Home dashboard clean-up: “Systems under monitoring” and “Network IO” singlestats
were refined to be based on the host variable; also avoiding using color as an indicator of state; “All”
row elements were relinked to the “Nodes Overview” dashboard with regards to the selected host.

• PMM-4800: The pmm-admin add mysql command has been modified to make help text more descriptive:
now when you enable tablestats you will get more detail on if they’re enabled for your environment
and where you stand with respect to the auto-disable limit

• PMM-4969: Update Grafana to version 6.5.1

• PMM-5053: A tooltip was added to the Head Block graph on the Prometheus dashboard

• PMM-5068: Drill-down links were added to the Node Summary dashboard graphs

• PMM-5050: Drill-down links were added to the graphs on all Services Compare dashboards

• PMM-5037: Drill-down links were added to all graphs on the Services Overview dashboards

• PMM-4988: Filtering in Query Analytics have undergone improvements to make group selection more
intuitive: Labels unavailable under the current selection are shown as gray/disabled, and the
percentage values are dynamically recalculated to reflect Labels available within the currently applied
filters

• PMM-4966: All passwords are now substituted with asterisk signs in the exporter logs for security
reasons when not in debug mode

354 of 359 Percona LLC, © 2021

 7.24.2 Fixed bugs

• PMM-527: node_exporter is now providing hardware monitoring information such as CPU

temperatures and fan statuses; while this information is being collected by PMM Server, it will not be
shown until a dedicated dashboard is added in a future release

• PMM-3198: Instead of showing All graphs for all services by default, MySQL Command/Handler
Counters Compare dashboard now shows the predefined set of ten most informative ones, to reduce
load on PMM Server at its first open

7.24.2 Fixed bugs

• PMM-4978: The “Top MySQL Questions” singlestat on the MySQL Instances Overview dashboard was
changed to show ops instead of percentage

• PMM-4917: The “Systems under monitoring” and “Monitored DB Instances” singlestats on the Home
dashboard now have a sparkline to make situation more clear with recently shut down nodes/instances

• PMM-4979: Set decimal precision 2 for all the elements, including charts and singlestats , on all
dashboards

• PMM-4980: Fix “Load Average” singlestat on the Node Summary dashboard to show decimal value
instead of percent

• PMM-4981: Disable automatic color gradient in filled graphs on all dashboards

• PMM-4941: Some charts were incorrectly showing empty fragments with high time resolution turned on

• PMM-5022: Fix outdated drill-down links on the Prometheus Exporters Overview and Nodes Overview
dashboards

• PMM-5023: Make the All instances uptime singlestat on the Home dashboard to show Min values
instead of Avg

• PMM-5029: Option to upload dashboard snapshot to Percona was disappearing after upgrade to 2.1.x

• PMM-4946: Rename singlestats on the Home dashboard for better clarity: “Systems under monitoring”
to “Nodes under monitoring” and “Monitored DB Instances” to “Monitored DB Services”, and make the
last one to count remote DB instances also

• PMM-5015: Fix format of Disk Page Buffers singlestat on the Compare dashboard for PostgreSQL to
have two digits precision for the consistency with other singlestats

• PMM-5014: LVM logical volumes were wrongly sized on a new AWS deployment, resulting in “no space
left on device” errors.

• PMM-4804: Incorrect parameters validation required both service-name and service-id parameters
of the pmm-admin remove command to be presented, while the command itself demanded only one of
them to identify the service.

• PMM-3298: Panic errors were present in the rds_exporter log after adding an RDS instance from the
second AWS account

• PMM-5089: The serialize-javascript package was updated to version 2.1.1 because of the possibility
of regular expressions cross-site scripting vulnerability in it (CVE-2019-16769). Please note PMM
versions were not affected by this vulnerability, as serialize-javascript package is used as a build
dependency only.

• PMM-5149: Disk Space singlestat was unable to show data for RDS instances because of not taking
into account sources with unknown file system type

355 of 359 Percona LLC, © 2021

 7.25 Percona Monitoring and Management 2.1.0

7.25 Percona Monitoring and Management 2.1.0

Date: November 11, 2019

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance. You can run PMM in your own environment for
maximum security and reliability. It provides thorough time-based analysis for MySQL, MongoDB, and
PostgreSQL servers to ensure that your data works as efficiently as possible.

For install instructions, see Installing Percona Monitoring and Management.

Important PMM 2 is designed to be used as a new installation — please don’t try to upgrade your existing
PMM 1 environment.

7.25.1 Improvements and new features

• PMM-4063: Update QAN filter panel to show only labels available for selection under currently applied
filters

• PMM-815: Latency Detail graph added to the MongoDB Instance Summary dashboard

• PMM-4768: Disable heavy-load collectors automatically when there are too many tables

• PMM-4821: Use color gradient in filled graphs on all dashboards

• PMM-4733: Add more log and configuration files to the downloadable logs.zip archive

• PMM-4672: Use integer percentage values in QAN filter panel

• PMM-4857: Update tooltips for all MongoDB dashboards

• PMM-4616: Rename column in the Query Details section in QAN from Total to Sum

• PMM-4770: Use Go 1.12.10

• PMM-4780: Update Grafana to version 6.4.1

• PMM-4918: Update Grafana plugins to newer versions, including the clickhouse-datasource plugin

7.25.2 Fixed bugs

• PMM-4935: Wrong instance name displayed on the MySQL Instance Summary dashboard due to the
incorrect string crop

• PMM-4916: Wrong values are shown when changing the time range for the Node Summary Dashboard
in case of remote instances

• PMM-4895 and PMM-4814: The update process reports completion before it is actually done and
therefore some dashboards, etc. may not be updated

• PMM-4876: PMM Server access credentials are shown by the pmm-admin status command instead of
hiding them for security reasons

• PMM-4875: PostgreSQL error log gets flooded with warnings when pg_stat_statements extension is
not installed in the database used by PMM Server or when PostgreSQL user is unable to connect to it

• PMM-4852: Node name has an incorrect value if the Home dashboard opened after QAN

• PMM-4847: Drill-downs from the Environment Overview dashboard doesn’t show data for the
preselected host

• PMM-4841 and PMM-4845: pg_stat_statement QAN Agent leaks database connections

• PMM-4831: Clean-up representation of selectors names on MySQL-related dashboards for a better

consistency

356 of 359 Percona LLC, © 2021

 7.25.2 Fixed bugs

• PMM-4824: Incorrectly calculated singlestat values on MySQL Instances Overview dashboard

• PMM-4819: In case of the only one monitored host, its uptime is shown as a smaller value than the all
hosts uptime due to the inaccurate rounding

• PMM-4816: Set equal thresholds to avoid confusing singlestat color differences on a Home dashboard

• PMM-4718: Labels are not fully displayed in the filter panel of the Query Details section in QAN

• PMM-4545: Long queries are not fully visible in the Query Examples section in QAN

Help us improve our software quality by reporting any Percona Monitoring and Management bugs you
encounter using our bug tracking system.

357 of 359 Percona LLC, © 2021

 7.26 Percona Monitoring and Management 2.0.1

7.26 Percona Monitoring and Management 2.0.1

Date: October 9, 2019

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance. You can run PMM in your own environment for
maximum security and reliability. It provides thorough time-based analysis for MySQL, MongoDB, and
PostgreSQL servers to ensure that your data works as efficiently as possible.

For install instructions, see Installing Percona Monitoring and Management.

Important PMM 2 is designed to be used as a new installation — please don’t try to upgrade your existing
PMM 1 environment.

7.26.1 Improvements

• PMM-4779: Securely share dashboards with Percona

• PMM-4735: Keep one old slowlog file after rotation

• PMM-4724: Alt+click on check updates button enables force-update

• PMM-4444: Return “what’s new” URL with the information extracted from the pmm-update package
change log

7.26.2 Fixed bugs

• PMM-4758: Remove Inventory rows from dashboards

• PMM-4757: qan_mysql_perfschema_agent failed querying events_statements_summary_by_digest due to

data types conversion

• PMM-4755: Fixed a typo in the InnoDB AHI Miss Ratio formula

• PMM-4749: Navigation from Dashboards to QAN when some Node or Service was selected now applies
filtering by them in QAN

• PMM-4742: General information links were updated to go to PMM 2 related pages

• PMM-4739: Remove request instances list

• PMM-4734: A fix was made for the collecting node_name formula at MySQL Replication Summary
dashboard

• PMM-4729: Fixes were made for formulas on MySQL Instances Overview

• PMM-4726: Links to services in MongoDB singlestats didn’t show Node name

• PMM-4720: machine_id could contain trailing \\n

• PMM-4640: It was not possible to add MongoDB remotely if password contained a # symbol

Help us improve our software quality by reporting any Percona Monitoring and Management bugs you
encounter using our bug tracking system.

358 of 359 Percona LLC, © 2021

 7.27 Percona Monitoring and Management 2.0.0

7.27 Percona Monitoring and Management 2.0.0

Date: September 19, 2019

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and
monitoring MySQL, MongoDB, and PostgreSQL performance. You can run PMM in your own environment for
maximum security and reliability. It provides thorough time-based analysis for MySQL, MongoDB, and
PostgreSQL servers to ensure that your data works as efficiently as possible.

For install instructions, see Installing Percona Monitoring and Management.

Important PMM 2 is designed to be used as a new installation — please don’t try to upgrade your existing
PMM 1 environment.

The new PMM2 introduces a number of enhancements and additional feature improvements, including:

• Detailed query analytics and filtering technologies which enable you to identify issues faster than ever
before.

• A better user experience: Service-level dashboards give you immediate access to the data you need.

• The new addition of PostgreSQL query tuning.

• Enhanced security protocols to ensure your data is safe.

• Our new API allows you to extend and interact with third-party tools.

More details about new and improved features available within the release can be found in the
corresponding blog post.

Help us improve our software quality by reporting any Percona Monitoring and Management bugs you
encounter using our bug tracking system.

359 of 359 Percona LLC, © 2021