check_postgres.pl - a Postgres monitoring script for Nagios, MRTG, and others This documents describes check_postgres.pl version 2.3.4
## Create all symlinks check_postgres.pl --symlinks
## Check connection to Postgres database 'pluto': check_postgres.pl --action=connection --db=pluto
## Same things, but using the symlink check_postgres_connection --db=pluto
## Warn if > 100 locks, critical if > 200, or > 20 exclusive check_postgres_locks --warning=100 --critical="total=200;exclusive=20"
## Show the current number of idle connections on port 6543: check_postgres_txn_idle --port=6543 --output=simple
## There are many other actions and options, please keep reading.
The latest news and documentation can always be found at: http://bucardo.org/check_postgres/
check_postgres.pl is a Perl script that runs many different tests against one or more Postgres databases. It uses the psql program to gather the information, and outputs the results in one of three formats: Nagios, MRTG, or simple.
The output can be changed by use of the --output option. The default output
is nagios, although this can be changed at the top of the script if you wish. The
current option choices are nagios, mrtg, and simple. To avoid having to
enter the output argument each time, the type of output is automatically set
if no --output argument is given, and if the current directory has one of the
output options in its name. For example, creating a directory named mrtg and
populating it with symlinks via the --symlinks argument would ensure that
any actions run from that directory will always default to an output of "mrtg"
As a shortcut for --output=simple, you can enter --simple, which also overrides
the directory naming trick.
The default output format is for Nagios, which is a single line of information, along with four specific exit codes:
The output line is one of the words above, a colon, and then a short description of what was measured. Additional statistics information, as well as the total time the command took, can be output as well: see the documentation on the arguments --showperf, --perflimit, and --showtime.
The MRTG output is four lines, with the first line always giving a single number of importance. When possible, this number represents an actual value such as a number of bytes, but it may also be a 1 or a 0 for actions that only return "true" or "false", such as check_postgres_version. The second line is an additional stat and is only used for some actions. The third line indicates an "uptime" and is not used. The fourth line is a description and usually indicates the name of the database the stat from the first line was pulled from, but may be different depending on the action.
Some actions accept an optional --mrtg argument to further control the output.
See the documentation on each action for details on the exact MRTG output for each one.
The simple output is simply a truncated version of the MRTG one, and simply returns the first number and nothing else. This is very useful when you just want to check the state of something, regardless of any threshold.
All actions accept a common set of database options. At least one is required.
Connect to the host indicated by NAME. Can be a comma-separated list of names. Multiple host arguments are allowed. If no host is given, defaults to a local Unix socket. You may also use "--dbhost".
Connects using the specified PORT number. Can be a comma-separated list of port numbers, and multiple port arguments are allowed. If no port number is given, the default is 5432. You may also use "--dbport"
Specifies which database to connect to. Can be a comma-separated list of names, and multiple dbname arguments are allowed. If no dbname option is provided, defaults to 'postgres' if psql is version 8 or greater, and 'template1' otherwise.
The name of the database user to connect as. Can be a comma-separated list of usernames, and multiple dbuser arguments are allowed. If this is not provided, the default is 'postgres'.
Provides the password to connect to the database with. Use of this option is highly discouraged. Instead, one should use a .pgpass file.
The database connection options can be grouped: --host=a,b --host=c --port=1234 --port=3344 would connect to a-1234, b-1234, and c-3344. Note that once set, an option carries over until it is changed again.
Examples:
--host=a,b --port=5433 --db=c Connects twice to port 5433, using database c, to hosts a and b: a-5433-c b-5433-c
--host=a,b --port=5433 --db=c,d Connects four times: a-5433-c a-5433-d b-5433-c b-5433-d
--host=a,b --host=foo --port=1234 --port=5433 --db=e,f Connects six times: a-1234-e a-1234-f b-1234-e b-1234-f foo-5433-e foo-5433-f
--host=a,b --host=x --port=5432,5433 --dbuser=alice --dbuser=bob -db=baz Connects three times: a-5432-alice-baz b-5433-alice-baz x-5433-bob-baz
Other options include:
States what action we are running. Required unless using a symlinked file, in which case the name of the file is used to figure out the action.
Sets the threshold at which a warning alert is fired. The valid options for this option depends on the action used.
Sets the threshold at which a critical alert is fired. The valid options for this option depends on the action used.
Sets the timeout in seconds after which the script will abort whatever it is doing and return an UNKNOWN status. The timeout is per Postgres cluster, not for the entire script. The default value is 10; the units are always in seconds.
Displays a help screen with a summary of all actions and options.
Shows the current version.
Set the verbosity level. Can call more than once to boost the level. Setting it to three
or higher (in other words, issuing -v -v -v) turns on debugging information for this
program which is sent to stderr.
Determines if we output additional performance data in standard Nagios format (at end of string, after a pipe symbol, using name=value). VAL should be 0 or 1. The default is 1. Only takes effect if using Nagios output mode.
Sets a limit as to how many items of interest are reported back when using the showperf option. This only has an effect for actions that return a large number of items, such as table_size. The default is 0, or no limit. Be careful when using this with the --include or --exclude options, as those restrictions are done after the query has been run, and thus your limit may not include the items you want. Only takes effect if using Nagios output mode.
Determines if the time taken to run each query is shown in the output. VAL should be 0 or 1. The default is 1. No effect unless showperf is on. Only takes effect if using Nagios output mode.
Enables test mode. See the TEST MODE section below.
Tells the script where to find the psql program. Useful if you have more than
one version of the psql executable on your system, or if there is no psql program
in your path. Note that this option is in all uppercase. By default, this option
is not allowed. To enable it, you must change the $NO_PSQL_OPTION near the
top of the script to 0. Avoid using this option if you can, and instead hard-code
your psql location into the $PSQL variable, also near the top of the script.
Creates symlinks to the main program for each action.
Determines the format of the output, for use in various programs. The default is 'nagios'. No other systems are supported yet.
Used only for the MRTG or simple output, for a few specific actions.
Outputs the exact string returned by psql, for use in debugging. The value is one or more letters, which determine if the output is displayed or not, where 'a' = all, 'c' = critical, 'w' = warning, 'o' = ok, and 'u' = unknown. Letters can be combined.
The script runs one or more actions. This can either be done with the --action flag, or by using a symlink to the main file that contains the name of the action inside of it. For example, to run the action "timesync", you may either issue:
check_postgres.pl --action=timesync
or use a program named:
check_postgres_timesync
All the symlinks are created for you in the current directory if use the option --symlinks
perl check_postgres.pl --symlinks
If the file name already exists, it will not be overwritten. If the file exists and is a symlink, you can force it to overwrite by using "--action=build_symlinks_force"
Most actions take a --warning and a --critical option, indicating at what point we change from OK to WARNING, and what point we go to CRITICAL. Note that because criticals are always checked first, setting the warning equal to the critical is an effective way to turn warnings off and always give a critical.
The current supported actions are:
(symlink: check_postgres_autovac_freeze) Checks how close each database is to the Postgres autovacuum_freeze_max_age setting. This
action will only work for databases version 8.2 or higher. The --warning and
--critical options should be expressed as percentages. The 'age' of the transactions
in each database is compared to the autovacuum_freeze_max_age setting (200 million by default)
to generate a rounded percentage. The default values are 90% for the warning and 95% for
the critical. Databases can be filtered by use of the --include and --exclude options. See
the BASIC FILTERING section for more details.
Example 1: Give a warning when any databases on port 5432 are above 80%
check_postgres_autovac_freeze --port=5432 --warning="80%"
For MRTG output, the highest overall percentage is reported on the first line, and the highest age is reported on the second line. All databases which have the percentage from the first line are reported on the fourth line, separated by a pipe symbol.
(symlink: check_postgres_backends) Checks the current number of connections for one or more databases, and optionally
compares it to the maximum allowed, which is determined by the
Postgres configuration variable max_connections. The --warning and
--critical options can take one of three forms. First, a simple number can be
given, which represents the number of connections at which the alert will be
given. This choice does not use the max_connections setting. Second, the
percentage of available connections can be given. Third, a negative number can
be given which represents the number of connections left until max_connections
is reached. The default values for --warning and --critical are '90%' and '95%'.
You can also filter the databases by use of the
--include and --exclude options. See the BASIC FILTERING section
for more details.
Example 1: Give a warning when the number of connections on host quirm reaches 120, and a critical if it reaches 150.
check_postgres_backends --host=quirm --warning=120 --critical=150
Example 2: Give a critical when we reach 75% of our max_connections setting on hosts lancre or lancre2.
check_postgres_backends --warning='75%' --critical='75%' --host=lancre,lancre2
Example 3: Give a warning when there are only 10 more connection slots left on host plasmid, and a critical when we have only 5 left.
check_postgres_backends --warning=-10 --critical=-5 --host=plasmid
Example 4: Check all databases except those with "test" in their name, but allow ones that are named "pg_greatest". Connect as port 5432 on the first two hosts, and as port 5433 on the third one. We want to always throw a critical when we reach 30 or more connections.
check_postgres_backends --dbhost=hong,kong --dbhost=fooey --dbport=5432 --dbport=5433 --warning=30 --critical=30 --exclude="~test" --include="pg_greatest,~prod"
For MRTG output, the number of connections is reported on the first line, and the fourth line gives the name of the database, plus the current maximum_connections. If more than one database has been queried, the one with the highest number of connections is output.
(symlink: check_postgres_bloat) Checks the amount of bloat in tables and indexes. (Bloat is generally the amount
of dead unused space taken up in a table or index. This space is usually reclaimed
by use of the VACUUM command.) This action requires that stats collection be
enabled on the target databases, and requires that ANALYZE is run frequently.
The --include and --exclude options can be used to filter out which tables
to look at. See the BASIC FILTERING section for more details.
The --warning and --critical options can be specified as sizes or percents. Valid size units are bytes, kilobytes, megabytes, gigabytes, terabytes, and exabytes. You can abbreviate all of those with the first letter. Items without units are assumed to be 'bytes'. The default values are '1 GB' and '5 GB'. The value represents the number of "wasted bytes", or the difference between what is actually used by the table and index, and what we compute that it should be.
Note that this action has two hard-coded values to avoid false alarms on
smaller relations. Tables must have at least 10 pages, and indexes at least 15,
before they can be considered by this test. If you really want to adjust these
values, you can look for the variables $MINPAGES and $MINIPAGES at the top of the
check_bloat subroutine.
Only the top 10 most bloated relations are shown. You can change this number by using the --perflimit option to set your own limit.
The schema named 'information_schema' is excluded from this test, as the only tables it contains are small and do not change.
Please note that the values computed by this action are not precise, and should be used as a guideline only. Great effort was made to estimate the correct size of a table, but in the end it is only an estimate. The correct index size is even more of a guess than the correct table size, but both should give a rough idea of how bloated things are.
Example 1: Warn if any table on port 5432 is over 100 MB bloated, and critical if over 200 MB
check_postgres_bloat --port=5432 --warning='100 M', --critical='200 M'
Example 2: Give a critical if table 'orders' on host 'sami' has more than 10 megs of bloat
check_postgres_bloat --host=sami --include=orders --critical='10 MB'
Example 3: Give a critical if table 'q4' on database 'sales' is over 50% bloated
check_postgres_bloat --db=sales --include=q4 --critical='50%'
For MRTG output, the first line gives the highest number of wasted bytes for the tables, and the
second line gives the highest number of wasted bytes for the indexes. The fourth line gives the database
name, table name, and index name information. If you want to output the bloat ration instead (how many
times larger the relation is compared to how large it should be), just pass in --mrtg=ratio.
(symlink: check_postgres_connection) Simply connects, issues a 'SELECT version()', and leaves.
Takes no --warning or --critical options.
For MRTG output, simply outputs a 1 (good connection) or a 0 (bad connection) on the first line.
(symlink: check_postgres_custom_query) Runs a custom query of your choosing, and parses the results. The query itself is passed in through
the custom_query argument, and should be kept as simple as possible. If at all possible, wrap it in
a view or a function to keep things easier to manage. The query should return one or two columns: the first
is the result that will be checked, and the second is any performance data you want sent.
At least one warning or critical argument must be specified. What these are set to depends on the type of
query you are running. There are four types of custom_queries that can be run, specified by the checktype
argument. If none is specified, this action defaults to 'integer'. The four types are:
integer: Does a simple integer comparison. The first column should be a simple integer, and the warning and critical values should be the same.
string: The warning and critical are strings, and are triggered only if the value in the first column matches it exactly. This is case-sensitive.
time: The warning and the critical are times, and can have units of seconds, minutes, hours, or days. Each may be written singular or abbreviated to just the first letter. If no units are given, seconds are assumed. The first column should be an integer representing the number of seconds to check.
size: The warning and the critical are sizes, and can have units of bytes, kilobytes, megabytes, gigabytes, terabytes, or exabytes. Each may be abbreviated to the first letter. If no units are given, bytes are assumed. The first column should be an integer representing the number of bytes to check.
Normally, an alert is triggered if the values returned are greater than or equal to the critical or warning value. However, an option of --reverse will trigger the alert if the returned value is lower than or equal to the critical or warning value.
Example 1: Warn if any relation over 100 pages is named "rad":
check_postgres_custom_query --checktype=string -w "rad" --query="SELECT relname FROM pg_class WHERE relpages > 100" --port=5432
Example 2: Give a critical if the "foobar" function returns a number over 5MB:
check_postgres_custom_query --port=5432 --critical='5MB'--checktype=size --query="SELECT foobar()"
Example 2: Warn if the function "snazzo" returns less than 42:
check_postgres_custom_query --port=5432 --critical=42 --query="SELECT snazzo()" --reverse
If you come up with a useful custom_query, consider sending in a patch to this program to make it into a standard action that other people can use.
This action does not support MRTG or simple output yet.
(symlink: check_postgres_database_size) Checks the size of all databases and complains when they are too big.
There is no need to run this command more than once per database cluster.
Databases can be filtered with
the --include and --exclude options. See the BASIC FILTERING section
for more details.
They can also be filtered by the owner of the database with the
--includeuser and --excludeuser options.
See the USER NAME FILTERING section for more details.
The warning and critical options can be specified as bytes, kilobytes, megabytes, gigabytes, terabytes, or exabytes. Each may be abbreviated to the first letter as well. If no unit is given, the units are assumed to be bytes. There are not defaults for this action: the warning and critical must be specified. The warning value cannot be greater than the critical value. The output returns all databases sorted by size largest first, showing both raw bytes and a "pretty" version of the size.
Example 1: Warn if any database on host flagg is over 1 TB in size, and critical if over 1.1 TB.
check_postgres_database_size --host=flagg --warning='1 TB' --critical='1.1 t'
Example 2: Give a critical if the database template1 on port 5432 is over 10 MB.
check_postgres_database_size --port=5432 --include=template1 --warning='10MB' --critical='10MB'
Example 3: Give a warning if any database on host 'tardis' owned by the user 'tom' is over 5 GB
check_postgres_database_size --host=tardis --includeuser=tom --warning='5 GB' --critical='10 GB'
For MRTG output, returns the size in bytes of the largest database on the first line, and the name of the database on the fourth line.
(symlink: check_postgres_disk_space) Checks on the available physical disk space used by Postgres. This action requires
that you have the executable "/bin/df" available to report on disk sizes, and it
also needs to be run as a superuser, so it can examine the data_directory
setting inside of Postgres. The --warning and --critical options are
given in either sizes or percentages. If using sizes, the standard unit types
are allowed: bytes, kilobytes, gigabytes, megabytes, gigabytes, terabytes, or
exabytes. Each may be abbreviated to the first letter only; no units at all
indicates 'bytes'. The default values are '90%' and '95%'.
This command checks the following things to determine all of the different physical disks being used by Postgres.
data_directory - The disk that the main data directory is on.
log directory - The disk that the log files are on.
WAL file directory - The disk that the write-ahead logs are on (e.g. symlinked pg_xlog)
tablespaces - Each tablespace that is on a separate disk.
The output shows the total size used and available on each disk, as well as the percentage, ordered by highest to lowest percentage used. Each item above maps to a file system: these can be included or excluded. See the BASIC FILTERING section for more details.
Example 1: Make sure that no file system is over 90% for the database on port 5432.
check_postgres_disk_space --port=5432 --warning='90%' --critical="90%'
Example 2: Check that all file systems starting with /dev/sda are smaller than 10 GB and 11 GB (warning and critical)
check_postgres_disk_space --port=5432 --warning='10 GB' --critical='11 GB' --include="~^/dev/sda"
For MRTG output, returns the size in bytes of the file system on the first line, and the name of the file system on the fourth line.
(symlink: check_postgres_fsm_pages) Checks how close a cluster is to the Postgres max_fsm_pages setting.
This action will only work for databases of 8.2 or higher, and it requires the contrib
module pg_freespacemap be installed. The --warning and --critical options should be expressed
as percentages. The number of used pages in the free-space-map is determined by looking in the
pg_freespacemap_relations view, and running a formula based on the formula used for
outputting free-space-map pageslots in the vacuum verbose command. The default values are 85% for the
warning and 95% for the critical.
Example 1: Give a warning when our cluster has used up 76% of the free-space pageslots, with pg_freespacemap installed in database robert
check_postgres_fsm_pages --dbname=robert --warning="76%"
While you need to pass in the name of the database where pg_freespacemap is installed (and optionally a schema name if you have installed the module in a non-standard schema), you only need to run this check once per cluster. Also, checking this information does require obtaining special locks on the free-space-map, so it is recommend you do not run this check with short intervals.
For MRTG output, returns the percent of free-space-map on the first line, and the number of pages currently used on the second line.
(symlink: check_postgres_fsm_relations) Checks how close a cluster is to the Postgres max_fsm_relations setting.
This action will only work for databases of 8.2 or higher, and it requires the contrib module pg_freespacemap be
installed. The --warning and --critical options should be expressed as percentages. The number of used relations
in the free-space-map is determined by looking in the pg_freespacemap_relations view. The default values are 85% for
the warning and 95% for the critical.
Example 1: Give a warning when our cluster has used up 80% of the free-space relations, with pg_freespacemap installed in database dylan, in non-standard schema emma
check_postgres_fsm_relations --dbname=dylan --warning="75%" --schema=emma
While you need to pass in the name of the database where pg_freespacemap is installed (and optionally a schema name if you have installed the module in a non-standard schema), you only need to run this check once per cluster. Also, checking this information does require obtaining special locks on the free-space-map, so it is recommend you do not run this check with short intervals.
For MRTG output, returns the percent of free-space-map on the first line, the number of relations currently used on the second line.
(symlinks: check_postgres_index_size, check_postgres_table_size, and check_postgres_relation_size)
The actions table_size and index_size are simply variations of the
relation_size action, which checks for a relation that has grown too big.
Relations (in other words, tables and indexes) can be filtered with the
--include and --exclude options. See the BASIC FILTERING section
for more details. Relations can also be filtered by the user that owns them,
by using the --includeuser and --excludeuser options.
See the USER NAME FILTERING section for more details.
The values for the --warning and --critical options are file sizes, and may have units of bytes, kilobytes, megabytes, gigabytes, terabytes, or exabytes. Each can be abbreviated to the first letter. If no units are given, bytes are assumed. There are no default values: both the warning and the critical option must be given. The return text shows the size of the largest relation found.
If the --showperf option is enabled, all of the relations with their sizes
will be given. To prevent this, it is recommended that you set the
--perflimit option, which will cause the query to do a
ORDER BY size DESC LIMIT (perflimit).
Example 1: Give a critical if any table is larger than 600MB on host burrick.
check_postgres_table_size --critical='600 MB' --warning='600 MB' --host=burrick
Example 2: Warn if the table products is over 4 GB in size, and give a critical at 4.5 GB.
check_postgres_table_size --host=burrick --warning='4 GB' --critical='4.5 GB' --include=products
Example 3: Warn if any index not owned by postgres goes over 500 MB.
check_postgres_index_size --port=5432 --excludeuser=postgres -w 500MB -c 600MB
For MRTG output, returns the size in bytes of the largest relation, and the name of the database and relation as the fourth line.
(symlinks: check_postgres_last_vacuum, check_postgres_last_autovacuum, check_postgres_last_analyze, and
check_postgres_last_autoanalyze)
Checks how long it has been since vacuum (or analyze) was last run on each
table in one or more databases. Use of these actions requires that the target
database is version 8.3 or greater, or that the version is 8.2 and the
configuration variable stats_rows_level is enabled. Tables can be filtered with the
--include and --exclude options. See the BASIC FILTERING section
for more details.
Tables can also be filtered by their owner by use of the
--includeuser and --excludeuser options.
See the USER NAME FILTERING section for more details.
The units for --warning and --critical are specified as times. Valid units are seconds, minutes, hours, and days; all can be abbreviated to the first letter. If no units are given, 'seconds' are assumed. The default values are '1 day' and '2 days'. Please note that there are cases in which this field does not get automatically populated. If certain tables are giving you problems, make sure that they have dead rows to vacuum, or just exclude them from the test.
The schema named 'information_schema' is excluded from this test, as the only tables it contains are small and do not change.
Example 1: Warn if any table has not been vacuumed in 3 days, and give a critical at a week, for host wormwood
check_postgres_last_vacuum --host=wormwood --warning='3d' --critical='7d'
Example 2: Same as above, but skip tables belonging to the users 'eve' or 'mallory'
check_postgres_last_vacuum --host=wormwood --warning='3d' --critical='7d' --excludeusers=eve,mallory
For MRTG output, returns (on the first line) the LEAST amount of time in seconds since a table was last vacuumed or analyzed. The fourth line returns the name of the database and name of the table.
(symlink: check_postgres_listener) Confirm that someone is listening for one or more specific strings. Only one of warning or critical is needed. The format
is a simple string representing the LISTEN target, or a tilde character followed by a string for a regular expression
check.
Example 1: Give a warning if nobody is listening for the string bucardo_mcp_ping on ports 5555 and 5556
check_postgres_listener --port=5555,5556 --warning=bucardo_mcp_ping
Example 2: Give a critical if there are no active LISTEN requests matching 'grimm' on database oskar
check_postgres_listener --db oskar --critical=~grimm
For MRTG output, returns a 1 or a 0 on the first, indicating success or failure. The name of the notice must be provided via the <--mrtg> option.
(symlink: check_postgres_locks) Check the total number of locks on one or more databases. There is no
need to run this more than once per database cluster. Databases can be filtered
with the --include and --exclude options. See the BASIC FILTERING section
for more details.
The --warning and --critical options can be specified as simple numbers,
which represent the total number of locks, or they can be broken down by type of lock.
Valid lock names are 'total', 'waiting', or the name of a lock type used by Postgres.
These names are case-insensitive and do not need the "lock" part on the end,
so exclusive will match 'ExclusiveLock'. The format is name=number, with different
items separated by semicolons.
Example 1: Warn if the number of locks is 100 or more, and critical if 200 or more, on host garrett
check_postgres_locks --host=garrett --warning=100 --critical=200
Example 2: On the host artemus, warn if 200 or more locks exist, and give a critical if over 250 total locks exist, or if over 20 exclusive locks exist, or if over 5 connections are waiting for a lock.
check_postgres_locks --host=artemus --warning=200 --critical="total=250;waiting=5;exclusive=20"
For MRTG output, returns the number of locks on the first line, and the name of the database on the fourth line.
(symlink: check_postgres_logfile) Ensures that the logfile is in the expected location and is being logged to.
This action issues a command that throws an error on each database it is
checking, and ensures that the message shows up in the logs. It scans the
various log_* settings inside of Postgres to figure out where the logs should be.
If you are using syslog, it does a rough (but not foolproof) scan of
/etc/syslog.conf. Alternatively, you can provide the name of the logfile
with the --logfile option. This is especially useful if the logs have a
custom rotation scheme driven be an external program. The --logfile option
supports the following escape characters: %Y %m %d %H, which represent
the current year, month, date, and hour respectively. An error is always
reported as critical unless the warning option has been passed in as a non-zero
value. Other than that specific usage, the --warning and --critical
options should not be used.
Example 1: On port 5432, ensure the logfile is being written to the file /home/greg/pg8.2.log
check_postgres_logfile --port=5432 --logfile=/home/greg/pg8.2.log
Example 2: Same as above, but raise a warning, not a critical
check_postgres_logfile --port=5432 --logfile=/home/greg/pg8.2.log -w 1
For MRTG output, returns a 1 or 0 on the first line, indicating success or failure. In case of a failure, the fourth line will provide more detail on the failure encountered.
(symlink: check_postgres_query_runtime) Checks how long a specific query takes to run, by executing a "EXPLAIN ANALYZE"
against it. The --warning and --critical options are the maximum amount of
time the query should take. Valid units are seconds, minutes, and hours; any can be
abbreviated to the first letter. If no units are given, 'seconds' are assumed.
Both the warning and the critical option must be given. The name of the view or
function to be run must be passed in to the --queryname option. It must consist
of a single word (or schema.word), with optional parens at the end.
Example 1: Give a critical if the function named "speedtest" fails to run in 10 seconds or less.
check_postgres_query_runtime --queryname='speedtest()' --critical=10 --warning=10
For MRTG output, reports the time in seconds for the query to complete on the first line. The fourth line lists the database.
(symlink: check_postgres_query_time) Checks the length of running queries on one or more databases. There is
no need to run this more than once on the same database cluster.
Databases can be filtered
by using the --include and --exclude options. See the BASIC FILTERING
section for more details. You can also filter on the user running the
query with the --includeuser and --excludeuser options.
See the USER NAME FILTERING section for more details.
The values for the --warning and --critical options are amounts of time, and default to '2 minutes' and '5 minutes' respectively. Valid units are 'seconds', 'minutes', 'hours', or 'days'. Each may be written singular or abbreviated to just the first letter. If no units are given, the unit is assumed to be seconds.
Example 1: Give a warning if any query has been running longer than 3 minutes, and a critical if longer than 5 minutes.
check_postgres_query_time --port=5432 --warning='3 minutes' --critical='5 minutes'
Example 2: Using default values (2 and 5 minutes), check all databases except those starting with 'template'.
check_postgres_query_time --port=5432 --exclude=~^template
Example 3: Warn if user 'don' has a query running over 20 seconds
check_postgres_query_time --port=5432 --inclucdeuser=don --warning=20s
For MRTG output, returns the length in seconds of the longest running query on the first line. The fourth line gives the name of the database.
(symlink: check_postgres_sequence) Checks how much room is left on all sequences in the database.
This is measured as the percent of total possible values that have been used for each sequence.
The --warning and --critical options should be expressed as percentages. The default values
are 85% for the warning and 95% for the critical. You may use --include and --exclude to
control which sequences are to be checked. Note that this check does account for unusual minvalue
and increment by values, but does not care if the sequence is set to cycle or not.
The output for Nagios gives the name of the sequence, the percentage used, and the number of 'calls' left, indicating how many more times nextval can be called on that sequence before running into the maximum value.
The output for MRTG returns the highest percentage across all sequences on the first line, and the name of each sequence with that percentage on the fourth line, separated by a "|" (pipe) if there are more than one sequence at that percentage.
Example 1: Give a warning if any sequences are approaching 95% full.
check_postgres_sequence --dbport=5432 --warning=95%
Example 2: Check that the sequence named "orders_id_seq" is not more than half full.
check_postgres_sequence --dbport=5432 --critical=50% --include=orders_id_seq
(symlink: check_postgres_replicate_row) Checks that master-slave replication is working to one or more slaves.
The slaves are specified the same as the normal databases, except with
the number 2 at the end of them, so "--port2" instead of "--port", etc.
The values or the --warning and --critical options are units of time, and
at least one must be provided (no defaults). Valid units are 'seconds', 'minutes', 'hours',
or 'days'. Each may be written singular or abbreviated to just the first letter.
If no units are given, the units are assumed to be seconds.
This check updates a single row on the master, and then measures how long it
takes to be applied to the slaves. To do this, you need to pick a table that
is being replicated, then find a row that can be changed, and is not going
to be changed by any other process. A specific column of this row will be changed
from one value to another. All of this is fed to the repinfo option, and should
contain the following options, separated by commas: table name, primary key, key id,
column, first value, second value.
Example 1: Slony is replicating a table named 'orders' from host 'alpha' to host 'beta', in the database 'sales'. The primary key of the table is named id, and we are going to test the row with an id of 3 (which is historical and never changed). There is a column named 'salesrep' that we are going to toggle from a value of 'slon' to 'nols' to check on the replication. We want to throw a warning if the replication does not happen within 10 seconds.
check_postgres_replicate_row --host=alpha --dbname=sales --host2=beta --dbname2=sales --warning=10 --repinfo=orders,id,3,salesrep,slon,nols
Example 2: Bucardo is replicating a table named 'receipt' from host 'green' to hosts 'red', 'blue', and 'yellow'. The database for both sides is 'public'. The slave databases are running on port 5455. The primary key is named 'receipt_id', the row we want to use has a value of 9, and the column we want to change for the test is called 'zone'. We'll toggle between 'north' and 'south' for the value of this column, and throw a critical if the change is not on all three slaves within 5 seconds.
check_postgres_replicate_row --host=green --port2=5455 --host2=red,blue,yellow --critical=5 --repinfo=receipt,receipt_id,9,zone,north,south
For MRTG output, returns on the first line the time in seconds the replication takes to finish. The maximum time is set to 4 minutes 30 seconds: if no replication has taken place in that long a time, an error is thrown.
(symlink: check_postgres_txn_time) Checks the length of open transactions on one or more databases.
There is no need to run this command more than once per database cluster.
Databases can be filtered by use of the
--include and --exclude options. See the BASIC FILTERING section
for more details. The owner of the transaction can also be filtered, by use of
the --includeuser and --excludeuser options.
See the USER NAME FILTERING section for more details.
The values or the --warning and --critical options are units of time, and must be provided (no default). Valid units are 'seconds', 'minutes', 'hours', or 'days'. Each may be written singular or abbreviated to just the first letter. If no units are given, the units are assumed to be seconds.
This action requires Postgres 8.3 or better.
Example 1: Give a critical if any transaction has been open for more than 10 minutes:
check_postgres_txn_time --port=5432 --critical='10 minutes'
Example 1: Warn if user 'warehouse' has a transaction open over 30 seconds
check_postgres_txn_time --port-5432 --warning=30s --includeuser=warehouse
For MRTG output, returns the maximum time in seconds a transaction has been open on the first line. The fourth line gives the name of the database.
(symlink: check_postgres_txn_idle) Checks the length of "idle in transaction" queries on one or more databases. There is
no need to run this more than once on the same database cluster. Databases can be filtered
by using the --include and --exclude options. See the BASIC FILTERING
section below for more details.
The --warning and --critical options are given as units of time, and both must be provided (there are no defaults). Valid units are 'seconds', 'minutes', 'hours', or 'days'. Each may be written singular or abbreviated to just the first letter. If no units are given, the unit are assumed to be seconds.
This action requires Postgres 8.3 or better.
Example 1: Give a warning if any connection has been idle in transaction for more than 15 seconds:
check_postgres_txn_idle --port=5432 --warning='15 seconds'
For MRTG output, returns the time in seconds the longest idle transaction has been running. The fourth line returns the name of the database.
This action requires no other arguments, and does not connect to any databases, but simply creates symlinks in the current directory for each action, in the form check_postgres_<action_name>. If the file already exists, it will not be overwritten. If the action is rebuild_symlinks_force, then symlinks will be overwritten. The option --symlinks is a shorter way of saying --action=rebuild_symlinks
(symlink: check_postgres_settings_checksum) Checks that all the Postgres settings are the same as last time you checked.
This is done by generating a checksum of a sorted list of setting names and
their values. Note that different users in the same database may have different
checksums, due to ALTER USER usage, and due to the fact that superusers see more
settings than ordinary users. Either the --warning or the --critical option
should be given, but not both. The value of each one is the checksum, a
32-character hexadecimal value. You can run with the special --critical=0 option
to find out an existing checksum.
This action requires the Digest::MD5 module.
Example 1: Find the initial checksum for the database on port 5555 using the default user (usually postgres)
check_postgres_settings_checksum --port=5555 --critical=0
Example 2: Make sure no settings have changed and warn if so, using the checksum from above.
check_postgres_settings_checksum --port=5555 --warning=cd2f3b5e129dc2b4f5c0f6d8d2e64231
For MRTG output, returns a 1 or 0 indicating success of failure of the checksum to match. A
checksum must be provided as the --mrtg argument. The fourth line always gives the
current checksum.
(symlink: check_postgres_timesync) Compares the local system time with the time reported by one or more databases.
The --warning and --critical options represent the number of seconds between
the two systems before an alert is given. If neither is specified, the default values
are used, which are '2' and '5'. The warning value cannot be greater than the critical
value. Due to the non-exact nature of this test, values of '0' or '1' are not recommended.
The string returned shows the time difference as well as the time on each side written out.
Example 1: Check that databases on hosts ankh, morpork, and klatch are no more than 3 seconds off from the local time:
check_postgres_timesync --host=ankh,morpork.klatch --critical=3
For MRTG output, returns one the first line the number of seconds difference between the local time and the database time. The fourth line returns the name of the database.
(symlink: check_postgres_txn_wraparound) Checks how close to transaction wraparound one or more databases are getting.
The --warning and --critical options indicate the number of transactions
left, and must be a positive integer. If either option is not given, the default
values of 1.3 and 1.4 billion are used. There is no need to run this command
more than once per database cluster. For a more detailed discussion of what this
number represents and what to do about it, please visit the page
http://www.postgresql.org/docs/current/static/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND
The warning and value can have underscores in the number for legibility, as Perl does.
Example 1: Check the default values for the localhost database
check_postgres_txn_wraparound --host=localhost
Example 2: Check port 6000 and give a critical at 1.7 billion transactions left:
check_postgres_txn_wraparound --port=6000 --critical=1_700_000_000t
For MRTG output, returns the highest number of transactions for all databases on line one, while line 4 indicates which database it is.
(symlink: check_postgres_wal_files) Checks how many WAL files exist in the pg_xlog directory, which is found
off of your data_directory, sometimes as a symlink to another physical disk for
performance reasons. This action must be run as a superuser, in order to access the
contents of the pg_xlog directory. The minimum version to use this action is
Postgres 8.1. The --warning and --critical options are simply the number of
files in the pg_xlog directory. What number to set this to will vary, but a general
guideline is to put a number slightly higher than what is normally there, to catch
problems early.
Normally, WAL files are closed and then re-used, but a long-running open transaction, or a faulty archive_command script, may cause Postgres to create too many files. Ultimately, this will cause the disk they are on to run out of space, at which point Postgres will shut down.
Example 1: Check that the number of WAL files is 20 or less on host "pluto"
check_postgres_txn_wraparound --host=pluto --critical=20
For MRTG output, reports the number of WAL files on line 1.
(symlink: check_postgres_version) Checks that the required version of Postgres is running. The
--warning and --critical options (only one is required) must be of
the format X.Y or X.Y.Z where X is the major version number,
Y is the minor version number, and Z is the revision.
Example 1: Give a warning if the database on port 5678 is not version 8.4.10:
check_postgres_version --port=5678 -w=8.4.10
Example 2: Give a warning if any databases on hosts valley,grain, or sunshine is not 8.3:
check_postgres_version -H valley,grain,sunshine --critical=8.3
For MRTG output, reports a 1 or a 0 indicating success or failure on the first line. The
fourth line indicates the current version. The version must be provided via the --mrtg option.
The options --include and --exclude can be combined to limit which things are checked, depending on the action. The name of the database can be filtered when using the following actions: backends, database_size, locks, query_time, txn_idle, and txn_time. The name of a relation can be filtered when using the following actions: bloat, index_size, table_size, relation_size, last_vacuum, last_autovacuum, last_analyze, and last_autoanalyze. The name of a setting can be filtered when using the settings_checksum action. The name of a file system can be filtered when using the disk_space action.
If only an include option is given, then ONLY those entries that match will be checked. However, if given both exclude and include, the exclusion is done first, and the inclusion after, to reinstate things that may have been excluded. Both --include and --exclude can be given multiple times, and/or as comma-separated lists. A leading tilde will match the following word as a regular expression.
To match a schema, end the search term with a single period. Leading tildes can be used for schemas as well.
Examples:
Only checks items named pg_class:
--include=pg_class
Only checks items containing the letters 'pg_':
--include=~pg_
Only check items beginning with 'pg_':
--include=~^pg_
Exclude the item named 'test':
--exclude=test
Exclude all items containing the letters 'test:
--exclude=~test
Exclude all items in the schema 'pg_catalog':
--exclude='pg_catalog.'
Exclude all items containing the letters 'ace', but allow the item 'faceoff':
--exclude=~ace --include=faceoff
Exclude all items which start with the letters 'pg_', which contain the letters 'slon', or which are named 'sql_settings' or 'green'. Specifically check items with the letters 'prod' in their names, and always check the item named 'pg_relname':
--exclude=~^pg_,~slon,sql_settings --exclude=green --include=~prod,pg_relname
The options --includeuser and --excludeuser can be used on some actions to only examine database objects owned by (or not owned by) one or more users. An --includeuser option always trumps an --excludeuser option. You can give each option more than once for multiple users, or you can give a comma-separated list. The actions that currently use these options are:
Examples:
Only check items owned by the user named greg:
--includeuser=greg
Only check items owned by either watson or crick:
--includeuser=watson,crick
Only check items owned by crick,franklin, watson, or wilkins:
--includeuser=watson --includeuser=franklin --includeuser=crick,wilkins
Check all items except for those belonging to the user scott:
--excludeuser=scott
To help in setting things up, this program can be run in a "test mode" by specifying the --test option. This will perform some basic tests to make sure that the databases can be contacted, and that certain per-action prerequisites are met, such as whether the user is a superuser, if the version of Postgres is new enough, and if stats_row_level is enabled.
Since this program uses the psql program, make sure it is accessible to the user running the script. If run as a cronjob, this often means modifying the PATH environment variable.
If you are using Nagios in embedded Perl mode, use the --action argument
instead of symlinks, so that the plugin only gets compiled one time.
Access to a working version of psql, and the following very standard Perl modules:
$opt{showtime} is set to true, which is the default)The settings_checksum action requires the Digest::MD5 module.
Some actions require access to external programs. If psql is not explicitly
specified, the command which is used to find it. The program /bin/df
is needed by the disk_space action.
Development happens using the git system. You can clone the latest version by doing:
git clone http://bucardo.org/check_postgres.git
Two mailing lists are available. For discussions about the program, bug reports, feature requests, and commit notices, send email to check_postgres@bucardo.org
https://mail.endcrypt.com/mailman/listinfo/check_postgres
A low-volume list for announcement of new versions and important notices is the 'check_postgres-announce' list:
https://mail.endcrypt.com/mailman/listinfo/check_postgres-announce
Items not specifically attributed are by Greg Sabino Mullane.
Do explicit cleanups of the temp directory, per problems reported by sb@nnx.com.
Account for cases where some rounding queries give -0 instead of 0. Thanks to Glyn Astill for helping to track this down.
Always quote identifiers in check_replicate_row action.
Give a better error if one of the databases cannot be reached.
Add the "sequence" action, thanks to Gavin M. Roy for the idea. Fix minor problem with autovac_freeze action when using MRTG output. Allow output argument to be case-insensitive. Documentation fixes.
Fix some minor typos
Expand range of allowed names for --repinfo argument (Glyn Astill) Documentation tweaks.
Fixes for minor output and scoping problems.
Add MRTG output to fsm_pages and fsm_relations. Force error messages to one-line for proper Nagios output. Check for invalid prereqs on failed command. From conversations with Euler Taveira de Oliveira. Tweak the fsm_pages formula a little.
Add fsm_pages and fsm_relations actions. (Robert Treat)
Fix for race condition in txn_time action. Add --debugoutput option.
Allow alternate arguments "dbhost" for "host" and "dbport" for "port". Output a zero as default value for second line of MRTG output.
Fix sorting error in the "disk_space" action for non-Nagios output. Allow --simple as a shortcut for --output=simple.
Don't check databases with datallowconn false for the "autovac_freeze" action.
Add the "autovac_freeze" action, thanks to Robert Treat for the idea and design. Put an ORDER BY on the "txn_wraparound" action.
Optimizations to speed up the "bloat" action quite a bit. Fix "version" action to not always output in mrtg mode.
Add support for MRTG and "simple" output options. Many small improvements to nearly all actions.
Fix an error in the bloat SQL in 1.9.0 Allow percentage arguments to be over 99% Allow percentages in the bloat --warning and --critical (thanks to Robert Treat for the idea)
Don't include information_schema in certain checks. (Jeff Frost) Allow --include and --exclude to use schemas by using a trailing period.
Output schema name before table name where appropriate. Thanks to Jeff Frost.
Better detection of problems in --replicate_row.
Fix 'backends' action: there may be no rows in pg_stat_activity, so run a second query if needed to find the max_connections setting. Thanks to Jeff Frost for the bug report.
Changes to allow working under Nagios' embedded Perl mode. (Ioannis Tambouras)
Allow 'bloat' action to work on Postgres version 8.0. Allow for different commands to be run for each action depending on the server version. Give better warnings when running actions not available on older Postgres servers.
Add the --reverse option to the custom_query action.
Fix 'query_time' action: account for race condition in which zero rows appear in pg_stat_activity. Thanks to Dustin Black for the bug report.
Add --replicate_row action
Add --symlinks option as a shortcut to --action=rebuild_symlinks
Add the custom_query action.
Fix problem with too eager creation of custom pgpass file.
Add example Nagios configuration settings (Brian A. Seklecki)
Add the --includeuser and --excludeuser options. Documentation cleanup.
Add in the 'output' concept for future support of non-Nagios programs.
Fix bug preventing --dbpass argument from working (Robert Treat).
Minor documentation fixes.
Have 'wal_files' action use pg_ls_dir (idea by Robert Treat). For last_vacuum and last_analyze, respect autovacuum effects, add separate autovacuum checks (ideas by Robert Treat).
Have txn_idle use query_start, not xact_start.
Add in txn_idle and txn_time actions.
Add the 'wal_files' action, which counts the number of WAL files in your pg_xlog directory. Fix some typos in the docs. Explicitly allow -v as an argument. Allow for a null syslog_facility in the 'logfile' action.
Fix error preventing --action=rebuild_symlinks from working.
Switch vacuum and analyze date output to use 'DD', not 'D'. (Glyn Astill)
Fixes, enhancements, and performance tracking. Add performance data tracking via --showperf and --perflimit Lots of refactoring and cleanup of how actions handle arguments. Do basic checks to figure out syslog file for 'logfile' action. Allow for exact matching of beta versions with 'version' action. Redo the default arguments to only populate when neither 'warning' nor 'critical' is provided. Allow just warning OR critical to be given for the 'timesync' action. Remove 'redirect_stderr' requirement from 'logfile' due to 8.3 changes. Actions 'last_vacuum' and 'last_analyze' are 8.2 only (Robert Treat)
First public release, December 2007
The index bloat size optimization is rough.
Some actions may not work on older versions of Postgres (before 8.0).
Please report any problems to greg@endpoint.com.
Greg Sabino Mullane <greg@endpoint.com>
Some example Nagios configuration settings using this script:
define command {
command_name check_postgres_size
command_line $USER2$/check_postgres.pl -H $HOSTADDRESS$ -u pgsql -db postgres --action database_size -w $ARG1$ -c $ARG2$
}
define command {
command_name check_postgres_locks
command_line $USER2$/check_postgres.pl -H $HOSTADDRESS$ -u pgsql -db postgres --action locks -w $ARG1$ -c $ARG2$
}
define service {
use generic-other
host_name dbhost.gtld
service_description dbhost PostgreSQL Service Database Usage Size
check_command check_postgres_size!256000000!512000000
}
define service {
use generic-other
host_name dbhost.gtld
service_description dbhost PostgreSQL Service Database Locks
check_command check_postgres_locks!2!3
}
Copyright (c) 2007-2008 Greg Sabino Mullane <greg@endpoint.com>.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE AUTHOR "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.