Webinar

Zabbix performance tuning

all our microphones are muted
ask your questions in Q&A, not in the Chat
use Chat for discussion, networking or applause
1
Performance tunning
Zabbix performance tuning

Zabbix data flow Notifications

Visualization

Database ZABBIX Server

History Analysis Data collection

Zabbix performance tuning

How to measure performance

Number of values processed per second (NVPS)
A rough estimate of NVPS is visible in Zabbix
Zabbix performance tuning

How to measure performance

Why the number of devices is not an indicator?
Zabbix performance tuning

How to measure performance

Update frequency greatly affects NVPS.

The calculation takes into account data from the monitored devices.
Data types “Zabbix trapper” or “SNMP trap” are not taken into account.
Zabbix performance tuning

Performance

Hardware: 10 Core CPU, 32GB, RAID10 BBWC

Budget: ~ 4K EUR

Zabbix is able to deliver 2 million of values per minute or around 30.000 of values per second
In real life performance would be worse. Why?!
Zabbix performance tuning

What affects performance?

Type of items, value types, SNMPv3, number of triggers and complexity of triggers.
Housekeeper settings and thus size of the database.
Number of users working with the WEB interface.
Zabbix performance tuning

What affects performance?

60 items per host, update Number of hosts Performance - NVPS
frequency once per minute

100 100

1 000 1 000

10 000 10 000

300 items per host, update

frequency once per minute Number of hosts Performance - NVPS

100 500

1 000 5 000

10 000 50 000
Choose update frequency and
duration of storage carefully
Zabbix performance tuning

Performance
History analysis
Slow Fast
affects performance
Database size Large Fits into memory
of Zabbix. But not so
much!
Low-level Update frequency Update frequency
detection 30s, 15m, 30m 1h, 1d, 7d

Errors in settings nodata(5m) and mult. event nodata(5m),

generation, min(3600)
min(#3600)
Trigger min(), max(), avg() last(), nodata()
expressions

Data collection Polling (SNMP, agent-less, Trapping (active agents)

passive agent)

Data types Text, string Numeric

Zabbix performance tuning

Performance
Different views on performance

“I just added 5 hosts and Zabbix died” :-(

“Zabbix is so slooooow, I have only 48 hosts” :-(

“Zabbix Milestone achieved - 1000 hosts and growing” :-)

“Our status update: 232623 hosts, 3878565 items, 591121 triggers,
19086 vps” :-)

What’s the difference?

Zabbix performance tuning

Performance
Common problems of initial setup

Default database settings

Tune database for the best performance (https://github.com/hermanekt/Zabbix_MYSQL_tunned_for_40k)
Not optimal configuration of Zabbix Server
Tune Zabbix Server configuration (Monitoring > Dashboard > Zabbix server health)
Housekeeper settings do not match hardware spec
(Use partitions in DB)
Use of default templates
Make your own smarter templates
Use of older releases
Always use the latest one!
Zabbix performance tuning

Performance
Visible symptoms of bad performance

Zabbix queue has too many delayed items Administration->Queue

Frequent gaps in graphs, no data for some of the items
False positives for triggers having nodata() function
Unresponsive WEB interface
No alerts or thousands of alerts
Zabbix performance tuning

Performance
Nice view of queue of items
Zabbix performance tuning

Performance
Nice view of queue of items during a problem state
Zabbix performance tuning

Performance

Identify Tune Improve

Step 1 Step 2 Step 3
2
Identify
Zabbix performance tuning

Identify Visualization

How to understand which one is the root cause of

Zabbix slowdown?

Step 1

Database ZABBIX Server

History Analysis
Zabbix performance tuning

Identify
Main utilities
top, ntop, iostat, vmstat, sar
Zabbix itself

DB Zabbix strace or log file with debugging mode enabled

ps aux | grep zabbix_server

# ps ax | grep sync
zabbix_server: history syncer #1 [synced 1845 items in 0.257111 sec, syncing history]
zabbix_server: history syncer #2 [synced 24 items in 0.060314 sec, idle 4 sec]
zabbix_server: history syncer #3 [synced 0 items in 0.000018 sec, idle 4 sec]
zabbix_server: history syncer #4 [synced 0 items in 0.000009 sec, syncing history]

Values change?
Zabbix performance tuning

Identify
Main utilities
top, ntop, iostat, vmstat, sar
Zabbix itself

DB Zabbix strace or log file with debugging mode enabled

ps aux | grep zabbix_server

# ps ax | grep sync
history syncer #1 [synced 1020 items in 285.198752 sec, syncing history]
history syncer #2 [synced 915 items in 285.177799 sec, syncing history]
history syncer #3 [synced 3401 items in 284.936376 sec, syncing history]
history syncer #4 [synced 1194 items in 285.280719 sec, syncing history]

During the problem?

Zabbix performance tuning

Identify
Get internal statistics
The actual VPS value
zabbix[wcache, values, all]
zabbix[queue,1m] amount of items with a delay of more than 1 minute
Zabbix server components
Alerter, Configuration syncer, DB watchdog, discoverer, escalator, history syncer, http poller,
housekeeper, icmp pinger, ipmi poller, poller, trapper, etc.
Zabbix server cache
history write cache, value cache, trend write cache, vmware cache, etc.
Ready templates:
Template App Zabbix Server
Template App Zabbix Proxy
Template App Zabbix Agent
Zabbix performance tuning

Identify
Zabbix performance tuning

Identify
Debug mode
There is a problem, but it is not clear what kind of problem?
Enable debugging mode for the process:
# zabbix_server -R log_level_increase=alerter
Search in the log for information about the problem (grep, etc.):
/var/log/zabbix/zabbix_server.log
Zabbix performance tuning

Identify
How to know that the performance of the DB is bad?
Zabbix server configuration file, zabbix_server.conf

LogSlowQueries=3000
Zabbix performance tuning

Identify
Main utilities
top, ntop, iostat, vmstat, sar
DB statistics, innotop
Zabbix performance tuning

Identify
Main utilities
top, ntop, iostat, vmstat, sar
DB statistics, innotop

# grep slow /var/log/zabbix/zabbix_server.log

slow query: 9.054528 sec, "insert into events (eventid, source, object, objectid,
clock...
slow query: 8.501505 sec, "update hosts set lastaccess=1421211815 where hostid...
slow query: 6.754405 sec, "insert into history (itemid,clock,ns,value) values...
slow query: 37.949541 sec, "select i.itemid, i.hostid, h.proxy_hostid, i.type,
i.data_type...
slow query: 70.877295 sec, "select distinct t.triggerid, t.description, t.expression,
t.error...
Zabbix performance tuning

Identify
Slow DB?
Zabbix performance tuning

Identify
Debug mode
Zabbix performance tuning

Identify
Debug mode

******************** Script profiler ********************

Total time: 0.960905
Load speed
Total SQL time: 0.749027 less than a
SQL count: 5636 (selects: 4065 | executes: 1571)
Peak memory usage: 180.5M
second
Memory limit: 2G
Zabbix performance tuning

Identify
Debug mode

******************** Script profiler ********************

Total time: 10.960905
Problem
Total SQL time: 0.749027 with web
SQL count: 5636 (selects: 4065 | executes: 1571)
Peak memory usage: 180.5M
server
Memory limit: 2G
Zabbix performance tuning

Identify
Debug mode

******************** Script profiler ********************

Total time: 10.960905
Problem
Total SQL time: 10.749027 with DB
SQL count: 5636 (selects: 4065 | executes: 1571)
Peak memory usage: 180.5M
Memory limit: 2G
3
Tune
Zabbix performance tuning

Tune
Tune number of processes (example)
Zabbix server configuration file, zabbix_server.conf:

StartPollers=80
StartPingers=10
StartPollersUnreachable=80
StartIPMIPollers=10
StartTrappers=20
StartDBSyncers=6
Zabbix performance tuning

Tune
How to know when it is time to tune Zabbix configuration?
Failures in graphs or 100% load
Zabbix performance tuning

Tune
InnoDB is better than MyISAM
Look at the data
mysqladmin status / variables (or innotop)
InnoDB
innodb_file_per_table = 1
innodb_buffer_pool_size=<large> (~75% of total RAM)
innodb_buffer_pool_instances = 8
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
innodb_log_file_size = 256M
Do not use
Query history
Zabbix performance tuning

Tune
Problem with Web server

******************** Script profiler ********************

Total time: 10.960905
Problem
Total SQL time: 0.749027 with web
SQL count: 5636 (selects: 4065 | executes: 1571)
Peak memory usage: 180.5M
server
Memory limit: 2G

Optimize configuration
Try nginx
4
Improve
Zabbix performance tuning

Improve
Table partitioning
It is a way to split large tables into smaller partitions.
Make sense for historical tables:
history_* and trends*
Benefits:
Easy to remove older data
Significantly better performance
Zabbix performance tuning

Improve
No table partitioning

Zabbix
Server History
& GUI
Zabbix performance tuning

Improve
How to know when it is time to apply partitioning?
Trigger “Zabbix housekeeper processes more than 75% busy” is in problem state for hours or days
The performance of housekeeper is dropping
Zabbix performance tuning

Improve
I still need better performance
Run Zabbix components on separate servers!

Zabbix server & Web-interface Database

8 core CPU 16 core CPU
8GB RAM 64GB RAM
Fast repository
Zabbix performance tuning

Improve
I still need better performance
Run Zabbix components on separate servers!

Zabbix server Web-interface Database

8 core CPU 2 core CPU 16 core CPU
4GB RAM 4GB RAM 64GB RAM
Fast repository
Zabbix performance tuning

Improve
I still need better performance
All data collection is done using a proxy

ZABBIX
ZABBIX
ZABBIX SERVER PROXY
ZABBIX
PROXY
PROXY
DB
Data DB
Zabbix performance tuning

Improve
Why to use proxy?
Zabbix Proxy "converts" passive checks into active
The load is distributed between the proxies
If one proxy is overloaded, network nodes can be moved to another proxy
Easy maintenance
Caching data when Zabbix server is not available
Zabbix performance tuning

Improve
Checklist
Zabbix internal checks are done
Otherwise, you don't know anything about Zabbix health!
Zabbix configuration is tuned
Database performance is tuned
Removing history is not used for history tables
Zabbix performance tuning

Improve
Additional reading

Performance Optimization Guide:

Mysql: https://www.percona.com/blog/2014/11/14/optimizing-mysql-zabbix/
PostgreSQL: https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server
PostgreSQL: https://pgtune.leopard.in.ua/

Partitioning tables in Zabbix:

MySQL: http://zabbix.org/wiki/Docs/howto/mysql_partitioning
PostgreSQL: https://www.zabbix.org/wiki/Docs/howto/zabbix2_postgresql_partitioning (OLD)
PostgreSQL: https://www.zabbix.com/documentation/current/manual/appendix/install/timescaledb

Zabbix internal checks

http://blog.zabbix.com/monitoring-how-busy-zabbix-processes-are
https://www.zabbix.com/documentation/current/manual/config/items/itemtypes/internal
5
Demonstration
Questions?
Why You Need Monitoring

CONTACT US:
Phone: +420 800 244 442

Web: https://www.initmax.cz

Email: [email protected]

LinkedIn: https://www.linkedin.com/company/initmax

Twitter: https://twitter.com/initmax1

Tomáš Heřmánek: +420 732 447 184