Performance Schema in

MySQL
Debugging performance issue and getting deeper
insight into server’s behavior
Danil Zburivsky
April, 2013

Thursday, April 25, 13

 About Pythian
• Recognized Leader:
– Global industry-leader in remote database administration services and consulting
for Oracle, Oracle Applications, MySQL and Microsoft SQL Server
– Work with over 250 multinational companies such as Forbes.com, Fox Sports,
Nordion and Western Union to help manage their complex IT deployments
• Expertise:
– Pythian’s data experts are the elite in their field. We have the highest concentration
of Oracle ACEs on staff—9 including 2 ACE Directors—and 2 Microsoft MVPs.
– Pythian holds 7 Specializations under Oracle Platinum Partner program, including
Oracle Exadata, Oracle GoldenGate & Oracle RAC
• Global Reach & Scalability:
– Around the clock global remote support for DBA and consulting, systems
administration, special projects or emergency response

7 © 2013 Pythian

Thursday, April 25, 13

 About Me
– Pythian Consulting Group
– Managing hundreds MySQL
servers for dozens of
customers
– MySQL, Hadoop, NoSQL
– http://www.pythian.com/
news/author/zburivsky/
– @zburivsky

6 © 2013 Pythian

Thursday, April 25, 13

 Debugging MySQL Performance

Thursday, April 25, 13

 MySQL is a great database, but
instrumentation sucks

Slow query log SHOW ENGINE

INNODB STATUS

OS STATS
SHOW
PROCESSLIST

Thursday, April 25, 13

 I think there is a problem with
the database

• Identify timeframe (use trending tools)

• Collect data during incident
• Collect more data
• Collect as much data as you can!
• Try to make sense of it
• Tune settings, SQL, hardware. Did it help?
• I think there is a problem with database...

Thursday, April 25, 13

 Tools that may help you

• pt-stalk (http://www.percona.com/doc/percona-
toolkit/2.1/pt-stalk.html)

• innotop (http://code.google.com/p/innotop/)

• Or you write your own stuff

Thursday, April 25, 13

 What do “Big Boys” have?

• Oracle: SQL-traces,
kernel events timings,
tons of books on
performance tuning

• SQL Server: SQL

profiler, Data
Management Views,
Extended Events,tons
of books on
performance tuning

Thursday, April 25, 13

 Performance Schema in MySQL 5.5

Thursday, April 25, 13

 The Idea

• Executing a query is broken down to hundreds of

smaller tasks

• There are background tasks as well

• We want to instrument it all to know where server

is spending time

10

Thursday, April 25, 13

 Implementation

• Instrumentation: measuring when event begins

and ends

• Implemented in MySQL code on server end

storage engine level

• Can be enabled/disabled or customized

11

Thursday, April 25, 13

 Schema and a Storage Engine
+----------------------------------------------+
| Tables_in_performance_schema |
+----------------------------------------------+
| cond_instances |
| events_waits_current |
| events_waits_history |
| events_waits_history_long |
| events_waits_summary_by_instance |
| events_waits_summary_by_thread_by_event_name |
| events_waits_summary_global_by_event_name |
| file_instances |
| file_summary_by_event_name |
| file_summary_by_instance |
| mutex_instances |
| performance_timers |
| rwlock_instances |
| setup_consumers |
| setup_instruments |
| setup_timers |
| threads |
+----------------------------------------------+
17 rows in set (0.00 sec)

12

Thursday, April 25, 13

 Schema and a Storage Engine
+----------------------------------------------+
| Tables_in_performance_schema |
+----------------------------------------------+
| cond_instances |
| events_waits_current |
| events_waits_history |
| events_waits_history_long |
| events_waits_summary_by_instance |
| events_waits_summary_by_thread_by_event_name |
| events_waits_summary_global_by_event_name |
| file_instances |
| file_summary_by_event_name |
| file_summary_by_instance |
| mutex_instances |
| performance_timers |
| rwlock_instances |
| setup_consumers |
| setup_instruments |
| setup_timers |
| threads |
+----------------------------------------------+
17 rows in set (0.00 sec)

12

Thursday, April 25, 13

 Schema and a Storage Engine
+----------------------------------------------+
| Tables_in_performance_schema |
+----------------------------------------------+
| cond_instances |
| events_waits_current |
| events_waits_history |
| events_waits_history_long |
| events_waits_summary_by_instance |
| events_waits_summary_by_thread_by_event_name |
| events_waits_summary_global_by_event_name |
| file_instances |
| file_summary_by_event_name |
| file_summary_by_instance |
| mutex_instances |
| performance_timers |
| rwlock_instances |
| setup_consumers |
| setup_instruments |
| setup_timers |
| threads |
+----------------------------------------------+
17 rows in set (0.00 sec)

12

Thursday, April 25, 13

 Schema and a Storage Engine
+----------------------------------------------+
| Tables_in_performance_schema |
+----------------------------------------------+
| cond_instances |
| events_waits_current |
| events_waits_history |
| events_waits_history_long |
| events_waits_summary_by_instance |
| events_waits_summary_by_thread_by_event_name |
| events_waits_summary_global_by_event_name |
| file_instances |
| file_summary_by_event_name |
| file_summary_by_instance |
| mutex_instances |
| performance_timers |
| rwlock_instances |
| setup_consumers |
| setup_instruments |
| setup_timers |
| threads |
+----------------------------------------------+
17 rows in set (0.00 sec)

12

Thursday, April 25, 13

 Schema and a Storage Engine
+----------------------------------------------+
| Tables_in_performance_schema |
+----------------------------------------------+
| cond_instances |
| events_waits_current |
| events_waits_history |
| events_waits_history_long |
| events_waits_summary_by_instance |
| events_waits_summary_by_thread_by_event_name |
| events_waits_summary_global_by_event_name |
| file_instances |
| file_summary_by_event_name |
| file_summary_by_instance |
| mutex_instances |
| performance_timers |
| rwlock_instances |
| setup_consumers |
| setup_instruments |
| setup_timers |
| threads |
+----------------------------------------------+
17 rows in set (0.00 sec)

12

Thursday, April 25, 13

 Schema and a Storage Engine
+----------------------------------------------+
| Tables_in_performance_schema |
+----------------------------------------------+
| cond_instances |
| events_waits_current |
| events_waits_history |
| events_waits_history_long |
| events_waits_summary_by_instance |
| events_waits_summary_by_thread_by_event_name |
| events_waits_summary_global_by_event_name |
| file_instances |
| file_summary_by_event_name |
| file_summary_by_instance |
| mutex_instances |
| performance_timers |
| rwlock_instances |
| setup_consumers |
| setup_instruments |
| setup_timers |
| threads |
+----------------------------------------------+
17 rows in set (0.00 sec)

12

Thursday, April 25, 13

 Instruments
+------------------------------------------------------------+
| NAME |
+------------------------------------------------------------+
| wait/synch/mutex/sql/PAGE::lock |
| wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_sync |
| wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_active |
| wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_pool |
..... |
| wait/synch/rwlock/sql/LOCK_grant |
| wait/synch/rwlock/sql/LOGGER::LOCK_logger |
......
| wait/synch/cond/sql/COND_thread_count |
| wait/synch/cond/sql/COND_thread_cache |
..... |
| wait/io/file/sql/binlog |
| wait/io/file/sql/binlog_index |
+------------------------------------------------------------+

13

Thursday, April 25, 13

 Consumers
SELECT * FROM setup_consumers;
+----------------------------------------------+---------+
| NAME | ENABLED |
+----------------------------------------------+---------+
| events_waits_current | YES |
| events_waits_history | YES |
| events_waits_history_long | YES |
| events_waits_summary_by_thread_by_event_name | YES |
| events_waits_summary_by_event_name | YES |
| events_waits_summary_by_instance | YES |
| file_summary_by_event_name | YES |
| file_summary_by_instance | YES |
+----------------------------------------------+---------+
8 rows in set (0.00 sec)

14

Thursday, April 25, 13

 What are top wait events in my
server?

SELECT COUNT_STAR, SUM_TIMER_WAIT, AVG_TIMER_WAIT

FROM events_waits_summary_global_by_event_name
ORDER BY SUM_TIMER_WAIT DESC LIMIT 10;
+---------------------------------------------+------------+-------------------+----------------+
| EVENT_NAME | COUNT_STAR | SUM_TIMER_WAIT | AVG_TIMER_WAIT |
+---------------------------------------------+-----------------------+---------------+---------|
| wait/io/file/innodb/innodb_data_file | 3347631 | 13547848609831118 | 4046995803 |
| wait/synch/rwlock/innodb/index_tree_rw_lock | 69535441 | 2000129499580014 | 28764173 |
| wait/synch/mutex/innodb/buf_pool_mutex | 800909600 | 433331246983734 | 541048 |
| wait/synch/mutex/innodb/log_sys_mutex | 181356697 | 420610626124392 | 2319245 |
| wait/synch/rwlock/innodb/index_tree_rw_lock | 3060704 | 232839801243104 | 76073936 |
| wait/io/file/innodb/innodb_data_file | 84259 | 163336572508164 | 1938505946 |
| wait/synch/rwlock/innodb/btr_search_latch | 192529196 | 161285917556962 | 837721 |
| wait/synch/mutex/sql/LOCK_open | 285022231 | 154822982324406 | 543196 |
| wait/synch/mutex/innodb/kernel_mutex | 403820715 | 130291049763820 | 322645 |
| wait/synch/mutex/mysys/THR_LOCK::mutex | 285017708 | 121061567338910 | 424751 |
+---------------------------------------------+------------+-------------------+----------------+
10 rows in set (0.03 sec)

15

Thursday, April 25, 13

 innodb_buffer_pool_instance=10

SELECT * FROM events_waits_summary_global_by_event_name

ORDER BY SUM_TIMER_WAIT DESC LIMIT 10;
+-----------------------------------------------+------------+-------------------+----------------+
| EVENT_NAME | COUNT_STAR | SUM_TIMER_WAIT | AVG_TIMER_WAIT |
+-----------------------------------------------+------------+-------------------+----------------+
| wait/io/file/innodb/innodb_data_file | 3600080 | 14130028463169872 | 3924920685 |
| wait/synch/rwlock/innodb/index_tree_rw_lock | 69313641 | 2045091134703870 | 29504886 |
| wait/synch/mutex/innodb/trx_doublewrite_mutex | 6432834 | 592011487365760 | 92029654 |
| wait/synch/mutex/innodb/log_sys_mutex | 183016201 | 459928639055248 | 2513048 |
| wait/synch/rwlock/innodb/index_tree_rw_lock | 3239574 | 277212671866944 | 85570717 |
| wait/io/file/innodb/innodb_data_file | 400484 | 219787308922610 | 548804219 |
| wait/synch/rwlock/innodb/btr_search_latch | 190363676 | 183251774563990 | 962640 |
| wait/synch/mutex/sql/LOCK_open | 285022522 | 144911560909258 | 508421 |
| wait/synch/mutex/mysys/THR_LOCK::mutex | 285018012 | 130317830152708 | 457226 |
| wait/synch/mutex/innodb/kernel_mutex | 401734769 | 126815854421630 | 315670 |
+-----------------------------------------------+------------+-------------------+----------------+
10 rows in set (0.02 sec)

buf_pool_mutex is no longer in top 10 wait

events

16

Thursday, April 25, 13

 Which files are being accessed the
most?

mysql> SELECT SUM_NUMBER_OF_BYTES_READ, SUM_NUMBER_OF_BYTES_WRITE

FROM file_summary_by_instance
ORDER BY SUM_NUMBER_OF_BYTES_READ+SUM_NUMBER_OF_BYTES_WRITE DESC LIMIT 10;
+------------------------------+--------------------------+---------------------------+
| FILE_NAME |SUM_NUMBER_OF_BYTES_READ | SUM_NUMBER_OF_BYTES_WRITE |
+------------------------------+--------------------------+---------------------------+
| /var/lib/mysql/ibdata1 | 166920192 | 363649155072 |
| /var/lib/mysql/WIR.ibd | 180310720512 | 114136367104 |
| /var/lib/mysql/EMIWPR.ibd | 51803815936 | 71776468992 |
| /var/lib/mysql/EWAPR.ibd | 38596755456 | 54104932352 |
| /var/lib/mysql/OFPR.ibd | 24171593728 | 33964572672 |
| /var/lib/mysql/OFIR.ibd | 38002688000 | 14265303040 |
| /var/lib/mysql/EQPO.ibd | 12718882816 | 21407596544 |
| /var/lib/mysql/FIR.ibd | 19646103552 | 10337140736 |
| /var/lib/mysql/EIR.ibd | 14708899840 | 12814794752 |
| /var/lib/mysql/FPR.ibd | 6360350720 | 9764585472 |
+------------------------------+--------------------------+---------------------------+
10 rows in set (0.01 sec)

17

Thursday, April 25, 13

 Threads table
(performance_schema) > SELECT * FROM threads;
+-----------+----------------+----------------------------------------+
| THREAD_ID | PROCESSLIST_ID | NAME |
+-----------+----------------+----------------------------------------+
| 0 | 0 | thread/sql/main |
| 12 | 0 | thread/innodb/srv_lock_timeout_thread |
| 14 | 0 | thread/innodb/srv_monitor_thread |
| 16 | 0 | thread/innodb/srv_purge_thread |
| 13 | 0 | thread/innodb/srv_error_monitor_thread |
| 2 | 0 | thread/innodb/io_handler_thread |
| 1 | 0 | thread/innodb/io_handler_thread |
| 9 | 0 | thread/innodb/io_handler_thread |
| 25 | 8 | thread/sql/one_connection |
| 3 | 0 | thread/innodb/io_handler_thread |
| 15 | 0 | thread/innodb/srv_master_thread |
| 10 | 0 | thread/innodb/io_handler_thread |
| 5 | 0 | thread/innodb/io_handler_thread |
| 8 | 0 | thread/innodb/io_handler_thread |
| 7 | 0 | thread/innodb/io_handler_thread |
| 17 | 0 | thread/sql/signal_handler |
| 4 | 0 | thread/innodb/io_handler_thread |
| 6 | 0 | thread/innodb/io_handler_thread |
+-----------+----------------+----------------------------------------+
18 rows in set (0.00 sec)

18

Thursday, April 25, 13

 Average wait per thread (I)

SELECT e.THREAD_ID, e.EVENT_NAME, MAX(e.AVG_TIMER_WAIT)

FROM events_waits_summary_by_thread_by_event_name e
LEFT JOIN threads t ON t.THREAD_ID = e.THREAD_ID
WHERE event_name LIKE 'wait/synch/mutex/innodb/%'
AND t.NAME='thread/sql/one_connection'
AND AVG_TIMER_WAIT > 0
GROUP BY t.thread_id, e.EVENT_NAME
ORDER BY t.thread_id, MAX(e.AVG_TIMER_WAIT)

19

Thursday, April 25, 13

 Average wait per thread (II)
+-----------+-------------------------------------------------------+-------------------------+
| THREAD_ID | EVENT_NAME | MAX(AVG_TIMER_WAIT) |
+-----------+-------------------------------------------------------+-------------------------+
| 27 | wait/synch/mutex/innodb/mutex_list_mutex | 49463 |
| 27 | wait/synch/mutex/innodb/purge_sys_bh_mutex | 83247 |
.....
.....
| 27 | wait/synch/mutex/innodb/ibuf_bitmap_mutex | 115681 |
| 27 | wait/synch/mutex/innodb/ibuf_pessimistic_insert_mutex | 168537 |
| 27 | wait/synch/mutex/innodb/ibuf_mutex | 282400 |
| 27 | wait/synch/mutex/innodb/log_sys_mutex | 1690694 |
----------------------------------------------------------------------------------------------
| 28 | wait/synch/mutex/innodb/mutex_list_mutex | 49504 |
| 28 | wait/synch/mutex/innodb/flush_list_mutex | 52375 |
| 28 | wait/synch/mutex/innodb/log_flush_order_mutex | 62040 |
.....
.....
| 28 | wait/synch/mutex/innodb/autoinc_mutex | 63012 |
| 28 | wait/synch/mutex/innodb/dict_sys_mutex | 181203 |
| 28 | wait/synch/mutex/innodb/ibuf_mutex | 222376 |
| 28 | wait/synch/mutex/innodb/innobase_share_mutex | 239514 |
| 28 | wait/synch/mutex/innodb/ibuf_pessimistic_insert_mutex | 266684 |
| 28 | wait/synch/mutex/innodb/log_sys_mutex | 1718503 |

20

Thursday, April 25, 13

 Average wait per thread (III)
innodb_change_buffering = none
+-----------+--------------------------------------------------+-------------------------+
| THREAD_ID | EVENT_NAME | MAX(ebt.AVG_TIMER_WAIT) |
+-----------+--------------------------------------------------+-------------------------+
| 27 | wait/synch/mutex/innodb/ibuf_mutex | 55872 |
| 27 | wait/synch/mutex/innodb/flush_list_mutex | 59495 |
| 27 | wait/synch/mutex/innodb/mutex_list_mutex | 59499 |
| 27 | wait/synch/mutex/innodb/autoinc_mutex | 64136 |
.....
.....
| 27 | wait/synch/mutex/innodb/ibuf_bitmap_mutex | 88242 |
| 27 | wait/synch/mutex/innodb/buf_pool_mutex | 101598 |
| 27 | wait/synch/mutex/innodb/kernel_mutex | 148332 |
| 27 | wait/synch/mutex/innodb/log_sys_mutex | 4671112 |
-----------------------------------------------------------------------------------------
| 28 | wait/synch/mutex/innodb/ibuf_mutex | 42886 |
| 28 | wait/synch/mutex/innodb/mutex_list_mutex | 56646 |
| 28 | wait/synch/mutex/innodb/flush_list_mutex | 58105 |
| 28 | wait/synch/mutex/innodb/autoinc_mutex | 66873 |
....
....
| 28 | wait/synch/mutex/innodb/purge_sys_bh_mutex | 83545 |
| 28 | wait/synch/mutex/innodb/ibuf_bitmap_mutex | 87697 |
| 28 | wait/synch/mutex/innodb/buf_pool_mutex | 101365 |
| 28 | wait/synch/mutex/innodb/kernel_mutex | 149135 |
| 28 | wait/synch/mutex/innodb/log_sys_mutex | 4829134 |

21

Thursday, April 25, 13

 Picoseconds, shmicoseconds!
ps_helper to rescue!

• http://www.markleith.co.uk/ps_helper/
• Great examples of using
performance_schema

• Useful tools for converting time, bytes,

paths into human readable format

22

Thursday, April 25, 13

 Example

(ps_helper) > SELECT event_name, total_read, avg_read, total_written, avg_written,avg_latency

FROM top_global_io_consumers_by_bytes_usage;
+-------------------------+------------+-----------+---------------+-------------+-------------+
| event_name | total_read | avg_read | total_written | avg_written | avg_latency |
+-------------------------+------------+-----------+---------------+-------------+-------------+
| innodb/innodb_data_file | 10.76 GiB | 16.00 KiB | 16.05 GiB | 30.90 KiB | 10.96 ms |
| innodb/innodb_log_file | 68.00 KiB | 11.33 KiB | 1.03 GiB | 369.66 KiB | 2.33 ms |
| myisam/dfile | 197.79 KiB | 8.60 KiB | 187.04 KiB | 5.84 KiB | 9.89 µs |
| sql/FRM | 142.28 KiB | 491 bytes | 4.50 KiB | 242 bytes | 3.66 µs |
| sql/ERRMSG | 43.68 KiB | 14.56 KiB | 0 bytes | 0 bytes | 15.42 µs |
| myisam/kfile | 13.22 KiB | 226 bytes | 5.96 KiB | 24 bytes | 8.26 µs |
| mysys/charset | 17.83 KiB | 17.83 KiB | 0 bytes | 0 bytes | 10.68 µs |
| sql/partition | 1.06 KiB | 32 bytes | 0 bytes | 0 bytes | 4.38 µs |
| sql/pid | 0 bytes | 0 bytes | 6 bytes | 6 bytes | 34.78 µs |
| sql/global_ddl_log | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 3.69 µs |
| archive/data | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 4.35 µs |
| sql/dbopt | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 4.55 µs |
| sql/casetest | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 28.08 µs |
+-------------------------+------------+-----------+---------------+-------------+-------------+
13 rows in set (0.00 sec)

23

Thursday, April 25, 13

 No free lunch: PFS overhead

• Instrumentation doesn’t come for free

• In CPU-bound workloads overhead can
be ~15-20%

• In IO-bound — 5%-8%
• Significant improvements in 5.6

24

Thursday, April 25, 13

 Performance Schema in MySQL 5.6

Thursday, April 25, 13

 What’s new in 5.6?

• performance_schema enabled by default!

• Less overhead: 5%-10% for CPU-bound
workload

• Statements, Stages, Actors and Objects

26

Thursday, April 25, 13

 Actors: filter event’s by user

(performance_schema) > SELECT * FROM setup_actors;

+------+------+------+
| HOST | USER | ROLE |
+------+------+------+
| % | % | % |
+------+------+------+
1 row in set (0.00 sec)

27

Thursday, April 25, 13

 Objects: filter events by database/table
name

(performance_schema) > SELECT * FROM setup_objects;

+-------------+--------------------+-------------+---------+-------+
| OBJECT_TYPE | OBJECT_SCHEMA | OBJECT_NAME | ENABLED | TIMED |
+-------------+--------------------+-------------+---------+-------+
| TABLE | mysql | % | NO | NO |
| TABLE | performance_schema | % | NO | NO |
| TABLE | information_schema | % | NO | NO |
| TABLE | % | % | YES | YES |
+-------------+--------------------+-------------+---------+-------+
4 rows in set (0.00 sec)

28

Thursday, April 25, 13

 Statements
SELECT * FROM events_statements_summary_by_digest\G

*************************** 4. row ***************************

DIGEST: f60b0866d7cbd78d39caf5fb52f43bfb
DIGEST_TEXT: SELECT * FROM `table_io_waits_summary_by_index_usage` ORDER BY
`SUM_TIMER_WAIT` DESC LIMIT ?
COUNT_STAR: 1
SUM_TIMER_WAIT: 511539000
MIN_TIMER_WAIT: 511539000
AVG_TIMER_WAIT: 511539000
MAX_TIMER_WAIT: 511539000
SUM_LOCK_TIME: 72000000
SUM_ERRORS: 0
SUM_WARNINGS: 0
SUM_ROWS_AFFECTED: 0
SUM_ROWS_SENT: 5
SUM_ROWS_EXAMINED: 91
SUM_CREATED_TMP_DISK_TABLES: 0
SUM_CREATED_TMP_TABLES: 0
SUM_SELECT_FULL_JOIN: 0
SUM_SELECT_FULL_RANGE_JOIN: 0
SUM_SELECT_RANGE: 0
SUM_SELECT_RANGE_CHECK: 0
SUM_SELECT_SCAN: 1
SUM_SORT_MERGE_PASSES: 0
SUM_SORT_RANGE: 0
SUM_SORT_ROWS: 5
SUM_SORT_SCAN: 1
SUM_NO_INDEX_USED: 1
SUM_NO_GOOD_INDEX_USED: 0
FIRST_SEEN: 2012-10-16 21:50:13
LAST_SEEN: 2012-10-16 21:50:13

29

Thursday, April 25, 13

 Summary. Pros.

• Provides insight into internal server

metric

• Flexible
• Relational model allows to build your own
views on top of this data

30

Thursday, April 25, 13

 Summary. Cons.

• Steep learning curve: not very well

documented

• Overhead for CPU-bound loads can be

significant

31

Thursday, April 25, 13

 Thank you – Q&A
To contact us
[email protected]

1-877-PYTHIAN

To follow us
http://www.pythian.com/blog

http://www.facebook.com/pages/The-Pythian-Group/163902527671

@pythian

http://www.linkedin.com/company/pythian

8 © 2013 Pythian

Thursday, April 25, 13