<Insert Picture Here>

MySql Performance Tuning Best Practices

Brian Miezejewski Practice Manager Oracle MySQL
Copyright 2010 Oracle The Worlds Most Popular Open Source Database

Agenda
Overview Hardware and OS Storage Engines Connections Sessions Query Cache Queries Schema Partitions What if I need more help?
Copyright 2010 Oracle The Worlds Most Popular Open Source Database The Worlds Most Popular Open Source Database

 Focus of this talk is the MySQL server, not:

Operating System, Disk performance, Network performance, etc.

Overview

Cover the main steps Show at least one example for each step Examples are things I run into most commonly in the field Include links to MySQL manual for additional information This will be technical! You cannot become a performance tuning wizard in 45 minutes - PT Class is 4 day class
http://www.mysql.com/training/courses/performance_tuning.html

MySQL Performance Forum

http://forums.mysql.com/list.php?24

Copyright 2010 Oracle

The Worlds Most Popular Open Source Database

What you will need to know first

The MySQL server is controlled by System Variables
mysql> SHOW VARIABLES [LIKE <str>]; linux1> mysqladmin -u <user> -p variables Set Via: my.cfg SET [GLOBAL] <variable>=<value> client, i.e mysql Can be local (session) or global http://dev.mysql.com/doc/refman/5.1/en/server-system-variables.html

You monitor how well your system variables are configured using Status Variables
mysql> SHOW STATUS [LIKE <str>]; linux1> mysqladmin -u <user> -p extended linux1> mysqladmin ... ex -i 15 -r | grep -v 0 http://dev.mysql.com/doc/refman/5.1/en/server-status-variables.html

Enabling the slow query log

http://dev.mysql.com/doc/refman/5.1/en/slow-query-log.html

Copyright 2010 Oracle

The Worlds Most Popular Open Source Database

MySQL Enterprise Monitor w/Query Analyzer

Single, consolidated view into entire MySQL environment Auto discovery of MySQL Servers, Replication Topologies Problem Query Detection, Analysis and Tuning New! Customizable rules-based monitoring and alerts Finds problems before they occur Reduces risk of downtime Makes it easier to scale-out without requiring more DBAs Virtual MySQL DBA Assistant

Copyright 2010 Oracle

The Worlds Most Popular Open Source Database

Never make a change in production first Have a good benchmark or reliable load Start with a good baseline Only change 1 thing at a time
identify a set of possible changes try each change separately try in combinations of 2, then 3, etc.

Rules of Tuning

Monitor the results

Query performance - query analyzer, slow query log, etc.
throughput single query time average query time

CPU - top, vmstat IO - iostat, top, vmstat, bonnie++ Network bandwidth

Document and save the results

Copyright 2010 Oracle The Worlds Most Popular Open Source Database

Were do I find a benchmark?

Make your own - Can use general query log output DBT2

- http://osdldbt.sourceforge.net/ - http://samurai-mysql.blogspot.com/2009/03/settingup-dbt-

mysqlslap MySQL 5.1 + SysBench

- http://dev.mysql.com/doc/refman/5.1/en/mysqlslap.html - http://sysbench.sourceforge.net/ - http://vegan.net/tony/supersmack/ - http://jeremy.zawodny.com/mysql/mybench/

supersmack mybench

Copyright 2010 Oracle

The Worlds Most Popular Open Source Database

Perfect Hardware?

4-8 Cores now, 8-16 cores for 5.4 and up x86_64 - 64 bit for more memory is important Linux or Solaris best, Windows and Unix also fine. RAID 10 for most, RAID 5 OK if very read intensive Hardware RAID battery backed up cache critical!
- More disks are always better!
- 4+ recommended, 8-16 can increase IO performance if needed

Memory should be at least 1/3 to 1/10 data size

- The more, the better

Sun X4170 for example

Copyright 2010 Oracle The Worlds Most Popular Open Source Database

Allocate swap space (at least 1/2 of RAM size) Set vm.swappiness = 0 and use O_DIRECT Set /sys/block/sdX/queue/scheduler = deadline or noop Filesystem Tuning

Linux Tuning (Yoshinori Matsunobu) Install at (Yoshinori Matsunobu)package) least sar, mpstat, iostat (sysstat Oprofile, gdb and SystemTap(stap) are recommended (Yoshinori Matsunobu)

ext3: tune2fs O dir_index -c l i 0 xfs: nobarrier Make sure write cache with battery is enabled http://en.oreilly.com/mysql2010/public/schedule/detail/13252

Copyright 2010 Oracle

The Worlds Most Popular Open Source Database

MySQL Supports Multiple Storage Engines

MyISAM - Original Storage Engine, great for web apps InnoDB - Robust transactional storage engine Memory Engine - Stores all data in Memory To see what tables are in what engines Selecting the storage engine to use is a tuning decision
mysql> alter table tab engine=myisam ;
mysql> SHOW TABLE STATUS ;

Copyright 2010 Oracle

The Worlds Most Popular Open Source Database

MyISAM
Fastest storage engine 3x or more when appropriate
Most web applications Perfect for web search databases 80/20 read/modify or higher pure inserts and deletes with partitions or merge engine no transactions reporting DB/ Data Warehouse

Most compact data of all non-compressed engines Table locking Not ACID compliant, non-transactional Supports concurrent inserts Full-Text and Geospatial support http://dev.mysql.com/doc/refman/5.1/en/myisam-storage-en

Copyright 2010 Oracle

The Worlds Most Popular Open Source Database

MyISAM Tuning
The primary tuning factor in MyISAM are its two caches:
key_buffer_cache - should be 25% of available memory system cache - leave 75% of available memory free

Available memory is:

All on a dedicated server, if the server has 8GB, use 2GB for the key_buffer_cache and leave the rest free for the system cache to use. Percent of the part of the server allocated for MySQL, i.e. if you have a server with 8GB, but are using 4GB for other applications then use 1GB for the key_buffer_cache and leave the remaining 3GB free for the system cache to use.

You can define multiple key buffers You can pre-load the key buffers For more details on configuring the MyISAM key cache see:
http://dev.mysql.com/doc/refman/5.1/en/myisam-key-cache.html

Copyright 2010 Oracle

The Worlds Most Popular Open Source Database

 mysql>show status like 'Key%' ; Key_blocks_not_flushed - Dirty key blocks not flushed to disk Key_blocks_unused - unused blocks in the cache Key_blocks_used - used Blocks in the cache % of cache free : Key_blocks_unused /(Key_blocks_unused + Key_blocks_used) Key_read_requests - key requests to the cache Key_reads - times a key read request went to disk Cache read hit % : Key_reads / Key_read_requests Key_write_requests - key write request to cache Key_writes - times a key write request went to disk Cache write hit % : Key_writes / Key_write_request cat /proc/meminfo to see the system cache in linux
MemFree + Cached = memory available for system cache
Copyright 2010 Oracle The Worlds Most Popular Open Source Database

Monitoring the MyISAM Key Buffer Cache

InnoDB
Transactional and fully ACID compliant Behavior most like traditional databases such as Oracle, DB2, SQL Server, etc. Data size is normally 2-3 X MyISAM MVCC = Non-blocking reads in most cases Fast, reliable recovery from crashes with zero committed data loss Always clustered on the primary key
Lookups by primary key, very fast Range scans on primary key also very fast Non-Primary key lookups use the primary key to find the record, this means 2 key lookups Important to keep primary key small

http://dev.mysql.com/doc/refman/5.1/en/innodb.html

Copyright 2010 Oracle

The Worlds Most Popular Open Source Database

InnoDB
Unlike MyISAM InnoDB uses a single cache for both index and data
Innodb_buffer_pool_size - should be 70-80% of available memory. It is not uncommon for this to be very large, i.e. 44GB on a system with 40GB of memory Make sure its not set so large as to cause swapping! mysql>show status like 'Innodb_buffer%' ; Innodb_flush_method = O_DIRECT

InnoDB can use direct IO on systems that support it, linux, FreeBSD, and Solaris. For more InnoDB tuning see

http://dev.mysql.com/doc/refman/5.1/en/innodb-tuning-troubleshootin

Copyright 2010 Oracle

The Worlds Most Popular Open Source Database

Cache hot application data in memory (Yoshinori Matsunobu)

DBT-2 (W200) Buffer pool 1G Buffer pool 2G Buffer pool 5G Buffer pool 30G (All data in cache)

Transactions per Minute %user 1125.44 1863.19 4385.18 36784.76 2% 3% 5.5% 36%

%iowait 30% 28% 33% 8%

DBT-2 benchmark (write intensive) 20-25GB hot data (200 warehouses, running 1 hour) Nehalem 2.93GHz x 8 cores, MySQL 5.5.2, 4 RAID1+0 HDDs RAM size affects everything. Not only for SELECT, but also for INSERT/UPDATE/DELETE INSERT: Random reads/writes happen when inserting into indexes in random order UPDATE/DELETE: Random reads/writes happen when modifying records

Copyright 2010 Oracle

The Worlds Most Popular Open Source Database

Connections
MySQL Caches the threads used by a connection
thread_cache_size - Number of threads to cache Setting this to 100 or higher is not unusual

Monitor Threads_created to see if this is an issue

Counts connections not using the thread cache Should be less that 1-2 a minute Usually only an issue if more than 1-2 a second

Only an issue is you create and drop a lot of connections, i.e. PHP Overhead is usually about 250k per thread Aborted_clients http://dev.mysql.com/doc/refman/5.1/en/communication-error Aborted_connections http://dev.mysql.com/doc/refman/5.1/en/communication-error

Copyright 2010 Oracle

The Worlds Most Popular Open Source Database

Sessions
Some session variables control space allocated by each session (connection)
Setting these to small can give bad performance Setting these too large can cause the server to swap! Can be set by connection Set small be default, increase in connections that need it
SET SORT_BUFFER_SIZE=1024*1024*128

sort_buffer_size - Used for ORDER BY, GROUP BY, SELECT DISTINCT, UNION DISTINCT
Monitor Sort_merge_passes < 1-2 an hour optimal Usually a problem in a reporting or data warehouse database

Other important session variables

read_rnd_buffer_size - Set to 1/2 sort_buffer_size join_buffer_size - (BAD) Watch Select_full_join read_buffer_size - Used for full table scans, watch Select_scan tmp_table_size - Max temp table size in memory, watch Created_tmp_disk_tables
Copyright 2010 Oracle The Worlds Most Popular Open Source Database

Query Cache
MySQLs Jekyll and Hyde of performance tuning options, when it is useful it really helps, when it hurts, it really hurts MySQL Query Cache caches both the query and the full result set
query_cache_type - Controls behavior
0 or OFF - Not used (buffer may still be allocated) 1 or ON cache all unless SELECT SQL_NO_CACHE (DEFAULT) 2 or DEMAND cache none unless SELECT SQL_CACHE

mysql> show status like 'Qc%' ; Gives great performance if:

query_cache_size - Determines the size of the cache

Identical queries returning identical data are used often No or rare inserts, updates or deletes

Best Practice Set to DEMAND Add SQL_CACHE to appropriate queries See http://dev.mysql.com/doc/refman/5.1/en/query-cache-configuratio
Copyright 2010 Oracle The Worlds Most Popular Open Source Database

Queries I
Often the # 1 issue in overall performance Always, Always have your slow query log on!
http://dev.mysql.com/doc/refman/5.1/en/slow-query-log.html Use: log_queries_not_using_indexes Check it regularly Use mysqldumpslow : http://dev.mysql.com/doc/refman/5.1/en/mysqldumpslow.html

Best practice is to automate running mysqldumpslow every morning and email results to DBA, DBDev, etc.

Understand and use EXPLAIN

http://dev.mysql.com/doc/refman/5.1/en/using-explain.html

Select_scan - Number of full table scans Select_full_join - Joins without indexes MySQL Query Analyzer
http://www.mysql.com/products/enterprise/query.html
Copyright 2010 Oracle The Worlds Most Popular Open Source Database

Queries II
The IN clause in MySLQ is very fast!
Select ... Where idx IN(1,23,345,456) Much faster than a join I have done tests with 80,000 items in the in list
1,000-2,000 not unusual

Dont wrap your indexes in expressions in Where

Select ... Where func(idx) = 20 [index ignored]
Select .. Where idx = otherfunc(20) [may use index]

Best practice : Keep index alone on left side of condition

Avoid % at the start of LIKE on an index

Select ... Where idx LIKE(ABC%) can use index Select ... Where idx LIKE(%XYZ) must do full table scan

Use union all when appropriate, default is union distinct! Understand left/right joins and use only when needed http://dev.mysql.com/doc/refman/5.1/en/query-speed.html
Copyright 2010 Oracle The Worlds Most Popular Open Source Database

MySQL Query Analyzer New!

Centralized monitoring of Queries across all servers with no reliance on Slow Query Logs Aggregated view of query execution counts, time, and rows returned = total query expense Saves time/effort parsing atomic executions for total query expense Saves time finding most expensive queries across all Production, Dev, and QA servers so SQL can be tuned. Finds code problems before your customers do.
Copyright 2010 Oracle The Worlds Most Popular Open Source Database

Schema I
Too many indexes slow down inserts/deletes
Use only the indexes you must have Check often

mysql>show create table tabname ; Dont duplicate leading parts of compound keys
index key123 (col1,col2,col3) index key12 (col1,col2) <- Not needed! index key1 (col1) <-- Not needed!

Use prefix indexes on large keys Best indexes are 16 bytes/chars or less Indexes bigger than 32 bytes/chars should be looked at very closely
should have there own cache if in MyISAM

For large strings that need to be indexed, i.e. URLs, consider using a separate column using the MySQL MD5 to create a hash key.
Copyright 2010 Oracle The Worlds Most Popular Open Source Database

Schema II
Size = performance, smaller is better
Size right! Do not automatically use 255 for VARCHAR
Temp tables, most caches, expand to full size

Use procedure analyse to determine the optimal types given the values in your table
http://dev.mysql.com/doc/refman/5.1/en/procedure-analyse.html

Consider the types:

mysql> select * from tab procedure analyse (64,2000) \G

enum : http://dev.mysql.com/doc/refman/5.1/en/enum.html set : http://dev.mysql.com/doc/refman/5.1/en/set.html

Compress large strings

Use the MySQL COMPRESS and UNCOMPRESS functions Very important in InnoDB!

Copyright 2010 Oracle

The Worlds Most Popular Open Source Database

Partitions: Deleting Data by Partition

Partitions can support deleting data by dropping a partition
If you insert 1,000,000 rows a day, eventually you need to delete 1,000,000 rows a day

Only works with Range and List partitioning Very useful for rolling date/time range Can be very useful even for small increments, i.e.1 hour Very fast, can be ~ 1-2 Secs

Copyright 2010 Oracle

The Worlds Most Popular Open Source Database

Partitions: Faster non-index Data Access

The MySQL optimizer is aware of the partitioning expression and can eliminate partitions to scan when the columns used in the partitioning expression are in the queries where condition Reduce or even eliminate indexes!

Copyright 2010 Oracle

The Worlds Most Popular Open Source Database

Partitions: Some operations are faster

Adding indexes can be faster Optimizations can be done by partition
If data is only being added to one partition then you can OPTIMIZE only that partition instead of running OPTIMIZE on the whole table

http://en.oreilly.com/mysql2010/public/schedule/detail/13383

Copyright 2010 Oracle

The Worlds Most Popular Open Source Database

MySQL PS Performance Tuning

When: Before go-live, or for application already deployed What: Tuning of database configuration, SQL, schema
Performance Tuning typically 3-5 days Analysis with benchmark, load testing Results typically 50-100% gain, sometimes 500%-1000%

Good for:
Applications with poor response times, hit wall recently? Solving system scalability issues Reduction of hardware $$$ outlay Prevention of new complex scaling architecture Load testing, benchmarking, capacity planning

Response Time

Starvation
Copyright 2010 Oracle

Concurrency

The Worlds Most Popular Open Source Database

Case Study: Performance Tuning

Situation:
Social Networking site having slow queries, crashes, & high disk i/o. Large number of mysql servers being optimized Benchmark based on general query log queries The conversion of tables to InnoDB, and the optimization of these tables Database configuration improvements and tuned for InnoDB Over 67 schema changes including:

What we did:

Results:

Adding multi-column indexes that are optimized for some queries Column data type optimizations The removal of unnecessary tables The removal of unnecessary indexes

68% faster on warm benchmark Dramatic reduction in disk i/o No more database crashing and corruption

Copyright 2010 Oracle

The Worlds Most Popular Open Source Database

Case Study: Performance Tuning

Copyright 2010 Oracle

The Worlds Most Popular Open Source Database

The presentation is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracles products remains at the sole discretion of Oracle.

Copyright 2010 Oracle

The Worlds Most Popular Open Source Database