Query Optimization

http://www.percona.com/training/

© 2011 - 2016 Percona, Inc. 1 / 138

 Table of Contents

1. Query Planning 5. JOINOptimization

2. Explaining the EXPLAIN 6. Subquery Optimization
3. Composite Indexes 7. Other EXPLAINtechniques
4. Other Indexing Techniques 8. Beyond EXPLAIN

© 2011 - 2016 Percona, Inc. 2 / 138

 Query Optimization
QUERY PLANNING

© 2011 - 2016 Percona, Inc. 3 / 138

 About This Chapter
The number one goal is to have faster queries.
The process is:
We first ask MySQL what its intended execution plan is.
If we don't like it, we make a change, and try again...

© 2011 - 2016 Percona, Inc. 4 / 138

 It All Starts with EXPLAIN
Bookmark this manual page:
http://dev.mysql.com/doc/refman/5.7/en/explain-output.html
It is the best source for anyone getting started.

© 2011 - 2016 Percona, Inc. 5 / 138

 Example Data
IMDB database loaded into InnoDB tables (~5GB)
Download it and import it for yourself using imdbpy2sql.py:
http://imdbpy.sourceforge.net

© 2011 - 2016 Percona, Inc. 6 / 138

 First Example
CREATE TABLE title (
id int NOT NULL AUTO_INCREMENT,
title text NOT NULL,
imdb_index varchar(12) DEFAULT NULL,
kind_id int NOT NULL,
production_year int DEFAULT NULL,
imdb_id int DEFAULT NULL,
phonetic_code varchar(5) DEFAULT NULL,
episode_of_id int DEFAULT NULL,
season_nr int DEFAULT NULL,
episode_nr int DEFAULT NULL,
series_years varchar(49) DEFAULT NULL,
title_crc32 int(10) unsigned DEFAULT NULL,
PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

© 2011 - 2016 Percona, Inc. 7 / 138

 Find the Title Bambi
mysql> EXPLAIN SELECT id,title,production_year FROM title
-> WHERE title = 'Bambi' ORDER BY production_year\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: title
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 1535171
Extra: Using where; Using filesort
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 8 / 138

 Aha! Now Add an Index
mysql> ALTER TABLE title ADD INDEX (title);
ERROR 1170 (42000): BLOB/TEXT column 'title' used in key
specification without a key length

© 2011 - 2016 Percona, Inc. 9 / 138

 Aha! Now Add an Index
mysql> ALTER TABLE title ADD INDEX (title);
ERROR 1170 (42000): BLOB/TEXT column 'title' used in key
specification without a key length

mysql> ALTER TABLE title ADD INDEX (title(50));

Query OK, 0 rows affected (8.09 sec)
Records: 0 Duplicates: 0 Warnings: 0

© 2011 - 2016 Percona, Inc. 10 / 138

 Let's Revisit
mysql> EXPLAIN SELECT id,title,production_year FROM title
-> WHERE title = 'Bambi' ORDER by production_year\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: title
type: ref
possible_keys: title
key: title
key_len: 152
ref: const
rows: 4
Extra: Using where; Using filesort
1 row in set (0.00 sec)

Using = for comparison, but not PK lookup.

Identified 'title' as a candidate index and chose it.
Size of the index used.
Anticipated number of rows.

© 2011 - 2016 Percona, Inc. 11 / 138

 Other Ways of Accessing
mysql> EXPLAIN SELECT id,title,production_year FROM title
-> WHERE id = 55327\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: title
type: const
possible_keys: PRIMARY
key: PRIMARY
key_len: 4
ref: const
rows: 1
Extra: NULL
1 row in set (0.00 sec)

const: at most, one matching row.

Primary Key in InnoDB is always faster than secondary keys.

© 2011 - 2016 Percona, Inc. 12 / 138

 LIKE
mysql> EXPLAIN SELECT id,title,production_year FROM title
-> WHERE title LIKE 'Bamb%'\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: title
type: range
possible_keys: title
key: title
key_len: 152
ref: NULL
rows: 98
Extra: Using where
1 row in set (0.00 sec)

Type is Range. BETWEEN, IN() and < > are also ranges.
Number of rows to examine has increased; we are not specific enough.

© 2011 - 2016 Percona, Inc. 13 / 138

 Why is That a Range?
We're looking for titles between BambA and BambZ*
When we say index in MySQL, we mean trees.
That is, B-Tree/B+Tree/T-Tree.
Pretend they're all the same (for simplification).
There is only radically different indexing methods for
specialized uses: MEMORY Hash, FULLTEXT, spatial or
3rd party engines.

© 2011 - 2016 Percona, Inc. 14 / 138

 What's That?

© 2011 - 2016 Percona, Inc. 15 / 138

 Could This Be a Range?
mysql> EXPLAIN SELECT id,title,production_year FROM title
-> WHERE title LIKE '%ulp Fiction'\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: title
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 1442263
Extra: Using where
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 16 / 138

 No, We Can't Traverse

© 2011 - 2016 Percona, Inc. 17 / 138

 LIKE 'Z%'
mysql> EXPLAIN SELECT id,title,production_year FROM title
-> WHERE title LIKE 'Z%'\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: title
type: range
possible_keys: title
key: title
key_len: 77
ref: NULL
rows: 13718
Extra: Using where
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 18 / 138

 LIKE 'T%'
mysql> EXPLAIN SELECT id,title,production_year FROM title
-> WHERE title LIKE 'T%'\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: title
type: ALL
possible_keys: title
key: NULL
key_len: NULL
ref: NULL
rows: 1442263
Extra: Using where
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 19 / 138

 LIKE 'The %'
mysql> EXPLAIN SELECT id,title,production_year FROM title
-> WHERE title LIKE 'The %'\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: title
type: ALL
possible_keys: title
key: NULL
key_len: NULL
ref: NULL
rows: 1442263
Extra: Using where
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 20 / 138

 MySQL is Reasonably Smart
It dynamically samples the data to choose which is the better
choice—or in some cases uses static statistics.
This helps the optimizer choose:
Which indexes will be useful.
Which indexes should be avoided.
Which is the better index when there is more than one.

© 2011 - 2016 Percona, Inc. 21 / 138

 Why Avoid Indexes?
B-Trees work like humans search a phone book;
Use an index if you want just a few rows.
Scan cover-to-cover if you want a large percentage.

© 2011 - 2016 Percona, Inc. 22 / 138

 Why Avoid Indexes (cont.)
Benchmark on a different schema (lower is better):

© 2011 - 2016 Percona, Inc. 23 / 138

 What You Should Take Away
Data is absolutely critical.
Development environments should contain sample data
exported from production systems.
A few thousands of rows is usually enough for the optimizer
to behave like it does in production.

© 2011 - 2016 Percona, Inc. 24 / 138

 What You Should Take Away (cont.)
Input values are absolutely critical.
Between two seemingly identical queries, execution plans
may be very different.
Just like you test application code functions with several
values for input arguments.

© 2011 - 2016 Percona, Inc. 25 / 138

 Query Optimization
EXPLAINING THE EXPLAIN

© 2011 - 2016 Percona, Inc. 26 / 138

 How to Explain the EXPLAIN
In queries with regular joins, tables are read in the order
displayed by EXPLAIN.
The id column is a sequential identifier of SELECT statements
in the query.
The select_type column indicates type of SELECT (simple,
primary, subquery, union, derived, ...).
The type column says which join type will be used.
The possible_keys column indicates which indexes MySQL can
choose from use to find the rows in this table.
The key column indicates which index is used.

© 2011 - 2016 Percona, Inc. 27 / 138

 How to Explain the EXPLAIN (cont.)
key_len tells the length of the key that was used (important to
find which parts of a composite index are used).
ref shows which columns or constants are compared to the
index named in key column to select rows from the table.
rows says how many rows have to be examined in order to
execute each step of the query.
Extra contains additional information about how MySQL
resolves the query
http://dev.mysql.com/doc/refman/5.7/en/explain-output.html#explain-extra-information

© 2011 - 2016 Percona, Inc. 28 / 138

 Types in EXPLAIN
The following slides show possible values for EXPLAIN type,
ordered (approximately) from the fastest to the slowest.
FULLTEXT access type (and its special indexes) are not
covered on this section.

© 2011 - 2016 Percona, Inc. 29 / 138

 NULL
Not really a plan: no data is returned
See ‘Extra’ for a reason
mysql> EXPLAIN SELECT * FROM title WHERE 1 = 2\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: NULL
type: NULL
possible_keys: NULL
key: NULL -- Internally equivalent to
key_len: NULL -- SELECT NULL WHERE 0;
ref: NULL
rows: NULL
Extra: Impossible WHERE
1 row in set (0.00 sec)

mysql> EXPLAIN SELECT * FROM title WHERE id = -1\G

...
type: NULL
Extra: Impossible WHERE noticed after reading const tables

© 2011 - 2016 Percona, Inc. 30 / 138

 system
The table has only one row (=system table)
A seldom used special case of the const joint type
mysql> EXPLAIN SELECT id FROM (SELECT * FROM title LIMIT 1) AS one\G

************* 1. row *************

id: 1
select_type: PRIMARY
table: <derived2>>
type: system
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 1
Extra: NULL
************* 2. row *************
id: 2
select_type: DERIVED
table: title
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 1396980
Extra: NULL
2 rows in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 31 / 138

 const
Used when comparing a literal with a non-prefix
PRIMARY/UNIQUE index.
The table has at the most one matching row, which will be read
at the start of the query.
Because there is only one row, the values can be regarded as
constants by the optimizer. *This is very fast since table is read
only once.

© 2011 - 2016 Percona, Inc. 32 / 138

 const (cont.)
mysql> EXPLAIN SELECT * FROM title WHERE id = 55327\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: title
type: const
possible_keys: PRIMARY
key: PRIMARY
key_len: 4
ref: const
rows: 1
Extra: NULL
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 33 / 138

 eq_ref
One row will be read from this table for each combination of
rows from the previous tables.
The best possible join type (after const).
Used when the whole index is used for the = operator with a
UNIQUE or PRIMARY KEY.

© 2011 - 2016 Percona, Inc. 34 / 138

 eq_ref (cont.)
mysql> EXPLAIN SELECT title.title, kind_type.kind
-> FROM kind_type JOIN title ON kind_type.id = title.kind_id
-> WHERE title.title = 'Bambi'\G

************* 1. row ************* ************* 2. row *************

id: 1 id: 1
select_type: SIMPLE select_type: SIMPLE
table: title table: kind_type
type: ALL type: eq_ref
possible_keys: NULL possible_keys: PRIMARY
key: NULL key: PRIMARY
key_len: NULL key_len: 4
ref: NULL ref: imdb.title.kind_id
rows: 1396980 rows: 1
Extra: Using where Extra: NULL

© 2011 - 2016 Percona, Inc. 35 / 138

 ref
Several rows will be read from this table for each combination
of rows from the previous tables.
Used if the join uses only a left-most prefix of the index, or if
the index is not UNIQUE or PRIMARY KEY.
Still not bad, if the index matches only few rows.

© 2011 - 2016 Percona, Inc. 36 / 138

 ref (cont.)
mysql> ALTER TABLE users ADD INDEX (first_name);
mysql> EXPLAIN SELECT distinct u1.first_name FROM users u1 JOIN users u2
-> WHERE u1.first_name = u2.first_name and u1.id <> u2.id\G

************* 1. row ************* ************* 2. row *************

id: 1 id: 1
select_type: SIMPLE select_type: SIMPLE
table: u1 table: u2
type: index type: ref
possible_keys: first_name possible_keys: first_name
key: first_name key: first_name
key_len: 102 key_len: 102
ref: NULL ref: imdb.u1.first_name
rows: 49838 rows: 8
Extra: Using index; Extra: Using where;
Using temporary Using index;
Distinct

Can you think of a more efficient way of writing this query?

© 2011 - 2016 Percona, Inc. 37 / 138

 ref (cont.)
mysql> ALTER TABLE users ADD INDEX (first_name);
mysql> EXPLAIN SELECT distinct u1.first_name FROM users u1 JOIN users u2
-> WHERE u1.first_name = u2.first_name and u1.id <> u2.id\G

************* 1. row ************* ************* 2. row *************

id: 1 id: 1
select_type: SIMPLE select_type: SIMPLE
table: u1 table: u2
type: index type: ref
possible_keys: first_name possible_keys: first_name
key: first_name key: first_name
key_len: 102 key_len: 102
ref: NULL ref: imdb.u1.first_name
rows: 49838 rows: 8
Extra: Using index; Extra: Using where;
Using temporary Using index;
Distinct

Can you think of a more efficient way of writing this query?

mysql> SELECT first_name FROM users GROUP BY first_name
-> HAVING count(first_name) > 1;
© 2011 - 2016 Percona, Inc. 38 / 138
 ref_or_null
This is a join type, like ref, but with the addition that MySQL
does an extra search for rows that contain NULL values.
This join type optimization is used most often in resolving
subqueries.

© 2011 - 2016 Percona, Inc. 39 / 138

 ref_or_null (cont.)
mysql> EXPLAIN SELECT * FROM cast_info
-> WHERE nr_order = 1 or nr_order IS NULL\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: cast_info
type: ref_or_null
possible_keys: nr_order
key: nr_order
key_len: 5
ref: const
rows: 12193688
Extra: Using index condition
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 40 / 138

 index_merge
Results from more than one index are combined either by
intersection or union.
In this case, the key column contains a list of indexes.
mysql> EXPLAIN SELECT * FROM title
-> WHERE title = 'Dracula' or production_year = 1922\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: title
type: index_merge
possible_keys: production_year,title
key: title,production_year
key_len: 77,5
ref: NULL
rows: 2895
Extra: Using sort_union(title,production_year); Using where
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 41 / 138

 unique_subquery/index_subquery
unique_subquery
The result of a subquery is covered by a unique index.
The subquery is used within an IN(...) predicate.
index_subquery
Similar to unique_subquery, only allowing for non-unique
indexes.

© 2011 - 2016 Percona, Inc. 42 / 138

 unique_subquery/index_subquery (cont.)
mysql> EXPLAIN SELECT * FROM title WHERE title = 'Bambi'
-> AND kind_id NOT IN
-> (SELECT id FROM kind_type WHERE kind like 'tv%')\G

************ 1. row ************ ************ 2. row ************

id: 1 id: 2
select_type: PRIMARY select_type: DEPENDENT SUBQUERY
table: title table: kind_type
type: ref type: unique_subquery
possible_keys: title possible_keys: PRIMARY
key: title key: PRIMARY
key_len: 77 key_len: 4
ref: const ref: func
rows: 4 rows: 1
Extra: Using where Extra: Using where
2 rows in set (0.04 sec)

For index_subquery, use a non-

PRIMARY, non-UNIQUE key

© 2011 - 2016 Percona, Inc. 43 / 138

 range
Only rows that are in a given range will be retrieved.
An index will still be used to select the rows
The key_len contains the longest key part that is used.
The ref column will be NULL for this type.

© 2011 - 2016 Percona, Inc. 44 / 138

 range (cont.)
mysql> EXPLAIN SELECT * FROM title
-> WHERE title = 'Bambi'
-> OR title = 'Dumbo' OR title = 'Cinderella'\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: title
type: range
possible_keys: title
key: title
key_len: 77
ref: NULL
rows: 49
Extra: Using where
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 45 / 138

 range (cont.)
mysql> EXPLAIN SELECT * FROM title
-> WHERE title like 'Bamb%';
***************** 1. row *****************
id: 1
select_type: SIMPLE
table: title
type: range
possible_keys: title
key: title
key_len: 62
ref: NULL
rows: 97
Extra: Using where

© 2011 - 2016 Percona, Inc. 46 / 138

 range (cont.)
mysql> EXPLAIN SELECT * FROM title
-> WHERE production_year BETWEEN 1998 AND 1999\G
***************** 1. row *****************
id: 1
select_type: SIMPLE
table: title
type: range
possible_keys: py
key: py
key_len: 5
ref: NULL
rows: 110484
Extra: Using index condition
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 47 / 138

 index
The whole index tree is scanned.
Otherwise same as ALL.
Faster than ALL since the index file is (should be) smaller than
the data file.
MySQL can use this join type when the query uses only
columns that are part of a single index.

© 2011 - 2016 Percona, Inc. 48 / 138

 index (cont.)
mysql> EXPLAIN SELECT count(*), production_year,
-> group_concat(DISTINCT kind_id ORDER BY kind_id) as kind_id
-> FROM title
-> GROUP BY production_year ORDER BY production_year\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: title
type: index
possible_keys: NULL
key: production_year
key_len: 5
ref: NULL
rows: 1396980
Extra: NULL
1 row in set (0.00 sec)

“How many releases per year, and what are their types”

© 2011 - 2016 Percona, Inc. 49 / 138

 ALL
A full table scan; the entire table is scanned.
Not good even for the first (non-const) table.
Very bad for subsequent tables, since it means a full table scan
for each combination of rows from the previous tables is
performed.
Solutions: rephrase query, add more indexes.

© 2011 - 2016 Percona, Inc. 50 / 138

 ALL (cont.)
mysql> EXPLAIN SELECT * from title
-> WHERE MAKEDATE(production_year, 1) >= now() - INTERVAL 3 YEAR\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: title
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 1442263
Extra: Using where
1 row in set (0.00 sec)

Assume an index exists on production_year. Same result. Why?

© 2011 - 2016 Percona, Inc. 51 / 138

 Extra:What You Would Like to See
Using index
Excellent! MySQL can search for the rows directly from the index tree,
without reading the actual table (covering index)
Distinct
Good! Only one row for each combination from the previous tables
Not exists
Good! MySQL is able to do a LEFT JOIN optimization, and some rows can
be left out

© 2011 - 2016 Percona, Inc. 52 / 138

 Extra:What You Don’t Like to See
Using filesort
Extra sorting pass needed! (Does not imply file-on-disk!)
Using temporary
Temporary table needed! (Does not imply on-disk.)
Typically happens with different ORDER BY and GROUP BY
Using join buffer
Tables are processed in large batches of rows, instead of by indexed
lookups.
Range checked for each record (index map: N)
Individual records are separately optimized for index retrieval
This is not fast, but faster than a join with no index at all.

© 2011 - 2016 Percona, Inc. 53 / 138

 Extra:Using where- Example 1
After fetching all rows, we look for 'Bambi'
mysql> EXPLAIN SELECT id,title,production_year FROM title
-> WHERE title = 'Bambi' ORDER BY production_year\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: title
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 1535171
Extra: Using where; Using filesort
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 54 / 138

 Extra:Using where- Example 2
After using the index (limited to 50 characters), we do
additional filtering on title:
mysql> EXPLAIN SELECT id,title,production_year FROM title
-> WHERE title LIKE 'Bamb%'\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: title
type: range
possible_keys: title
key: title
key_len: 152
ref: NULL
rows: 98
Extra: Using where
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 55 / 138

 Extra:Using where- Example 3
No Using whereas the full column last_nameis
indexed. All filtering is done using indexes:
mysql> EXPLAIN SELECT * FROM users
-> WHERE last_name = 'Ascencio'\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: users
type: ref
possible_keys: last_name
key: last_name
key_len: 102
ref: const
rows: 2
Extra: Using index condition
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 56 / 138

 Extra:Using where- Example 3
After filtering using last_nameindex, extra filtering happens
on first_name:
mysql> EXPLAIN SELECT * FROM users
-> WHERE last_name = 'Ascencio' and first_name = 'Virginia'\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: users
type: ref
possible_keys: last_name
key: last_name
key_len: 102
ref: const
rows: 2
Extra: Using index condition; Using where
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 57 / 138

 Extra:Using where- Thoughts
After fetching rows from the storage engine (independent on how the query was
executed), extra filtering has to happen for each row.

Not having Using whereis good.

Trying to 'optimize' away Using wherecompletely is not good as it would

require a lot of indexes in most/all applications.

Ok to have Using where:

If after using the index to filter within the engine, only a smaller number of
rows need to be additonally filtered.
Not ok to have Using where:
A very poor index was used, matching a lot more rows than necessary
which have to be returned and filtered above storage layer.

© 2011 - 2016 Percona, Inc. 58 / 138

 Query Optimization
COMPOSITE INDEXES

© 2011 - 2016 Percona, Inc. 59 / 138

 EXERCISE: Add Index(es) to Query
mysql> EXPLAIN SELECT * FROM title WHERE title = 'Pilot' AND
-> production_year BETWEEN 1997 AND 2009\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: title
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 1496878
Extra: Using where
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 60 / 138

 We Are Spoiled for Choice
Which one do we choose?
ALTER TABLE title ADD INDEX py (production_year);
ALTER TABLE title ADD INDEX t (title(30));
ALTER TABLE title ADD INDEX py_t (production_year, title(30));
ALTER TABLE title ADD INDEX t_py (title(30), production_year);

Let's try the first one:

mysql> ALTER TABLE title ADD INDEX py (production_year);
Query OK, 0 rows affected (4.99 sec)
Records: 0 Duplicates: 0 Warnings: 0

© 2011 - 2016 Percona, Inc. 61 / 138

 Index on (production_year)
mysql> EXPLAIN SELECT * FROM title WHERE title = 'Pilot' AND
-> production_year BETWEEN 1997 AND 2009\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: title
type: ALL
possible_keys: py
key: NULL
key_len: NULL
ref: NULL
rows: 1496878
Extra: Using where
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 62 / 138

 How about a Smaller Range?
mysql> EXPLAIN SELECT * FROM title WHERE title = 'Pilot' AND
-> production_year BETWEEN 2008 AND 2009\G

*************************** 1. row ***************************

id: 1
select_type: SIMPLE
table: title
type: range
possible_keys: py
key: py
key_len: 5
ref: NULL
rows: 190666
Extra: Using index condition; Using where

1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 63 / 138

 Index on (title)
mysql> ALTER TABLE title ADD INDEX t (title(30));
Query OK, 0 rows affected (6.35 sec)
Records: 0 Duplicates: 0 Warnings: 0

mysql> EXPLAIN SELECT * FROM title WHERE title = 'Pilot' AND

-> production_year BETWEEN 2008 AND 2009\G

********* 1. row *********

id: 1
select_type: SIMPLE
table: title
type: ref
possible_keys: py,t
key: t
key_len: 92
ref: const
rows: 1411
Extra: Using where
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 64 / 138

 Comparing the two
mysql> EXPLAIN SELECT * FROM title WHERE title = 'Pilot' AND
-> production_year BETWEEN 2008 AND 2009\G

************ 1. row ***************** ********* 1. row *********

id: 1 id: 1
select_type: SIMPLE select_type: SIMPLE
table: title table: title
type: range type: ref
possible_keys: py possible_keys: py,t
key: py key: t
key_len: 5 key_len: 92
ref: NULL ref: const
rows: 190666 rows: 1411
Extra: Using index condition; Extra: Using where
Using where

© 2011 - 2016 Percona, Inc. 65 / 138

 Composite Indexes
Which is better?
INDEX py_t (production_year, title)
INDEX t_py (title, production_year)

© 2011 - 2016 Percona, Inc. 66 / 138

 Index on (py_t)
mysql> ALTER TABLE title ADD INDEX py_t (production_year, title(30));
Query OK, 0 rows affected (9.11 sec)
Records: 0 Duplicates: 0 Warnings: 0

mysql> EXPLAIN SELECT * FROM title WHERE title = 'Pilot'

-> AND production_year BETWEEN 2008 AND 2009\G

********* 1. row *********

id: 1
select_type: SIMPLE
table: title
type: ref
possible_keys: py,t,py_t
key: t
key_len: 92
ref: const
rows: 1411
Extra: Using where
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 67 / 138

 Index on (py_t) Visualized

© 2011 - 2016 Percona, Inc. 68 / 138

 Index on (t_py)
mysql> ALTER TABLE title ADD INDEX t_py (title(30), production_year);
Query OK, 0 rows affected (9.11 sec)
Records: 0 Duplicates: 0 Warnings: 0

mysql> EXPLAIN SELECT * FROM title WHERE title = 'Pilot' AND

-> production_year BETWEEN 2008 AND 2009\G

********* 1. row *********

id: 1
select_type: SIMPLE
table: title
type: range
possible_keys: py,t,py_t,t_py
key: t_py
key_len: 97
ref: NULL
rows: 73
Extra: Using where
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 69 / 138

 Index on (t_py) Visualized

© 2011 - 2016 Percona, Inc. 70 / 138

 Recommendations
Don’t know what order to specify the columns?
RULE: Think about how the equality comparisons narrow
down the subset of rows to examine. Define the index so the
leftmost columns filter most effectively.
EXCEPTION: If you have a range comparison (!=, <, >,
BETWEEN, LIKE), those columns should go to the right in
the index.

© 2011 - 2016 Percona, Inc. 71 / 138

 Recommendations (cont.)
Columns after the range-comparison column can’t be used for
filtering in MySQL <5.6
but may still be useful in the index, as we’ll see
We can still push down those extra columns to the engine (>=
5.6), having a speed up if the condition is very selective

© 2011 - 2016 Percona, Inc. 72 / 138

 Using index condition (5.6+)
mysql> EXPLAIN SELECT * FROM title WHERE title LIKE 'B%' AND
-> production_year BETWEEN 1945 AND 1950\G

********* 1. row *********

id: 1
select_type: SIMPLE
table: title
type: range
possible_keys: title,production_year,production_year_title
key: production_year_title
key_len: 82
ref: NULL
rows: 23496
Extra: Using index condition; Using where

© 2011 - 2016 Percona, Inc. 73 / 138

 Query Optimization
OTHER INDEXING TECHNIQUES

© 2011 - 2016 Percona, Inc. 74 / 138

 Indexes are Multi-Purpose
So far indexes have only been used for filtering.
This is the most typical case—don’t forget it.
There are also other ways MySQL can use indexes:
Avoiding having to sort.
Preventing temporary tables.
Avoiding reading rows from the tables.

© 2011 - 2016 Percona, Inc. 75 / 138

 The First Example Again
mysql> EXPLAIN SELECT id, title, production_year FROM title
-> WHERE title = 'Bambi' ORDER BY production_year\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: title
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 1496878
Extra: Using where; Using filesort
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 76 / 138

 Index Prevents Sort
mysql> EXPLAIN SELECT id, title, production_year FROM title
-> WHERE title = 'Bambi' ORDER BY production_year\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: title
type: ref
possible_keys: t_py
key: t_py
key_len: 92
ref: const
rows: 3
Extra: Using where
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 77 / 138

 Temporary Table
mysql> EXPLAIN SELECT COUNT(*) AS c, production_year
-> FROM title GROUP BY production_year\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: title
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 1496878
Extra: Using temporary; Using filesort
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 78 / 138

 Full Index Scan
mysql> ALTER TABLE title ADD INDEX py (production_year);

mysql> EXPLAIN SELECT COUNT(*) AS c, production_year FROM title

-> GROUP BY production_year\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: title
type: index
possible_keys: py
key: py
key_len: 5
ref: NULL
rows: 1496878
Extra: Using index
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 79 / 138

 Retrieving Only Limited Columns
mysql> SELECT person_id FROM cast_info WHERE person_role_id = 35721;

What's the difference between indexes on

(person_role_id)and (person_role_id,
person_id)?

© 2011 - 2016 Percona, Inc. 80 / 138

 Retrieving Only Limited Columns (cont.)
mysql> ALTER TABLE cast_info ADD INDEX (person_role_id);

mysql> EXPLAIN SELECT person_id FROM cast_info

-> WHERE person_role_id = 35721\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: cast_info
type: ref
possible_keys: person_role_id
key: person_role_id
key_len: 5
ref: const
rows: 146
Extra: NULL
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 81 / 138

 Covering Index Optimization
mysql> ALTER TABLE cast_info
-> ADD INDEX person_role_id_person_id (person_role_id, person_id);

mysql> EXPLAIN SELECT person_id FROM cast_info

-> WHERE person_role_id = 35721\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: cast_info
type: ref
possible_keys: person_role_id,person_role_id_person_id
key: person_role_id_person_id
key_len: 5
ref: const
rows: 146
Extra: Using index
1 row in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 82 / 138

 Prefix Indexes
The problem with this schema, is there’s just a couple of
outliers with really long names:
mysql> SELECT MAX(LENGTH(title)) mysql> SELECT MAX(LENGTH(name))
-> FROM title; -> FROM char_name;
+--------------------+ +-------------------+
| MAX(LENGTH(title)) | | MAX(LENGTH(name)) |
+--------------------+ +-------------------+
| 334 | | 478 |
+--------------------+ +-------------------+
1 row in set (0.98 sec) 1 row in set (1.42 sec)

Two Ways to Solve This

Prefix-Index
Emulate Hash Index

© 2011 - 2016 Percona, Inc. 83 / 138

 Option 1: Prefix-Index Selection
Pick a good length to get a lot of uniqueness:
mysql> SELECT count(distinct(title)) AS n_unique, count(distinct(LEFT(title, 100))) AS n100,
count(distinct(LEFT(title, 75))) AS n75, count(distinct(LEFT(title, 50))) AS n50,
count(distinct(LEFT(title, 40))) AS n40, count(distinct(LEFT(title, 30))) AS n30,
count(distinct(LEFT(title, 20))) AS n20, count(distinct(LEFT(title, 10))) AS n10,
count(distinct(LEFT(title, 4))) AS n4 FROM title;
+----------+--------+--------+--------+--------+--------+--------+--------+-------+
| n_unique | n100 | n75 | n50 | n40 | n30 | n20 | n10 | n4 |
+----------+--------+--------+--------+--------+--------+--------+--------+-------+
| 998335 | 998320 | 998291 | 997887 | 996727 | 991532 | 960894 | 624949 | 52306 |
+----------+--------+--------+--------+--------+--------+--------+--------+-------+
1 row in set (48.68 sec)

96% uniqueness with only 20 chars instead of 300+ Looks

pretty good!
mysql> ALTER TABLE title ADD INDEX (title(20));

© 2011 - 2016 Percona, Inc. 84 / 138

 Option 2: Emulate Hash Index
mysql> ALTER TABLE title ADD INDEX (title_crc32);

mysql> UPDATE title SET title_crc32 = crc32(title);

mysql> SELECT COUNT(DISTINCT(BINARY title)),

COUNT(DISTINCT(title_crc32)) FROM title;
+-------------------------------+------------------------------+
| COUNT(DISTINCT(BINARY title)) | COUNT(DISTINCT(title_crc32)) |
+-------------------------------+------------------------------+
| 1001509 | 1001393 |
+-------------------------------+------------------------------+

© 2011 - 2016 Percona, Inc. 85 / 138

 Option 2: Hash Index (cont.)
All queries need to be transformed slightly to:
mysql> SELECT * FROM title WHERE title_crc32 = crc32('Bambi')
-> AND title = 'Bambi';

All UPDATEs/INSERTs also need to update the value of

title_crc32every time a title changes.
This can be done easily via the application or you can use a
trigger if your write load is low enough.

© 2011 - 2016 Percona, Inc. 86 / 138

 Pros and Cons of the Two Solutions
Prefix Index: Hash Index:
Pro: Built in to MySQL/no Pro: Very Good when there
magic required. is not much uniqueness until
Cons: Not very effective very far into the string.
when the start of the string is Cons: Equality searches
not very unique. only. Requires ugly magic to
work with collations/ case
sensitivity.

© 2011 - 2016 Percona, Inc. 87 / 138

 Index Hints
Optimizer decision making is all about tradeoffs.
MySQL wants to pick the best plan but it can’t be exhaustive in
deciding if it takes too long.
If MySQL doesn’t pick correctly, you can override:
USE INDEX
FORCE INDEX
IGNORE INDEX

http://dev.mysql.com/doc/refman/5.7/en/index-hints.html
© 2011 - 2016 Percona, Inc. 88 / 138
 USE INDEX
Tell the optimizer to consider only the named index.
mysql> SELECT * FROM title USE INDEX (py_t)
-> WHERE production_year = 2009;

© 2011 - 2016 Percona, Inc. 89 / 138

 FORCE INDEX
Like USE INDEX, consider only the named index.
But also tell the optimizer that a table-scan is very expensive,
so prefer to use the index, instead of analyzing the breakpoint
when a table-scan may be easier.
mysql> SELECT * FROM title FORCE INDEX (title)
-> WHERE title LIKE 'The %';

© 2011 - 2016 Percona, Inc. 90 / 138

 IGNORE INDEX
Tells the optimizer not to use a specified index.
mysql> SELECT * FROM title IGNORE INDEX (t_p)
-> WHERE title LIKE 'The %';

© 2011 - 2016 Percona, Inc. 91 / 138

 Caveats of Index Hints
Even USE INDEXor FORCE INDEXdoesn’t make an index
help if it’s totally inapplicable to the query:
mysql> SELECT * FROM title USE INDEX (t_py)
-> WHERE production_year = 2009;

The optimizer handles most cases well. If your query is costly,

it’s more likely that you have the wrong indexes than the
optimizer is making a mistake.
Hard-coding your application to use a specific index means that
after you create the right index, you’ll have to change your
code as well.

© 2011 - 2016 Percona, Inc. 92 / 138

 Optimizer Hints MySQL >= 5.7
Very granular configuration of optimizer_switch
https://dev.mysql.com/doc/refman/5.7/en/optimizer-hints.html

SELECT /*+ NO_RANGE_OPTIMIZATION(t3 PRIMARY, f2_idx) */ f1

FROM t3 WHERE f1 > 30 AND f1 < 33;

SELECT /*+ BKA(t1) NO_BKA(t2) */ * FROM t1 INNER JOIN t2 WHERE ...;

SELECT /*+ NO_ICP(t1, t2) */ * FROM t1 INNER JOIN t2 WHERE ...;

SELECT /*+ SEMIJOIN(FIRSTMATCH, LOOSESCAN) */ * FROM t1 ...;

EXPLAIN SELECT /*+ NO_ICP(t1) */ * FROM t1 WHERE ...;

© 2011 - 2016 Percona, Inc. 93 / 138

 EXERCISE: Query Tuning
It’s now your turn to optimize these queries before we continue
on:
mysql> SELECT * FROM name WHERE name = 'Brosnan, Pierce';

mysql> SELECT count(*) c, person_id

-> FROM person_info GROUP by person_id;

© 2011 - 2016 Percona, Inc. 94 / 138

 Query Optimization
JOIN OPTIMIZATION

© 2011 - 2016 Percona, Inc. 95 / 138

 JOINAnalysis
mysql> EXPLAIN SELECT person_info.* FROM name INNER JOIN person_info
-> ON (name.id = person_info.person_id) AND name.name = 'Johansson, Scarlett'\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: name
type: ref
possible_keys: PRIMARY,name
key: name
key_len: 62
ref: const
rows: 1
Extra: Using where
************* 2. row *************
id: 1
select_type: SIMPLE
table: person_info
type: ref
possible_keys: person_id
key: person_id
key_len: 4
ref: imdb.name.id
rows: 2
Extra: NULL
2 rows in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 96 / 138

 mysql> EXPLAIN SELECT name.* FROM name INNER JOIN cast_info ON name.id =
-> cast_info.person_id INNER JOIN char_name ON cast_info.person_role_id =
-> char_name.id WHERE char_name.name = 'James Bond'\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: cast_info
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 21526132
Extra: Using where
************* 2. row *************
id: 1
select_type: SIMPLE
table: char_name
type: eq_ref
possible_keys: PRIMARY
key: PRIMARY
key_len: 4
ref: imdb.cast_info.person_role_id
rows: 1
Extra: Using where
************* 3. row *************
id: 1
select_type: SIMPLE
table: name
type: eq_ref
possible_keys: PRIMARY
key: PRIMARY
key_len: 4
ref: imdb.cast_info.person_id
rows: 1
Extra: NULL
3 rows in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 97 / 138

 First Index; A Logical Choice
mysql> ALTER TABLE char_name ADD index name_idx (name(30));
Query OK, 0 rows affected (1 min 56.10 sec)
Records: 0 Duplicates: 0 Warnings: 0

© 2011 - 2016 Percona, Inc. 98 / 138

 mysql> EXPLAIN SELECT name.* FROM name INNER JOIN cast_info ON name.id =
-> cast_info.person_id INNER JOIN char_name ON cast_info.person_role_id =
-> char_name.id WHERE char_name.name = 'James Bond'\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: cast_info
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL The index didn’t really help
rows: 21526132
Extra: Using where MySQL. Same EXPLAIN plan.
************* 2. row *************
id: 1
select_type: SIMPLE
table: char_name
type: eq_ref
possible_keys: PRIMARY
key: PRIMARY
key_len: 4
ref: imdb.cast_info.person_role_id
rows: 1
Extra: Using where
************* 3. row *************
id: 1
select_type: SIMPLE
table: name
type: eq_ref
possible_keys: PRIMARY
key: PRIMARY
key_len: 4
ref: imdb.cast_info.person_id
rows: 1
Extra: NULL
3 rows in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 99 / 138

 Next Index; A Better Choice
mysql> ALTER TABLE cast_info ADD INDEX (person_role_id, person_id);
Query OK, 0 rows affected (1 min 52.70 sec)
Records: 0 Duplicates: 0 Warnings: 0

© 2011 - 2016 Percona, Inc. 100 / 138

 mysql> EXPLAIN SELECT name.* FROM name INNER JOIN cast_info ON name.id =
-> cast_info.person_id INNER JOIN char_name ON cast_info.person_role_id =
-> char_name.id WHERE char_name.name = 'James Bond'\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: char_name
type: ref
possible_keys: PRIMARY,name_idx
key: name_idx
key_len: 92
ref: const Notice the order of the tables changed!
rows: 1
Extra: Using where
************* 2. row *************
id: 1
select_type: SIMPLE
table: cast_info
type: ref
possible_keys: person_role_id
key: person_role_id
key_len: 5
ref: imdb.char_name.id
rows: 4
Extra: Using index
************* 3. row *************
id: 1
select_type: SIMPLE
table: name
type: eq_ref
possible_keys: PRIMARY
key: PRIMARY
key_len: 4
ref: imdb.cast_info.person_id
rows: 1
Extra: NULL
3 rows in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 101 / 138

 JOINMethods
You need to design queries and indexes to filter as fast as
possible.
MySQL main join method: a nested loop join.
Alternative methods:
Batched Key Access (nested loop join, optimized to avoid
random disk access) − Only in 5.6+, limited usage
Hash joins: Only for equijoins, only in MariaDB

© 2011 - 2016 Percona, Inc. 102 / 138

 Nested-Loop Join Example
Find all actors that were active between 1960 and 1970:
id first_name last_name id title production_year

1 Sean Connery 1 Dr. No 1962

2 George Lazenby From Russia with

2 1963
Love
3 Roger Moore
3 Goldfinger 1964
4 Timothy Dalton
4 You Only Live Twice 1967
5 Pierce Brosnan
5 On Her Majesty's S.S. 1969
6 Daniel Craig
Diamonds Are
6 1971
Forever

© 2011 - 2016 Percona, Inc. 103 / 138

 If That Query is Common
When you can’t filter enough on one table, bring some of the
other filters from the other tables to the first one.
id first_name last_name start_date finish_date

1 Sean Connery 1962 1971

2 George Lazenby 1969 1969

3 Roger Moore 1973 1985

4 Timothy Dalton 1987 1989

5 Pierce Brosnan 1995 2002

6 Daniel Craig 2006 2011

© 2011 - 2016 Percona, Inc. 104 / 138

 STRAIGHT JOIN
Tells the optimizer not to reorder tables; access tables in exactly
the order you gave in the query.
Use it like a query modifier like DISTINCT:
mysql> SELECT STRAIGHT_JOIN name.*
-> FROM char_name INNER JOIN cast_info
-> ON name.id = cast_info.person_role_id INNER JOIN name
-> ON cast_info.person_id = name.id
-> WHERE char_name.name = 'James Bond';

© 2011 - 2016 Percona, Inc. 105 / 138

 Things Are Looking Good?
Please don’t take away that adding indexes is the only secret to
performance.
There’s more to consider:
Optimizer limitations for subqueries.
Estimating index prefix length.
Join methods.
Optimizer hints.
Advanced query profiling.

© 2011 - 2016 Percona, Inc. 106 / 138

 Query Optimization
SUBQUERY OPTIMIZATION

© 2011 - 2016 Percona, Inc. 107 / 138

 Subquery Analysis, < 5.6
mysql> EXPLAIN SELECT * FROM title WHERE kind_id IN
-> (SELECT id FROM kind_type WHERE kind = 'video game')\G

************* 1. row *************

id: 1
select_type: PRIMARY
table: title
type: ALL Will it fix it if we add an index on
possible_keys: NULL title.kind_id?
key: NULL
key_len: NULL
ref: NULL
rows: 1496878
Extra: Using where
************* 2. row *************
id: 2
select_type: SUBQUERY
table: kind_type
type: const
possible_keys: PRIMARY,kind
key: kind
key_len: 47
ref: const
rows: 1
Extra: Using index
2 rows in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 108 / 138

 With Index on kind_id
mysql> ALTER TABLE title ADD INDEX (kind_id);

mysql> EXPLAIN SELECT * FROM title WHERE kind_id IN

-> (SELECT id FROM kind_type WHERE kind = 'video game')\G

************* 1. row *************

id: 1
select_type: PRIMARY
table: title
type: ALL No help! Why is this?
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 1496878
Extra: Using where
************* 2. row *************
id: 2
select_type: SUBQUERY
table: kind_type
type: const
possible_keys: PRIMARY,kind
key: kind
key_len: 47
ref: const
rows: 1
Extra: Using index
2 rows in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 109 / 138

 Scalar Subquery
mysql> EXPLAIN SELECT * FROM title WHERE kind_id =
-> (SELECT id FROM kind_type WHERE kind = 'video game')\G

************* 1. row *************

id: 1
select_type: PRIMARY
table: title
type: ref
possible_keys: kind_id Change to using equality; it works!
key: kind_id
key_len: 4 The optimizer treats scalar subqueries
ref: const
rows: 6649
differently.
Extra: Using where
************* 2. row *************
But it only checks that the subquery is
id: 2 scalar if you use =
select_type: SUBQUERY
table: kind_type
type: const
possible_keys: kind
key: kind
key_len: 47
ref: const
rows: 1
Extra: Using index
2 rows in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 110 / 138

 Solving via JOIN
mysql> EXPLAIN SELECT * FROM title
-> INNER JOIN kind_type ON (title.kind_id = kind_type.id)
-> WHERE kind_type.kind IN ('video game')\G

************* 1. row *************

id: 1
select_type: SIMPLE
table: kind_type
It’s okay to have multiple kind’s
type: const specified using this syntax.
possible_keys: PRIMARY,kind
key: kind
key_len: 47
ref: const
rows: 1
Extra: Using index
************* 2. row *************
id: 1
select_type: SIMPLE
table: title
type: ref
possible_keys: kind_id
key: kind_id
key_len: 4
ref: const
rows: 6649
Extra: NULL
2 rows in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 111 / 138

 Subquery Analysis >= 5.6
In some cases, the optimizer can recognize that the IN clause of
the subquery returns only one row or a set of unique rows.
In this case, the query can be executed as a semi-join.

© 2011 - 2016 Percona, Inc. 112 / 138

 Subquery Analysis >= 5.6 (cont.)
mysql> ALTER TABLE title ADD INDEX (kind_id);

mysql> EXPLAIN SELECT * FROM title WHERE kind_id IN

-> (SELECT id FROM kind_type WHERE kind = 'video game')\G

************* 1. row ************* ************* 1. row *************

id: 1 id: 1
select_type: SIMPLE select_type: SIMPLE
table: kind_type table: kind_type
type: const type: const
possible_keys: PRIMARY,kind possible_keys: PRIMARY,kind
key: kind key: kind
key_len: 47 key_len: 47
ref: const ref: const
rows: 1 rows: 1
Extra: Using index Extra: Using index
************* 2. row ************* ************* 2. row *************
id: 1 id: 1
select_type: SIMPLE select_type: SIMPLE
table: title table: title
type: ALL type: ref
possible_keys: NULL possible_keys: kind_id
key: NULL key: kind_id
key_len: NULL key_len: 4
ref: NULL ref: const
rows: 1496878 rows: 6649
Extra: Using where Extra: NULL
2 rows in set (0.00 sec) 2 rows in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 113 / 138

 Query Optimization
OTHER EXPLAINTECHNIQUES

© 2011 - 2016 Percona, Inc. 114 / 138

 EXPLAIN FORMAT=JSON
mysql> EXPLAIN FORMAT=JSON SELECT * FROM users
WHERE last_name = 'Ascencio' AND first_name = 'Virginia'\G
{
"query_block": {
"select_id": 1,
"table": {
"table_name": "users",
"access_type": "ref",
"possible_keys": ["last_name"],
"key": "last_name",
"used_key_parts": ["last_name"],
"key_length": "102",
"ref": ["const"],
"rows": 2,
"filtered": 100,
"index_condition": "(`imdb`.`users`.`last_name` = 'Ascencio')",
"attached_condition": "(`imdb`.`users`.`first_name` = 'Virginia')"
}
}
}

© 2011 - 2016 Percona, Inc. 115 / 138

 EXPLAIN FORMAT=JSON(cont.)
Contains a lot more detail than regular output.
Blog series:
https://www.percona.com/blog/tag/explain-formatjson-is-cool/

Example:
See which columns in composite index are used:
"used_columns": [
"first_name",
"last_name"
]

© 2011 - 2016 Percona, Inc. 116 / 138

 EXPLAIN FORMAT=JSON- Cost
Since MySQL >= 5.7, cost information is added.
Compare with other execution plans.
Lowest cost is better.
Can indicate optimizer bugs.
{
"cost_info": {"query_cost": "285588.80"},
"ordering_operation": {
"cost_info": {
"read_cost": "5928.00",
"eval_cost": "279660.80",
"prefix_cost": "285588.80",
"data_read_per_join": "330M"
},
}
}

© 2011 - 2016 Percona, Inc. 117 / 138

 EXPLAIN FOR CONNECTION
MySQL 5.7+ Feature
Find a connection ID you are interested in:
mysql> SHOW PROCESSLIST\G
******************** 1. row *******************
Id: 16
User: root
Host: localhost
db: imdb
Command: Query
Time: 14
State: Sending data
Info: select title, count(*) from title
group by title having count(*) > 10
order by count(*) desc limit 1

© 2011 - 2016 Percona, Inc. 118 / 138

 EXPLAIN FOR CONNECTION(cont.)
Then EXPLAINinformation about a running query:
mysql> EXPLAIN FOR CONNECTION 16\G
********************** 1. row **********************
id: 1
select_type: SIMPLE
table: title
partitions: NULL
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 1398304
filtered: 100.00
Extra: Using temporary; Using filesort

© 2011 - 2016 Percona, Inc. 119 / 138

 Query Optimization
BEYOND EXPLAIN

© 2011 - 2016 Percona, Inc. 120 / 138

 The Limitations of EXPLAIN
EXPLAIN shows MySQL's intentions; there is no post-
execution analysis.
How many rows actually had to be sorted?
Was that temporary table created on disk?
Did the LIMIT 10 result in a quick match, resulting in fewer
rows scanned?
... we don't know.

© 2011 - 2016 Percona, Inc. 121 / 138

 Getting more than EXPLAIN
Combine EXPLAIN with other MySQL diagnostics:
SHOW SESSION STATUS
SHOW PROFILES
Slow Query Log Extended Statistics in Percona Server

© 2011 - 2016 Percona, Inc. 122 / 138

 Double Checking
mysql> SHOW STATUS LIKE 'Ha%';
+----------------------------+----------+
| Variable_name | Value |
+----------------------------+----------+
| Handler_commit |0 |
| Handler_delete |0 |
| Handler_discover |0 |
| Handler_prepare |0 |
| Handler_read_first |1 | <-- First entry in index
| Handler_read_key | 13890229 | <-- Read a row based on key
| Handler_read_next | 14286456 | <-- Read the next row
| Handler_read_prev |0 | -- in key order
| Handler_read_rnd |0 |
| Handler_read_rnd_next | 2407004 | <-- Read the next row
| Handler_rollback |0 | -- from disk
| Handler_savepoint |0 |
| Handler_savepoint_rollback | 0 |
| Handler_update |0 |
| Handler_write | 2407001 | <-- Insert row to table
+----------------------------+----------+

© 2011 - 2016 Percona, Inc. 123 / 138

 SHOW PROFILES
Enable profiling:
mysql> SET profiling = 1;

Run some query(s):

mysql> SELECT STRAIGHT_JOIN COUNT(*) AS c, person_id
-> FROM cast_info FORCE INDEX(person_id)
-> INNER JOIN title ON (cast_info.movie_id=title.id)
-> WHERE title.kind_id = 1 GROUP BY cast_info.person_id
-> ORDER by c DESC LIMIT 1;

View the report:

mysql> SHOW PROFILES;
| Query ID | Duration | Query
| 1 | 211.21064300 | SELECT STRAIGHT_JOIN ...

Deprecated Feature; Switch to performance_schema.

© 2011 - 2016 Percona, Inc. 124 / 138
 SHOW PROFILES (cont.)
mysql> show profile for query 1;
+---------------------------+-----------+
| Status | Duration |
+---------------------------+-----------+
| starting | 0.000079 |
| checking permissions | 0.000005 |
| checking permissions | 0.000006 |
| Opening tables | 0.000025 |
| init | 0.000034 |
| System lock | 0.000013 |
| optimizing | 0.000018 |
| statistics | 0.000095 |
| preparing | 0.000029 |
| Creating tmp table | 0.000022 |
| Sorting for group | 0.000012 |
| Sorting result | 0.000004 |
| executing | 0.000004 |
| Sending data | 28.653018 |
| converting HEAP to MyISAM | 0.100713 |
| Sending data | 17.346021 |
| Creating sort index | 0.148820 |
| end | 0.000007 |
| removing tmp table | 0.007086 |
| end | 0.000009 |
| query end | 0.000013 |
| closing tables | 0.000022 |
| freeing items | 0.000263 |
| logging slow query | 0.000007 |
| logging slow query | 0.000005 |
| cleaning up | 0.000052 |
+---------------------------+-----------+
26 rows in set (0.00 sec)

© 2011 - 2016 Percona, Inc. 125 / 138

 Verbose Slow Log in Percona Server
SET GLOBAL long_query_time = 0;
SET GLOBAL log_slow_verbosity = 'full'; /* Percona Server */

# Time: 100924 13:58:47

# User@Host: root[root] @ localhost []
# Thread_id: 10 Schema: imdb Last_errno: 0 Killed: 0
# Query_time: 399.563977 Lock_time: 0.000110 Rows_sent: 1 Rows_examined: 46313608
# Rows_affected: 0 Rows_read: 1
# Bytes_sent: 131 Tmp_tables: 1 Tmp_disk_tables: 1 Tmp_table_sizes: 25194923
# InnoDB_trx_id: 1403
# QC_Hit: No Full_scan: Yes Full_join: No Tmp_table: Yes Tmp_table_on_disk: Yes
# Filesort: Yes Filesort_on_disk: Yes Merge_passes: 5
# InnoDB_IO_r_ops: 1064749 InnoDB_IO_r_bytes: 17444847616 InnoDB_IO_r_wait: 26.935662
# InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000
# InnoDB_pages_distinct: 65329
SET timestamp=1285336727;
select STRAIGHT_JOIN count(*) as c, person_id FROM cast_info FORCE INDEX(person_id)
INNER JOIN title ON (cast_info.movie_id=title.id) WHERE title.kind_id = 1
GROUP BY cast_info.person_id ORDER by c DESC LIMIT 1;

© 2011 - 2016 Percona, Inc. 126 / 138

 Query Planning Overhead
Some queries might spend too much time on the query planning
and optimization phase
There is no query plan cache in MySQL
It can be easily identified when EXPLAINis "slow"
Solution:
Force the plan with STRAIGHT_JOIN, etc.
Reduce optimizer_search_depth variable

© 2011 - 2016 Percona, Inc. 127 / 138

 Index Merge Optimization
Index Merge access type allows the use of more than one index
per table access:
Union of several conditions
SELECT * FROM title where title = 'Bambi' or production_year = 1927

Intersection
SELECT * FROM title where title = 'Bambi' and production_year = 1927

© 2011 - 2016 Percona, Inc. 128 / 138

 The Merge Access Problem
In both cases, there is usually faster alternative query plans:
Union: UNION clauses, single index usage, ...
Intersection: composite indexes, secondary index
extensions, single index usage, ...
MySQL merge algorithm is selected in most cases even if those
other methods were available and slower
MySQL 5.6+ fixes this, although not in all cases

© 2011 - 2016 Percona, Inc. 129 / 138

 Intersection Merge Fixes
Drop the single-column indexes (if not used for other queries)
and create a composite index with all columns
Even if the condition cannot be applied, a single column
index or Index Condition Pushdown will be probably faster
You can disable the merge intersection with:
SET optimizer_switch = 'index_merge_intersection = OFF'

© 2011 - 2016 Percona, Inc. 130 / 138

 Union Merge
Union Merge may or may not be faster than other methods
In 5.6+, ref or range over composite indexes should have
higher preference
The alternative would be converting the clause to a
UNION/UNION ALL
UNION is not always better as it has a problem: for most
cases it will create a temporary table on disk

© 2011 - 2016 Percona, Inc. 131 / 138

 Example Union Merge
SELECT id FROM title WHERE (title = 'Pilot' or episode_nr = 1)
AND production_year > 1977;

is slower than the equivalent:

SELECT id FROM title WHERE (title ='Pilot' and production_year > 1977)
UNION
SELECT id FROM title WHERE (episode_nr = 1 and production_year > 1977);

© 2011 - 2016 Percona, Inc. 132 / 138

 More Features and Workarounds
“Delayed Join”
http://www.percona.com/blog/2007/04/06/using-delayed-
join-to-optimize-count-and-limit-queries/
The IN() list workaround
http://www.percona.com/blog/2010/01/09/getting-around-
optimizer-limitations-with-an-in-list/

© 2011 - 2016 Percona, Inc. 133 / 138

 Server-side query rewrite framework (5.7+)
Write your own plugin functions for query rewriting.
http://mysqlserverteam.com/write-yourself-a-query-rewrite-plugin-part-1/
http://mysqlserverteam.com/write-yourself-a-query-rewrite-plugin-part-2/
http://dev.mysql.com/doc/refman/5.7/en/writing-plugins.html

© 2011 - 2016 Percona, Inc. 134 / 138

 Rewriter
A MySQL-included plugin to rewrite queries.
Enable/Disable:

mysql> SHOW GLOBAL VARIABLES LIKE 'rewriter_enabled';

+------------------+-------+
| Variable_name | Value |
+------------------+-------+
| rewriter_enabled | ON |
+------------------+-------+
mysql> SET GLOBAL rewriter_enabled = OFF;
mysql> SET GLOBAL rewriter_enabled = ON;

© 2011 - 2016 Percona, Inc. 135 / 138

 Rewriter(cont.)
A slow running query:
SELECT name.id, name.name FROM imdb.name
LEFT JOIN imdb.aka_name ON name.id = aka_name.person_id
WHERE name.name = '25 Cent';

Create a filter:
INSERT INTO query_rewrite.rewrite_rules (pattern, replacement)
VALUES (
"SELECT name.id, name.name FROM imdb.name
LEFT JOIN imdb.aka_name ON name.id = aka_name.person_id
WHERE name.name = ?",
"SELECT name.id, name.name FROM imdb.name
WHERE name.name = ?");

CALL query_rewrite.flush_rewrite_rules();

© 2011 - 2016 Percona, Inc. 136 / 138

 Additional Exercises
SELECT * FROM movie_info
WHERE movie_id IN (SELECT id FROM title WHERE title = 'Batman Begins');

SELECT * FROM title WHERE season_nr != 6 AND title LIKE 'Best of%';

SELECT t.title
FROM name n
INNER JOIN cast_info i ON n.id = i.person_id
INNER JOIN title t ON i.movie_id = t.id
INNER JOIN char_name c ON c.id = i.person_role_id
WHERE n.name = 'Brosnan, Pierce'
AND t.kind_id = 1
AND c.name='James Bond';

http://bit.ly/220ln2V

© 2011 - 2016 Percona, Inc. 137 / 138

 Let's Take a Break!

© 2011 - 2016 Percona, Inc. 138 / 138