Explaining The Postgres Query Optimizer
Explaining The Postgres Query Optimizer
BRUCE MOMJIAN
1 / 56
Postgres Query Execution
User
Terminal
PostgreSQL
Application Database
Code
Server
Libpq
Queries
Results
2 / 56
Postgres Query Execution
Main
Libpq
Postmaster
Postgres Postgres
Parse Statement
utility Utility
Traffic Cop
Command
Query e.g. CREATE TABLE, COPY
SELECT, INSERT, UPDATE, DELETE
Rewrite Query
Generate Paths
Optimal Path
Generate Plan
Plan
Execute Plan
3 / 56
Postgres Query Execution
Parse Statement
utility Utility
Traffic Cop
Command
Query e.g. CREATE TABLE, COPY
SELECT, INSERT, UPDATE, DELETE
Rewrite Query
Generate Paths
Optimal Path
Generate Plan
Plan
Execute Plan
4 / 56
The Optimizer Is the Brain
https://www.flickr.com/photos/dierkschaefer/
5 / 56
What Decisions Does the Optimizer Have to Make?
◮ Scan Method
◮ Join Method
◮ Join Order
6 / 56
Which Scan Method?
◮ Sequential Scan
◮ Bitmap Index Scan
◮ Index Scan
7 / 56
A Simple Example Using pg_class.relname
SELECT relname
FROM pg_class
ORDER BY 1
LIMIT 8;
relname
-----------------------------------
_pg_foreign_data_wrappers
_pg_foreign_servers
_pg_user_mappings
administrable_role_authorizations
applicable_roles
attributes
check_constraint_routine_usage
check_constraints
8 / 56
Let’s Use Just the First Letter of pg_class.relname
SELECT substring(relname, 1, 1)
FROM pg_class
ORDER BY 1
LIMIT 8;
substring
-----------
_
_
_
a
a
a
c
c
9 / 56
Create a Temporary Table with an Index
10 / 56
Create an EXPLAIN Function
11 / 56
What is the Distribution of the sample Table?
12 / 56
What is the Distribution of the sample Table?
letter | count | %
--------+-------+------
p | 199 | 78.7
s | 9 | 3.6
c | 8 | 3.2
r | 7 | 2.8
t | 5 | 2.0
v | 4 | 1.6
f | 4 | 1.6
d | 4 | 1.6
u | 3 | 1.2
a | 3 | 1.2
_ | 3 | 1.2
e | 2 | 0.8
i | 1 | 0.4
k | 1 | 0.4
13 / 56
Is the Distribution Important?
14 / 56
Is the Distribution Important?
15 / 56
Is the Distribution Important?
16 / 56
Running ANALYZE Causes
a Sequential Scan for a Common Value
ANALYZE sample;
17 / 56
Sequential Scan
Heap
D D D D D D D D D D D D
A A A A A A A A A A A A
T T T T T T T T T T T T
A A A A A A A A A A A A
8K
18 / 56
A Less Common Value Causes a Bitmap Index Scan
19 / 56
Bitmap Index Scan
1 1 1
& =
0 1 0
1 0 0
20 / 56
An Even Rarer Value Causes an Index Scan
21 / 56
Index Scan
Heap
D D D D D D D D D D D D
A A A A A A A A A A A A
T T T T T T T T T T T T
A A A A A A A A A A A A
22 / 56
Let’s Look at All Values and their Effects
l | count | lookup_letter
---+-------+-----------------------------------------------------------------------
p | 199 | Seq Scan on sample (cost=0.00..13.16 rows=199 width=2)
p | 199 | Filter: (letter = ’p’::text)
s | 9 | Seq Scan on sample (cost=0.00..13.16 rows=9 width=2)
s | 9 | Filter: (letter = ’s’::text)
c | 8 | Seq Scan on sample (cost=0.00..13.16 rows=8 width=2)
c | 8 | Filter: (letter = ’c’::text)
r | 7 | Seq Scan on sample (cost=0.00..13.16 rows=7 width=2)
r | 7 | Filter: (letter = ’r’::text)
…
23 / 56
OK, Just the First Lines
24 / 56
Just the First EXPLAIN Lines
l | count | lookup_letter
---+-------+-----------------------------------------------------------------------
p | 199 | Seq Scan on sample (cost=0.00..13.16 rows=199 width=2)
s | 9 | Seq Scan on sample (cost=0.00..13.16 rows=9 width=2)
c | 8 | Seq Scan on sample (cost=0.00..13.16 rows=8 width=2)
r | 7 | Seq Scan on sample (cost=0.00..13.16 rows=7 width=2)
t | 5 | Bitmap Heap Scan on sample (cost=4.29..12.76 rows=5 width=2)
f | 4 | Bitmap Heap Scan on sample (cost=4.28..12.74 rows=4 width=2)
v | 4 | Bitmap Heap Scan on sample (cost=4.28..12.74 rows=4 width=2)
d | 4 | Bitmap Heap Scan on sample (cost=4.28..12.74 rows=4 width=2)
a | 3 | Bitmap Heap Scan on sample (cost=4.27..11.38 rows=3 width=2)
_ | 3 | Bitmap Heap Scan on sample (cost=4.27..11.38 rows=3 width=2)
u | 3 | Bitmap Heap Scan on sample (cost=4.27..11.38 rows=3 width=2)
e | 2 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)
i | 1 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)
k | 1 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)
25 / 56
We Can Force an Index Scan
26 / 56
Notice the High Cost for Common Values
l | count | lookup_letter
---+-------+-----------------------------------------------------------------------
p | 199 | Index Scan using i_sample on sample (cost=0.00..39.33 rows=199 width=
s | 9 | Index Scan using i_sample on sample (cost=0.00..22.14 rows=9 width=2)
c | 8 | Index Scan using i_sample on sample (cost=0.00..19.84 rows=8 width=2)
r | 7 | Index Scan using i_sample on sample (cost=0.00..19.82 rows=7 width=2)
t | 5 | Index Scan using i_sample on sample (cost=0.00..15.21 rows=5 width=2)
d | 4 | Index Scan using i_sample on sample (cost=0.00..15.19 rows=4 width=2)
v | 4 | Index Scan using i_sample on sample (cost=0.00..15.19 rows=4 width=2)
f | 4 | Index Scan using i_sample on sample (cost=0.00..15.19 rows=4 width=2)
_ | 3 | Index Scan using i_sample on sample (cost=0.00..12.88 rows=3 width=2)
a | 3 | Index Scan using i_sample on sample (cost=0.00..12.88 rows=3 width=2)
u | 3 | Index Scan using i_sample on sample (cost=0.00..12.88 rows=3 width=2)
e | 2 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)
i | 1 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)
k | 1 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)
RESET ALL;
27 / 56
This Was the Optimizer’s Preference
l | count | lookup_letter
---+-------+-----------------------------------------------------------------------
p | 199 | Seq Scan on sample (cost=0.00..13.16 rows=199 width=2)
s | 9 | Seq Scan on sample (cost=0.00..13.16 rows=9 width=2)
c | 8 | Seq Scan on sample (cost=0.00..13.16 rows=8 width=2)
r | 7 | Seq Scan on sample (cost=0.00..13.16 rows=7 width=2)
t | 5 | Bitmap Heap Scan on sample (cost=4.29..12.76 rows=5 width=2)
f | 4 | Bitmap Heap Scan on sample (cost=4.28..12.74 rows=4 width=2)
v | 4 | Bitmap Heap Scan on sample (cost=4.28..12.74 rows=4 width=2)
d | 4 | Bitmap Heap Scan on sample (cost=4.28..12.74 rows=4 width=2)
a | 3 | Bitmap Heap Scan on sample (cost=4.27..11.38 rows=3 width=2)
_ | 3 | Bitmap Heap Scan on sample (cost=4.27..11.38 rows=3 width=2)
u | 3 | Bitmap Heap Scan on sample (cost=4.27..11.38 rows=3 width=2)
e | 2 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)
i | 1 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)
k | 1 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)
28 / 56
Which Join Method?
◮ Nested Loop
◮ With Inner Sequential Scan
◮ With Inner Index Scan
◮ Hash Join
◮ Merge Join
29 / 56
What Is in pg_proc.oid?
SELECT oid
FROM pg_proc
ORDER BY 1
LIMIT 8;
oid
-----
31
33
34
35
38
39
40
41
30 / 56
Create Temporary Tables
from pg_proc and pg_class
31 / 56
Join the Two Tables
with a Tight Restriction
32 / 56
Nested Loop Join
with Inner Sequential Scan
Outer Inner
aag aai
aay aag
aar aas
aai aar
aay
aaa
aag
No Setup Required
34 / 56
Join the Two Tables with a Looser Restriction
35 / 56
Hash Join
Outer Inner
36 / 56
Pseudocode for Hash Join
37 / 56
Join the Two Tables with No Restriction
38 / 56
Merge Join
Outer Inner
aaa aaa
39 / 56
Pseudocode for Merge Join
sort(outer);
sort(inner);
i = 0;
j = 0;
save_j = 0;
while (i < length(outer))
if (outer[i] == inner[j])
output(outer[i], inner[j]);
if (outer[i] <= inner[j] && j < length(inner))
j++;
if (outer[i] < inner[j])
save_j = j;
else
i++;
j = save_j;
40 / 56
Order of Joined Relations Is Insignificant
41 / 56
Add Optimizer Statistics
ANALYZE sample1;
ANALYZE sample2;
42 / 56
This Was a Merge Join without Optimizer Statistics
43 / 56
Outer Joins Can Affect Optimizer Join Usage
44 / 56
Cross Joins Are Nested Loop Joins
without Join Restriction
45 / 56
Create Indexes
46 / 56
Nested Loop with Inner Index Scan Now Possible
47 / 56
Nested Loop Join with Inner Index Scan
Outer Inner
aag aai
aay aag
aar aas
aai aar
aay
aaa
Index Lookup
aag
No Setup Required
48 / 56
Pseudocode for Nested Loop Join
with Inner Index Scan
49 / 56
Query Restrictions Affect Join Usage
50 / 56
All ’junk’ Columns Begin with ’xxx’
Hash join was chosen because many more rows are expected. The smaller
table, e.g., sample2, is always hashed.
51 / 56
Without LIMIT, Hash Is Used
for this Unrestricted Join
52 / 56
LIMIT Can Affect Join Usage
53 / 56
LIMIT 10
54 / 56
LIMIT 100 Switches to Hash Join
55 / 56
Conclusion
http://momjian.us/presentations https://www.flickr.com/photos/trevorklatko/
56 / 56