Chapter 15: Query Processing

Dario Della Monica
Chapter 15: Query Processing
These slides are a modified version of the slides provided with the book:
Database System Concepts, 6th Ed.

©Silberschatz, Korth and Sudarshan
See www.db-book.com for conditions on re-use
(however, chapter numeration refers to 7th Ed.)
The original version of the slides is available at: https://www.db-book.com/

Chapter 15: Query Processing
 Overview
 How to measure query costs
 Establishing a cost model
 Algorithms for evaluating relational
algebra operations (cost estimates)
 Selection
 Sorting
 Join
 Evaluation of Expressions
(How to combine algorithms for individual
operations in order to evaluate a complex
expression) Silberschatz, Korth, Sudarshan,
Database System Concepts,
 Materialization 7° edition, 2011
 Pipelining
Database System Concepts - 7th Edition 15.2 ©Silberschatz, Korth and Sudarshan
Basic Steps in Query Processing
1. Parsing and translation We mainly focus on
the optimization phase
2. Optimization
3. Evaluation
Basic Steps in Query Processing (cont.)
 Parser and translator
 Translate the (SQL) query into relational algebra
 Parser checks syntax (e.g., correct relation and operator names)
 Evaluation engine
 The query-execution engine takes a query-evaluation plan, executes
that plan, and returns the answers to the query
 Optimizer (in a nutshell – more details in the next slides)

 Chooses the most efficient implementation to execute the query
 Produces equivalent relational algebra expressions
 Annotates them with instructions (algorithms): query execution plan (QEP)
 Estimates the cost of each equivalent QEP, according to a given cost model
 Choose the “best” QEP
Basic Steps: Optimization
 Input of optimization: a query in the form of an algebra expression
 Output of optimization: the “best” annotated relational algebra expression

specifying detailed evaluation strategy (query evaluation plan or query
execution plan – QEP) answering the input query
 1st level of optimization: order of operations – an SQL query has many
equivalent relational algebra expressions
 Consider the SQL query SELECT salary
FROM instructor
WHERE salary < 75000
 salary75000(salary(instructor)) and salary(salary75000(instructor))
are equivalent and they both correspond to the above query
 2nd level of optimization: implementation of operations – a relational

algebra operation can be evaluated via several different algorithms
 e.g., block nested-loop join VS. merge-join; file scan VS. index scan
Basic Steps: Optimization (Cont.)
 Different query evaluation plans have different costs
 User is not expected to specify least-cost plans
 Query Optimization: amongst all equivalent QEP choose the one

with lowest cost
 Cost is estimated using statistical information from the database catalog
 # of tuples in relations, tuple sizes, # of distinct values for a given attribute, etc.
 We study… (Chapter 15⋆ – evaluation of QEP)

 How to measure query costs (establish a cost model)
 Algorithms for evaluating relational algebra operations and their cost
 How to combine algorithms for individual operations in order to evaluate a
complex expression (QEP)
 … and (Chapter 16⋆ – choosing the best QEP)
 How to optimize queries, that is, how to find a QEP with lowest estimated
cost
⋆
Silberschatz, Korth, and Sudarshan, Database System Concepts, 7° ed.
How to measure query costs
(cost model)


Measures of Query Cost
Response time (wall-clock time needed to execute a plan) depends on several factors
 system configuration
 amount of dedicated buffer in RAM (aka, memory, main memory)
 whether or not indices are always available in the buffer
 runtime conditions
 amount of free buffer at the time the plan is executed
 content of the buffer at the time the plan is executed
 parameters, embedded in queries, which are resolved at runtime only
SELECT salary
FROM instructor
WHERE salary < $a
where $a is a variable provided by the application (user)
Thus
1. cost models (like ours) focus on resource consumption rather than response time
(optimizers minimize resource consumption rather than response time)
2. different optimizers may make different assumptions (parameters): every theoretical
analysis must be recast with the actual parameters used by the concrete system
(optimizer) to which the analysis is going to be applied
Measures of Query Cost (Cont.)
 Query cost (total elapsed time for answering a query) is measured in terms of
different resources
 disk access (I/O operation on disk)
 CPU usage
 (network communication for distributed DBMS – later in this course)
 Typically disk access is the predominant cost, and is also relatively easy to
estimate. Measured by taking into account
 Number of seeks (number of random I/O accesses)
 Number of blocks read
 Number of blocks written
 It is generally assumed cost for writing to be twice as the cost for reading
(data is read back after being written to ensure the write was successful)
VERY IMPORTANT!!!
- “disk” refers to permanent drive for file storage, hard-disk, secondary memory, permanent memory
- “memory” refers to volatile drive for data storage, RAM, main memory, buffer
These are all used as synonims
This is a so far accepted choice for measuring query costs (cost model).
New technologies: faster hard-disks (solid-state drives – SSD) and cheaper (thus bigger) RAM
might direct towards different cost models (e.g., based also on CPU usage or RAM I/O operations)
Measures of Query Cost (Cont.)
 We ignore difference between writing and reading: we just consider
 tS – time for one seek
 tT – time to transfer one block
 Example: cost for b block transfers plus S seeks
b * tT + S * t S
 Values of tT and tS must be calibrated for the specific disk system
 Typical values (2018): tS = 4 ms, tT = 0.1 ms
 Some DBMS performs, during installation, seeks and block transfers to
estimate average values
 We ignore CPU costs for simplicity
 Real systems usually do take CPU cost into account
 We do not include cost to writing output to disk in our cost formulae
Algorithms for evaluating relational
algebra operations


Physical organization of records
 At the physical level, records are stored (on permanent disks) in files
(managed and organized by the filesystem)
 We assume files are organized according to sequential file
organization
 i.e., a file is stored in contiguous blocks, with records ordered according
to some attribute(s) – not necessarily ordered by primary key
 Other file organization techniques exist (e.g., B+-tree file

organization), leading to different formulas for cost estimate
Selection Operation
 File scan (relation scan without indices)
PROs: can be applied to any file, regardless of its ordering, availability of indices,
nature of selection operation, etc.
CONs: it is slow
 Algorithm A1 (linear search). Retrieve and scan each file block and
test all records to see whether they satisfy the selection condition
 br denotes number of blocks containing records from relation r
 Cost estimate??? (selection on a generic, non-key attribute)
 cost = br block transfers + 1 seek = tS + br * tT
We assume blocks are stored contiguously so 1 seek operation is enough (disk head
does not need to move to seek next block)
 Selection on a key attribute. Cost estimate???

 stop on finding record
 cost = (br /2) block transfers + 1 seek = tS + (br / 2)* tT
Selections Using Indices
⋆
Silberschatz, Korth, and Sudarshan, Database System Concepts, 6° ed.
Selections Using Indices
 A4 (secondary index, equality on key). Cost?

 Equal to A2
 cost = (hi + 1) * (tT + tS)
 A4 (secondary index, equality on nonkey)

 Retrieve multiple records. Cost?
 in the worst case, for each of n matching records a new
block access (both transfer and seek) is needed (even if 2
records are in the same block but pointed by leaves that
are far apart in the index tree)
 Cost = (hi + n) * (tT + tS)
– Can be very expensive! Can be worse than file scan
Selections Involving Comparisons
 Can implement selections of the form AV (r) or A  V(r) by using

 a linear file scan,
 or by using indices in the following ways:
 A5 (primary index, comparison).
 A  V(r)
 use index to find first tuple  v and scan relation sequentially from there
 RECALL: b is the number of blocks containing matching records
 Equal to A3: Cost = hi * (tT + tS) + tS + tT * b
 AV(r)
 just scan relation sequentially till first tuple > v; do not use the index
 Similar to A1 (file scan, equality on key): Cost = tS + b* tT
Selections Involving Comparisons
 A6 (secondary index, comparison). Cost?

 A  V(r)
 use index to find first index entry  v and scan index sequentially from there, to find
pointers to records, and finally retrieve records that are pointed to
 requires an I/O for each record
 Equal to A4, equality on nonkey: cost = (hi + n) * (tT + tS)
 Linear file scan may be cheaper
 AV (r)
 just scan leaf pages of index from the beginning up to the first entry > v, finding
pointers to records, and finally retrieve records that are pointed to
 equal to the above case
Summary of costs for selections
Complex Selections
conjunctions, disjunctions, and negation of simple conditions
 A7 (conjunctive selection using 1 index)
 θ1 AND θ2 AND … AND θn
 If there is at least 1 index useful for 1 simple condition θi, then
 use the right algorithm among A2-A6 to retrieve tuple satisfying θi
 and in the meantime check for the other simple conditions on records selected .
 Cost?
 Cost is given by the cost of chosen algorithm for the chosen condition
 cost depend on the choice of the condition (and the choice of the algorithm)
 example: id=x AND dept=y (teacher)
 primary index over id and secondary index over dept
 id is primary key
 it is convenient to choose id=x (with algorithm A2)
 A8 (conjunctive selection using composite index)
 use a composite index over attributes involved in all or some of the simple conditions
(it only works with equivalences), if any
 example: name=x AND dept=y (teacher)
 use composite index over pair (name,dept) with algorithm A4 (secondary index, equality on non-key)
Complex Selections (cont’d)
 A9 (conjunctive selection by intersection of identifiers)
 Recall that indices we consider have pointers to records (rather than actual records)
 Scan indices but do not access records, just collect sets of pointers (one per index)
 Compute the intersection, and then access records (select only the tuples that
match also the conditions for which there is no index). Cost?
 Cost: cost of scanning indices plus cost of accessing blocks with matching records
(some block can be accessed more than once if pointers are not sorted)
 Optimization: order records in the intersection and then access them in sorted order.
Advantages:
 less seeks and transfers: no block is accessed twice (2 records in the same block are retrieved together)
 also, some additional seek time is saved as blocks are transferred in sorted order (disk-arm is minimized)
 A10 (disjunctive selection by union of identifiers)
 If ALL conditions can be checked through some index, then similar to A9
 Scan indices but do not access records, just collect sets of pointers (one per index)
 Compute the union, and then access records (in sorted order)
 Cost: cost of scanning all indices plus cost of accessing records
 If even only 1 condition has no associate index, then A1 (linear scan)
2 more things on selections
1. Negation of a simple condition

 NOT (Attr < v) is equivalent to Attr >= v NOT (Attr > v) is equivalent to Attr <= v
 NOT (Attr <= v) is equivalent to Attr > v NOT (Attr >= v) is equivalent to Attr < v
 NOT (Attr = v) is equivalent (Attr < v) OR (Attr > v) – (probably a linear file scan is needed)
Negation of a complex condition can be “pushed inside” the expressionn
 NOT (c1 AND c2) is equivalent to (NOT c1) OR (NOT c2)
2. A4, A6, A9, A10 are very inefficient (possibly worse than linear scan) due to some
blocks possibly accessed more than once
 few records to be retrieved/no block is accessed more than once: better to use index scan,
a lot of records to be retrieved/several blocks accessed more than once: better to use liner scan
 Solution: collect and sort pointers before accessing records (see optimization for A9)
 Improved solution, based on bitmap structure: bitmap is a vector of bits (as many as number of blocks
used by the relation)
 visit index without accessing records: in the bitmap set to 1 the bits corresponding to blocks to be accessed
 linear scan guided by bitmap (sorted order access thanks to bitmap without actually performing a sorting)
 hybrid solution (mix between linear scan and index access)
 no block accessed more than once : slightly worse than index access,
all blocks containing at least one matching condition: slightly worse than liner scan
 thus, the cost is slightly worse than the optimal plan (linear or index scan): good compromise
Sorting
 Reasons for sorting
 Explicitly requested by SQL query
 SELECT …
FROM …
SORT BY …
 Needed to efficient executions of join operations
 We may build an index on the relation, and then use the index
to read the relation in sorted order. May lead to one disk block
access for each tuple
 For relations that fit in memory, standard sorting techniques
like quick-sort can be used. For relations that don’t fit in
memory, external sort-merge algorithm is a good choice
External Sort-Merge
Let M denote number of blocks that can fit in memory.
1. Create sorted runs (files containing sorted pieces of relation)
Let i be 0 initially.
Repeatedly do the following till the end of the relation:
(a) Read M blocks of relation into memory
(b) Sort the in-memory blocks
(c) Write sorted data to run Ri
(d) Increment i
Let the final value of i be N (number of runs)
2. Merge the runs (next slide)…..
External Sort-Merge (Cont.)
2. Merge the runs (N-way merge). We assume (for now) that N < M.
1. Use N blocks of memory to buffer input runs, and 1 block to

buffer output. Read the first block of each run into its buffer
page
2. repeat
1. Select the first record (in sort order) among the N blocks
for the runs
2. Write the record to the output buffer. If the output buffer is
full write it to disk.
3. Delete the record from its input buffer block.
If the buffer block becomes empty then
transfer the next block (if any) of the run into the buffer.
until all blocks of all runs are processed
External Sort-Merge (Cont.)
 If N  M, several merge passes are required.

 In each pass, contiguous groups of M - 1 runs are merged.
 A pass reduces the number of runs by a factor of M -1 (and
creates runs longer by the same factor)
 E.g. If M=11, and there are 90 runs, one pass merge
together groups of 10 runs into 9 new runs
Thus, one pass reduces the number of runs to 9, each 10
times the size of the initial runs
 Repeated passes are performed till all runs have been
merged into one.
Example: External Sorting Using Sort-Merge
M=3
Join Operation
 Several different algorithms to implement joins
 Nested-loop join
 Block nested-loop join
 Indexed nested-loop join
 Merge-join
 Hash-join
 Choice based on cost estimate
 Running example : students takes
where
 Number of records of student: 5,000
 Number of blocks of student: 100
 Number of records of takes: 10,000
 Number of blocks of takes: 400
Nested-Loop Join
 To compute the theta join r  s
for each tuple tr in r do

for each tuple ts in s do
test pair (tr,ts) to see if they satisfy the join condition 
if it does, add tr • ts to the result
end
end
 r is called the outer relation and s the inner relation of the join
 Requires no indices and can be used with any kind of join
condition
 Expensive since it examines every pair of tuples in the two
relations
Nested-Loop Join (Cont.)
 If the smaller relation fits entirely in memory, use that as the inner relation
 br + bs block transfers and 2 seeks
(same cost in the best case scenario, when both relations fit in memory)
 Worst case (there is enough memory forn only one block for each relation)
# of block transfer:
nr  bs + br (br transfers to read relation r + nrbs transfers to read s for each tuple in r)
# of seeks:
nr + b r (br seeks to read relation r + nr seeks to read s for each tuple in r)
 Running example (join between students and takes – assuming worst case memory availability)
 with student as outer relation:
 5,000  400 + 100 = 2,000,100 block transfers
 5,000 + 100 = 5,100 seeks
 assuming t = 40 * t we have a total of: 2,204,100 * t
S T, T
 with takes as the outer relation
 10,000  100 + 400 = 1,000,400 block transfers
 10,000 + 400 = 10,400 seeks
 assuming t = 40 * t we have a total of: 1,416,400 * t
S T, T
 if smaller relation (student) fits entirely in memory, the cost estimate will be 500 block transfers and 2 seeks
 Block nested-loops algorithm (next slide) is preferable
Block Nested-Loop Join
 Variant of nested-loop join in which every block of inner
relation is paired with every block of outer relation.
for each block Br of r do
for each block Bs of s do
for each tuple tr in Br do
for each tuple ts in Bs do
check if (tr,ts) satisfy the join condition
if it does, add tr • ts to the result
end
end
end
end
Indexed Nested-Loop Join
 If an index is available for one of the relations on the attribute of the join condition
 then use such a relation as inner relation in a nested-loop join
 Instead of doing a linear scan as inner loop, do an index scan
for each tuple tr in r do

index scan over s to find tuples ts satisfying the join condition with tuple tr end
(basically, for every tuple in r, do a selection on s using the index)
 It might be convenient to create an ad-hoc index for the join if it does not exist
 Cost (worst case: space in memory for only 1 block for each relation)
 br seeks and block transfers to read r: br * (tT + tS)
 for each record in r, index scan on s: nr * c (where c is the cost of index scan on s)
 thus, total cost = br * (tT + tS) + nr * c
 NOTICE: if there are indexes for both relations, then it is often better to use relation with
less records as outer relation
How to combine algorithms for
individual operations in order to
evaluate a complex expression


Evaluation of Expressions
 So far: we have seen algorithms for individual operations
 Alternatives for evaluating an entire expression tree
 Materialization: store (materialize) on disk results of
evaluation of sub-expressions into temporary relations for
subsequent use
 Disadvantage: several disk writing and reading to store
temporary relations
 Always possible
 Pipelining: pass on tuples to parent operations as they are
generated by inner operations being executed
 Advantage: less disk writing
 Not always possible
Materialization
 Materialized evaluation: evaluate one operation at a time,
starting at the lowest-level. Use intermediate results materialized
into temporary relations to evaluate next-level operations
E.g.,  name
( building "Watson " (department) instructor)
1. compute and store on disk

 building "Watson " (department )
2. then compute and store on disk its join with instructor
3. finally, compute the projection on name
Pipelining
 Pipelined evaluation: evaluate several operations simultaneously, passing
(partial) results of one operation on to the next as they are generated (es.,
single records), without writing them on disk
 E.g., in previous expression tree, don’t store result of
 building"Watson " (department )

Instead, pass tuples directly to the join as they are found
Similarly, don’t store result of join, pass tuples directly to projection as they are
generated
 Cheaper than materialization: no need to store a temporary relation to disk
 (partial) Results are output earlier, before waiting for complete query
execution
 Parallelization of operations
 Pipelining may not always be possible
 indexed nested-loop join cannot have its input inner relation pipelined as the
whole relation with associated index must be available
 some sorting algorithms cannot output tuples early, only after all input tuples have
been examined (the curious case of the External Sort-Merge)
End of Chapter


Chapter 15: Query Processing

Uploaded by

Copyright:

Available Formats

Chapter 15: Query Processing

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Chapter 15: Query Processing

Uploaded by

Copyright:

Available Formats

Dario Della Monica

Chapter 15: Query Processing

Database System Concepts, 6th Ed.

(however, chapter numeration refers to 7th Ed.)

The original version of the slides is available at: https://www.db-book.com/

 Optimizer (in a nutshell – more details in the next slides)

 Output of optimization: the “best” annotated relational algebra expression

 2nd level of optimization: implementation of operations – a relational

 Query Optimization: amongst all equivalent QEP choose the one

 We study… (Chapter 15⋆ – evaluation of QEP)

Database System Concepts, 6th Ed.

(however, chapter numeration refers to 7th Ed.)

The original version of the slides is available at: https://www.db-book.com/

Database System Concepts, 6th Ed.

(however, chapter numeration refers to 7th Ed.)

The original version of the slides is available at: https://www.db-book.com/

 Other file organization techniques exist (e.g., B+-tree file

 Selection on a key attribute. Cost estimate???

 A4 (secondary index, equality on key). Cost?

 A4 (secondary index, equality on nonkey)

 Can implement selections of the form AV (r) or A  V(r) by using

 A6 (secondary index, comparison). Cost?

1. Negation of a simple condition

1. Create sorted runs (files containing sorted pieces of relation)

2. Merge the runs (next slide)…..

1. Use N blocks of memory to buffer input runs, and 1 block to

 If N  M, several merge passes are required.

 To compute the theta join r  s

for each tuple tr in r do

 Block nested-loops algorithm (next slide) is preferable

for each tuple tr in r do

Database System Concepts, 6th Ed.

(however, chapter numeration refers to 7th Ed.)

The original version of the slides is available at: https://www.db-book.com/

1. compute and store on disk

3. finally, compute the projection on name

 building"Watson " (department )

Database System Concepts, 6th Ed.

(however, chapter numeration refers to 7th Ed.)

The original version of the slides is available at: https://www.db-book.com/

You might also like