Lecture 3

Chapter 13: Query Processing

Database System Concepts, 6th Ed.

Silberschatz, Korth and Sudarshan See www.db-book.com for conditions on re-use

Chapter 13: Query Processing

Overview

Measures of Query Cost

Selection Operation Sorting

Join Operation
Other Operations Evaluation of Expressions

Database System Concepts - 6th Edition

12.2

Silberschatz, Korth and Sudarshan

Basic Steps in Query Processing

1. Parsing and translation 2. Optimization 3. Evaluation

Database System Concepts - 6th Edition

12.3

Silberschatz, Korth and Sudarshan

Basic Steps in Query Processing (Cont.)

Parsing and translation

translate the query into its internal form. This is then translated into relational algebra.

Parser checks syntax, verifies relations

The query-execution engine takes a query-evaluation plan, executes that plan, and returns the answers to the query.

Evaluation

Database System Concepts - 6th Edition

12.4

Silberschatz, Korth and Sudarshan

Basic Steps in Query Processing : Optimization

A relational algebra expression may have many equivalent

expressions

E.g., salary75000(salary(instructor)) is equivalent to salary(salary75000(instructor))

Each relational algebra operation can be evaluated using one of

several different algorithms

Correspondingly, a relational-algebra expression can be evaluated in many ways.

Annotated expression specifying detailed evaluation strategy is

called an evaluation-plan.

E.g., can use an index on salary to find instructors with salary < 75000, or can perform complete relation scan and discard instructors with salary 75000
12.5 Silberschatz, Korth and Sudarshan

Database System Concepts - 6th Edition

Basic Steps: Optimization (Cont.)

Query Optimization: Amongst all equivalent evaluation plans

choose the one with lowest cost.

Cost is estimated using statistical information from the database catalog

e.g.

number of tuples in each relation, size of tuples, etc.

You have to learn

How to measure query costs Algorithms for evaluating relational algebra operations

How to combine algorithms for individual operations in order to evaluate a complete expression
We study how to optimize queries, that is, how to find an evaluation plan with lowest estimated cost

Next lecture

Database System Concepts - 6th Edition

12.6

Silberschatz, Korth and Sudarshan

Measures of Query Cost

Cost is generally measured as total elapsed time for answering

query

Many factors contribute to time cost

disk

accesses, CPU, or even network communication

Typically disk access is the predominant cost, and is also

relatively easy to estimate. Measured by taking into account

Number of seeks Number of blocks read

Cost

* average-seek-cost * average-block-read-cost

Number of blocks written * average-block-write-cost

to write a block is greater than cost to read a block data is read back after being written to ensure that the write was successful

Database System Concepts - 6th Edition

12.7

Silberschatz, Korth and Sudarshan

Measures of Query Cost (Cont.)

For simplicity we just use the number of block transfers from disk

and the number of seeks as the cost measures

tT time to transfer one block

tS time for one seek

Cost for b block transfers plus S seeks b * tT + S * tS Real systems do take CPU cost into account

We ignore CPU costs for simplicity

We do not include cost to writing output to disk in our cost formulae

Database System Concepts - 6th Edition

12.8

Silberschatz, Korth and Sudarshan

Measures of Query Cost (Cont.)

Several algorithms can reduce disk IO by using extra buffer

space

Amount of real memory available to buffer depends on other concurrent queries and OS processes, known only during execution
We

often use worst case estimates, assuming only the minimum amount of memory needed for the operation is available

Required data may be buffer resident already, avoiding disk I/O

But hard to take into account for cost estimation

Database System Concepts - 6th Edition

12.9

Silberschatz, Korth and Sudarshan

Selection Operation
File scan

Algorithm A1 (linear search). Scan each file block and test all

records to see whether they satisfy the selection condition. Cost estimate = br block transfers + 1 seek
br

denotes number of blocks containing records from relation r

If selection is on a key attribute, can stop on finding record cost = (br /2) block transfers + 1 seek Linear search can be applied regardless of

selection condition or

ordering of records in the file, or availability of indices Note: binary search generally does not make sense since data is not

stored consecutively

except when there is an index available, and binary search requires more seeks than index search
12.10 Silberschatz, Korth and Sudarshan

Database System Concepts - 6th Edition

Other Operations
Duplicate elimination can be implemented via hashing or

sorting.

On sorting duplicates will come adjacent to each other, and all but one set of duplicates can be deleted. Optimization: duplicates can be deleted during run generation as well as at intermediate merge steps in external sort-merge. Hashing is similar duplicates will come into the same bucket. perform projection on each tuple followed by duplicate elimination.

Projection:

Database System Concepts - 6th Edition

12.11

Silberschatz, Korth and Sudarshan

Other Operations : Aggregation

Aggregation can be implemented in a manner similar to duplicate

elimination.

Sorting or hashing can be used to bring tuples in the same group together, and then the aggregate functions can be applied on each group.

Optimization: combine tuples in the same group during run generation and intermediate merges, by computing partial aggregate values
For

count, min, max, sum: keep aggregate values on tuples found so far in the group. When combining partial aggregate for count, add up the aggregates

For

avg, keep sum and count, and divide sum by count at the end

Database System Concepts - 6th Edition

12.12

Silberschatz, Korth and Sudarshan

Other Operations : Set Operations

Set operations (, and ): can either use variant of merge-join

after sorting, or variant of hash-join. E.g., Set operations using hashing: 1. Partition both relations using the same hash function 2. Process each partition i as follows. 1. Using a different hashing function, build an in-memory hash index on ri. 2. Process si as follows r s: 1. Add tuples in si to the hash index if they are not already in it. 2. At end of si add the tuples in the hash index to the result.

Database System Concepts - 6th Edition

12.13

Silberschatz, Korth and Sudarshan

Other Operations : Set Operations

E.g., Set operations using hashing:
1. 2.

as before partition r and s, as before, process each partition i as follows

1. 2.

build a hash index on ri Process si as follows r s:

1.

output tuples in si to the result if they are already there in the hash index
for each tuple in si, if it is there in the hash index, delete it from the index. At end of si add remaining tuples in the hash index to the result.
12.14 Silberschatz, Korth and Sudarshan

r s:
1.

2.

Database System Concepts - 6th Edition

Evaluation of Expressions
So far: we have seen algorithms for individual operations Alternatives for evaluating an entire expression tree

Materialization: generate results of an expression whose inputs are relations or are already computed, materialize (store) it on disk. Repeat. Pipelining: pass on tuples to parent operations even as an operation is being executed

We study above alternatives in more detail

Database System Concepts - 6th Edition

12.15

Silberschatz, Korth and Sudarshan

Materialization
Materialized evaluation: evaluate one operation at a time,

starting at the lowest-level. Use intermediate results materialized into temporary relations to evaluate next-level operations.
E.g., in figure below, compute and store

building "Watson" (department )

then compute the store its join with instructor, and finally compute the projection on name.

Database System Concepts - 6th Edition

12.16

Silberschatz, Korth and Sudarshan

Materialization (Cont.)
Materialized evaluation is always applicable Cost of writing results to disk and reading them back can be

quite high

Our cost formulas for operations ignore cost of writing results to disk, so
Overall

cost = Sum of costs of individual operations + cost of writing intermediate results to disk

Double buffering: use two output buffers for each operation,

when one is full write it to disk while the other is getting filled

Allows overlap of disk writes with computation and reduces execution time

Database System Concepts - 6th Edition

12.17

Silberschatz, Korth and Sudarshan

Pipelining
Pipelined evaluation : evaluate several operations

simultaneously, passing the results of one operation on to the next. E.g., in previous expression tree, dont store result of

building "Watson" (department )

instead, pass tuples directly to the join.. Similarly, dont store result of join, pass tuples directly to projection. Much cheaper than materialization: no need to store a temporary relation to disk. Pipelining may not always be possible e.g., sort, hash-join. For pipelining to be effective, use evaluation algorithms that generate output tuples even as tuples are received for inputs to the operation. Pipelines can be executed in two ways: demand driven and producer driven

Database System Concepts - 6th Edition

12.18

Silberschatz, Korth and Sudarshan

Pipelining (Cont.)
In demand driven or lazy evaluation

system repeatedly requests next tuple from top level operation Each operation requests next tuple from children operations as required, in order to output its next tuple In between calls, operation has to maintain state so it knows what to return next

In producer-driven or eager pipelining

Operators produce tuples eagerly and pass them up to their parents

Buffer maintained between operators, child puts tuples in buffer, parent removes tuples from buffer if buffer is full, child waits till there is space in the buffer, and then generates more tuples

System schedules operations that have space in output buffer and can process more input tuples

Alternative name: pull and push models of pipelining

Database System Concepts - 6th Edition 12.19 Silberschatz, Korth and Sudarshan

Pipelining (Cont.)
Implementation of demand-driven pipelining

Each operation is implemented as an iterator implementing the following operations open() E.g. file scan: initialize file scan state: pointer to beginning of file E.g.merge join: sort relations; state: pointers to beginning of sorted relations next() E.g. for file scan: Output next tuple, and advance and store file pointer E.g. for merge join: continue with merge from earlier state till next output tuple is found. Save pointers as iterator state. close()
12.20 Silberschatz, Korth and Sudarshan

Database System Concepts - 6th Edition

Evaluation Algorithms for Pipelining

Some algorithms are not able to output results even as they get

input tuples

E.g. merge join, or hash join

intermediate results written to disk and then read back

Algorithm variants to generate (at least some) results on the fly, as

input tuples are read in

E.g. hybrid hash join generates output tuples even as probe relation tuples in the in-memory partition (partition 0) are read in Double-pipelined join technique: Hybrid hash join, modified to buffer partition 0 tuples of both relations in-memory, reading them as they become available, and output results of any matches between partition 0 tuples

When a new r0 tuple is found, match it with existing s0 tuples, output matches, and save it in r0
Symmetrically for s0 tuples
12.21 Silberschatz, Korth and Sudarshan

Database System Concepts - 6th Edition