Algorithms for Query

Processing and Optimization

Introduction to Query Processing
 Query optimization:
 The process of choosing a suitable
execution strategy for processing a query.
 Two internal representations of a query:
 Query Tree
 Query Graph
Introduction to Query Processing

Copyright © 2011 Ramez Elmasri and Shamkant Navathe

 1. Translating SQL Queries into
Relational Algebra
 Query block:
 The basic unit that can be translated into
the algebraic operators and optimized.
 A query block contains a single SELECT-FROM-
WHERE expression, as well as GROUP BY and
HAVING clause if these are part of the block.
 Nested queries within a query are identified as
separate query blocks.
 Aggregate operators in SQL must be included in
the extended algebra.
Translating SQL Queries into
Relational Algebra (2)
SELECT LNAME, FNAME
FROM EMPLOYEE
WHERE SALARY > ( SELECT MAX (SALARY)
FROM EMPLOYEE
WHERE DNO = 5);

SELECT LNAME, FNAME SELECT MAX (SALARY)

FROM EMPLOYEE FROM EMPLOYEE
WHERE SALARY > C WHERE DNO = 5

ℱMAX SALARY (σDNO=5 (EMPLOYEE))

π LNAME, FNAME (σSALARY>C(EMPLOYEE))
Algorithms for External Sorting
 External sorting:
 Refers to sorting algorithms that are suitable for large files of records

stored on disk that do not fit entirely in main memory, such as most database files.
 Sort-Merge strategy:
 Starts by sorting small subfiles (runs) of the main file and then merges the sorted
runs, creating larger sorted subfiles that are merged in turn.
 Sorting phase:
 nR = (b/nB)
 Merging phase:
 dM = Min (nB-1, nR)
 nP = (logdM(nR))
 Parameters
 b: number of file blocks;
 nB: available buffer space (in blocks);
 nR: number of initial runs;
 dM: degree of merging;
 nP: number of passes.
External Sort-Merge
Let M denote memory size (in pages).
1. Create sorted runs. Let i be 0 initially.
Repeatedly do the following till the end of the relation:
(a) Read M blocks of relation into memory
(b) Sort the in-memory blocks
(c) Write sorted data to run Ri; increment i.
Let the final value of i be N
2. Merge the runs
External Sort-Merge (Cont.)
2. Merge the runs (N-way merge). We assume (for now) that N < M.
1. Use N blocks of memory to buffer input runs, and 1 block to buffer output. Read the first
block of each run into its buffer page
2. repeat
1. Select the first record (in sort order) among all buffer pages
2. Write the record to the output buffer. If the output buffer is full write it to disk.
3. Delete the record from its input buffer page.
If the buffer page becomes empty then
read the next block (if any) of the run into the buffer.
3. until all input buffer pages are empty:
External Sort-Merge (Cont.)
• If N  M, several merge passes are required.
• In each pass, contiguous groups of M - 1 runs are merged.
• A pass reduces the number of runs by a factor of M -1, and creates
runs longer by the same factor.
• E.g. If M=11, and there are 90 runs, one pass reduces the
number of runs to 9, each 10 times the size of the initial runs
• Repeated passes are performed till all runs have been merged into
one.
Example: External Sorting Using Sort-Merge
Algorith
ms for
External
Sorting
 Algorithms for SELECT
 Examples:
 (OP1):  SSN='123456789' (EMPLOYEE)
 (OP2):  DNUMBER>5(DEPARTMENT)
 (OP3):  DNO=5(EMPLOYEE)
 (OP4):  DNO=5 AND SALARY>30000 AND SEX=F(EMPLOYEE)

 (OP5):  ESSN=123456789 AND PNO=10(WORKS_ON)

Algorithms for SELECT
 Search Methods for Simple Selection:
 S1 Linear search (brute force):
 Retrieve every record in the file, and test whether its attribute
values satisfy the selection condition.
 S2 Binary search:
 If the selection condition involves an equality comparison on a
key attribute on which the file is ordered, binary search (which
is more efficient than linear search) can be used. (See OP1).
 S3 Single record search:
 If the selection condition involves an equality comparison on a
key attribute with a primary index (or a hash key), use the
primary index (or the hash key) to retrieve the record.
Algorithms for SELECT
 Search Methods for Simple Selection:
 S4 Multiple search (primary index):

 If the comparison condition is >, ≥, <, or ≤ on a key field with a

primary index, use the index to find the record satisfying the
corresponding equality condition, then retrieve all subsequent records
in the (ordered) file.
 S5 Multiple search (clustering index):
 If the selection condition involves an equality comparison on a non-
key attribute with a clustering index, use the clustering index to
retrieve all the records satisfying the selection condition.
 S6 Range query (B+-tree) :
 To be used to retrieve records on conditions involving >,>=, <, or
<=.
 Can also be used for an equality comparison, for single record search
if the indexing field has unique values (is a key) or to retrieve multiple
records if the indexing field is not a key.
Algorithms for SELECT
 Search Methods for Simple Selection:
 S7 Conjunctive selection:
 If an attribute involved in any single simple condition in the
conjunctive condition has an access path that permits the use
of one of the methods S2 to S6, use that condition to retrieve
the records and then check whether each retrieved record
satisfies the remaining simple conditions in the conjunctive
condition.
Algorithms for JOIN
 Join (EQUIJOIN, NATURAL JOIN)
 two–way join: a join on two files
 e.g.
A=B S
 R
multi-way joins: joins involving more than two files.
 e.g. R A=B
C=D T
S
 Examples
 (OP6): EMPLOYEE
DNO=DNUMBER DEPARTMENT
 (OP7): DEPARTMENT
MGRSSN=SSN EMPLOYEE
 Algorithms for
JOIN
 Methods for implementing joins:
 J1 Nested-loop join (brute force):
 For each record t in R (outer loop), retrieve
every record s from S (inner loop) and test
whether the two records satisfy the join
condition t[A] = s[B].
 J2 Single-loop join (Using an access
structure to retrieve the matching
records):
 If an index (or hash key) exists for one of
the two join attributes
— say, B of S — retrieve each record t in
R, one at a time, and then use the access
structure to retrieve directly all matching
records s from S that satisfy s[B] = t[A].
Nested-loop join (brute force):
 Algorithms for
JOIN
 Methods for implementing joins:
 J3 Sort-merge join:
 If the records of R and S are physically sorted (ordered) by value of the join
attributes A and B, respectively, we can implement the join in the most efficient
way possible.
 Both files are scanned in order of the join attributes, matching the records that have
the same values for A and B.

 In this method, the records of each file are scanned only once each for matching
with the other file—unless both A and B are non-key attributes, in which case the
method needs to be modified slightly.
Algorithm for SORT MERGE JOIN
Example
Algorithm for
SORT
MERGE JOIN
 Algorithms for JOIN
Operations
 Methods for implementing joins:
 J4 Hash-join:
 The records of files R and S are both hashed to the same hash file, using
the same hashing function on the join attributes A of R and B of S as hash
keys.
 A simple hash join is simply a two-phase algorithm that:
 Build phase: A single pass through the file with fewer records (say, R)

hashes its records to the hash file buckets.

 Probe phase: A single pass through the other file (S) then hashes each

of its records to the appropriate bucket, where the record is combined

with all matching records from R.
Relational Algebra Revisited….

Link for further study

http://www.mathcs.emory.edu/
~cheung/Courses/377/Syllabus/4-
RelAlg/intro.html
Set operations

• set union is:

• Suppose A = {i, j, k} and B = {k, x, y}
• Then the union of A and B is {i, j, k, x, y}
• set intersection is:
• Suppose A = {i, j, k} and B = {k, x, y}
• Then the intersection of A and B is {k}
• set difference is:
• Suppose A = {i, j, k} and B = {k, x, y}
• Then the difference of A and B is {i, j}
Note that tuples in sets A and B must have the same FORMAT (same number and type of attributes) to be
used in the set difference operation.
• σ (Condition) (R) (selection)
• Selects tuples from a relation R that satisfies the condition Condition
• π (attribute-list)
(projection)
• Selects out only the attribute values attribute-list from all tuples in
relation R
Algorithms for –
b)Projection without
Duplication
c)Union
Algorithms for-
d)Intersection
e)Difference
Algorithms for JOIN Operations
 Factors affecting JOIN performance
 Available buffer space
 Join selection factor
 Choice of inner VS outer relation
Query Optimization in Centralized Systems

• In a centralized system, query processing is done with the

following aim −
• Minimization of response time of query (time taken to produce
the results to user’s query).
• Maximize system throughput (the number of requests that are
processed in a given amount of time).
• Reduce the amount of memory and storage required for
processing.
• Increase parallelism.
Steps for Query Optimization

• Step 1 − Query Tree Generation

• Step 2 − Query Plan Generation
• Step 3− Code Generation
 Using Heuristics in Query

Optimization
Query tree:
 A tree data structure that corresponds to a relational algebra expression. It
represents the input relations of the query as leaf nodes of the tree, and
represents the relational algebra operations as internal nodes.

 An execution of the query tree consists of executing an internal node

operation whenever its operands are available and then replacing that
internal node by the relation that results from executing the operation.

 Query graph:
 A graph data structure that corresponds to a relational calculus expression. It
does not indicate an order on which operations to perform first. There is only a
single graph corresponding to each query.
Using Heuristics in Query Optimization
 Example:
 For every project located in ‘Stafford’, retrieve the project number, the controlling

department number and the department manager’s last name, address and
birthdate.
 Relation algebra:
PNUMBER, DNUM, LNAME, ADDRESS, BDATE
(((PLOCATION=‘STAFFORD’(PROJECT))
DNUM=DNUMBER (DEPARTMENT)) MGRSSN=SSN (EMPLOYEE))

 SQL query:
Q2: SELECT P.NUMBER,P.DNUM,E.LNAME,
E.ADDRESS, E.BDATE
FROM PROJECT AS P,DEPARTMENT AS D,
EMPLOYEE AS E
WHERE P.DNUM=D.DNUMBER AND
D.MGRSSN=E.SSN AND
P.PLOCATION=‘STAFFORD’;
Using Heuristics in Query
Optimization̤

Notation for Query Trees

and Query Graph
 Using Heuristics in Query
Optimization
 Heuristic Optimization of Query Trees:
 The same query could correspond to many different relational algebra expressions
and hence many different query trees.
 The task of heuristic optimization of query trees is to find a
final query tree that is efficient to execute.
 Example:

Q: SELECT LNAME
FROM EMPLOYEE, WORKS_ON, PROJECT
WHERE PNAME = ‘AQUARIUS’ AND
PNMUBER=PNO AND ESSN=SSN
AND BDATE > ‘1957-12-31’;
Using
Heuristics in
Query
Optimization
Using
Heuristics in
Query
Optimization
Using Heuristics in
Query Optimization
Example….
Optimization Steps
Using Heuristics in Query Optimization
 General Transformation Rules for Relational Algebra Operations:
1. Cascade of : A conjunctive selection condition can be broken up into a cascade (sequence) of
individual  operations:
  c1 AND c2 AND ... AND cn(R) = c1 (c2 (...(cn(R))...) )
2. Commutativity of : The  operation is commutative:
 c1 (c2(R)) = c2 (c1(R))

3. Cascade of : In a cascade (sequence) of  operations, all but the last one can be ignored:
 List1 (List2 (...(Listn(R))...) ) = List1(R)

4. Commuting  with : If the selection condition c involves only the attributes A1, ..., An in the
projection list, the two operations can be commuted:
 A1, A2, ..., An (c (R)) = c (A1, A2, ..., An (R))
General Transformation Rules for Relational
Algebra Operations (contd.):

5. Commutativity of ( and x ): The operation is commutative as

the x operation: is
 R
CS = S

6. Commuting  withC R;(orRxx): SIf =

allSthe
x Rattributes in the selection
condition c involve only the attributes of one of the relations being
joined—say, R—the two operations can be commuted as follows:
 c ( R S ) = (c (R)) S

 Alternatively, if the selection condition c can be written as (c1 and c2),

where condition c1 involves only the attributes of R and condition c2
involves only the attributes of S, the operations commute as follows:
 c ( R S ) = (c1 (R)) (c2 (S))
Using Heuristics in Query
Optimization
 General Transformation Rules for Relational Algebra
Operations (contd.):
7. Commuting  with (or x): Suppose that the projection list
is L = {A1, ..., An, B1, ..., Bm}, where A1, ..., An are attributes of R and
B1, ..., Bm are attributes of S. If the join condition c involves only
attributes in L, the two operations can be commuted as follows:

 L ( R
C S) = (A1, ..., An (R)) C ( B1, ..., Bm (S))
 If the join condition C contains additional attributes not in
L, these must be added to the projection list, and a final
 operation is needed.
Using Heuristics in Query
Optimization
 General Transformation Rules for Relational Algebra
Operations (contd.):
8. Commutativity of set operations: The set operations υ and
∩ are commutative but “–” is not.
9. Associativity of , x, υ, and ∩ : These four operations are
individually associative; that is, if  stands for any one of these four
operations (throughout the expression), we have
 (RS) T = R(ST )
10.Commuting  with set operations: The  operation commutes
with υ , ∩ , and –. If  stands for any one of these three
operations, we have
 c ( R  S ) = (c (R))  (c (S))
Using Heuristics in Query
Optimization
 General Transformation Rules for Relational Algebra
Operations (contd.):
 The  operation commutes with υ.
L ( R υ S ) = (L (R)) υ (L (S))

 Converting a (, x) sequence into : If

the condition c of a
 that follows a x Corresponds to a join condition,
convert the (, x) sequence into a as follows:
(C (R x S)) = (R C S)

 Other transformations
Using Heuristics in Query
Optimization
Outline of a Heuristic Algebraic Optimization Algorithm:
1. Using rule 1, break up any select operations with conjunctive conditions
into a cascade of select operations.
2. Using rules 2, 4, 6, and 10 concerning the commutativity of select with other
operations, move each select operation as far down the query tree as is
permitted by the attributes involved in the select condition.
3. Using rule 9 concerning associativity of binary operations, rearrange the leaf nodes
of the tree so that the leaf node relations with the most restrictive select operations
are executed first in the query tree representation.
4. Using Rule 12, combine a Cartesian product operation with a subsequent
select operation in the tree into a join operation.
5. Using rules 3, 4, 7, and 11 concerning the cascading of project and the commuting
of project with other operations, break down and move lists of projection attributes
down the tree as far as possible by creating new project operations as needed.
6. Identify subtrees that represent groups of operations that can be executed by
a single algorithm.
Using Heuristics in Query
Optimization (15)
 Summary of Heuristics for Algebraic Optimization:
1. The main heuristic is to apply first the operations that reduce
the size of intermediate results.
2. Perform select operations as early as possible to reduce the number
of tuples and perform project operations as early as possible to
reduce the number of attributes. (This is done by moving select and
project operations as far down the tree as possible.)
3. The select and join operations that are most restrictive should be
executed before other similar operations. (This is done by
reordering the leaf nodes of the tree among themselves and
adjusting the rest of the tree appropriately.)
Using Selectivity and Cost Estimates in
Query Optimization
 Cost Components for Query Execution
1. Access cost to secondary storage
2. Storage cost
3. Computation cost
4. Memory usage cost
5. Communication cost

 Note: Different database systems may focus on

different cost components.
Using Selectivity and Cost Estimates in
Query Optimization
 Catalog Information Used in Cost Functions
 Information about the size of a file
 number of records (tuples) (r),
 record size (R),
 number of blocks (b)
 blocking factor (bfr)
 Information about indexes and indexing attributes of a file
 Number of levels (x) of each multilevel index
 Number of first-level index blocks (bI1)
 Number of distinct values (d) of an attribute
 Selectivity (sl) of an attribute
 Selection cardinality (s) of an attribute. (s = sl * r)
Semantic Query Optimization
 Semantic Query Optimization:
 Uses constraints specified on the database schema in order to

modify one query into another query that is more efficient to

execute.
 Consider the following SQL query,
SELECT E.LNAME, M.LNAME
FROM EMPLOYEE E, EMPLOYEE M
WHERE E.SUPERSSN=M.SSN AND
 Explanat E.SALARY>M.SALARY
ion:
 Suppose that we had a constraint on the database schema that

stated that no employee can earn more than his or her direct
supervisor. If the semantic query optimizer checks for the
existence of this constraint, it need not execute the query at all
because it knows that the result of the query will be empty.