DATA3404

Scalable Data Management

Query Processing
(Garcia-Molina/Ullman/Widom – Chapter 15 (start);
Ramakrishnan/Gehrke – Chapters 13 and 14;
Kifer/Bernstein/Lewis – Chapter 10 (start))

A/Prof Uwe Roehm

School of Computer Science

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 1

 Learning Objectives
– Understanding of Role and Structure of Query Processing
– From SQL to physical data access
1) Query Parsing,
2) Query Optimization,
3) Plan Execution
– Expression Tree vs. Evaluation Plan
– Query Execution Algorithms
– External Merge Sort
– Next week: Joins

With thanks to Bryn Jeffries for many of the example slides.

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 2
 Overview of Query Processing

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 3

 SQL Commands

Weeks 6 - 8

Parser Plan Executor Query

Evaluation
Optimizer Operator Evaluator Engine

File and Access Methods

Transaction
Manager
Recovery
Buffer Manager Manager
Lock
Manager
Weeks 2 - 4
Disk Space Manager
Concurrency Control DBMS

DATABASE

Inde System
x Catalog
Data
File
Files
s

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 4

 Database Querying with SQL
“List all students (by SID) who are enrolled in Data Systems”
SELECT studID
FROM Enrolled E, UnitOfStudy U
WHERE E.uosCode = U.uosCode 'Data Systems’

AND U.uosName = 'Data Systems'

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 5

 Compiling a Query
– SQL is declarative
– It indicates what data (information) is to be returned
– DBMS must execute a concrete, imperative program, describing step-by-step what to do
– The DBMS must therefore begin by finding an imperative program which will calculate the
result described in the declarative query
– This is somewhat similar to the way a programming language can be compiled into machine
instructions
– The process is done in stages
– Convert the text of SQL into a structured logical form (described using Relational Algebra)
– Convert the Relational Algebra logical plan into a plan with physical operations (each of which
has a program already available)
– Actually, there are many possible physical plans for a single SQL query
• The way the DBMS chooses a good physical plan is called “query optimisation”

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 6

 Basic Steps in Query Processing
SELECT name
PROJECT(name)
FROM Student
query
NATURAL JOIN Enrolled NESTED LOOPS
WHERE uosCode=‘DATA3404’;

parser and INDEX FULL SCAN TABLE ACCESS BY

translator (Enrolled_idx, INDEX ROWID
uosCode=‘DATA3404') (Student)

INDEX UNIQUE SCAN

relational algebra (Student_idx,
expression Student.sid=Enrolled.sid)

pname optimizer execution plan

⋈
evaluation engine

suosCode=‘DATA3404’ Student
query output
Enrolled Name
statistics data
about data John Smith
Sally Waters
DATA3404 ”Scalable Data Management" - 2023 (Roehm) 7
 Relational Algebra
(abbreviation: RA)

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 8

 The Central Idea
– Users request information from a database using a query language
– A query that extracts information can be seen as calculating a relation from
combining one or more relations in the current state of the database
– Relational algebra (RA) defines some basic operators that can be used to
express a calculation like that
– Each operator takes one or more relation instances, and produces a relation
instance (the operation part of the relational data model)

– Why is it important?
– Helps us understanding the precise meaning of declarative SQL queries.
– Intermediate language used within DBMS (cf. chapter on db tuning)

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 9

 The Role of Relational Algebra in a DBMS

[cf. Kifer/Bernstein/Lewis, Figure 5.1]

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 10
 What is an Algebra?
– A language based on operators and a domain of values:
– basic operators
– atomic operands (constants and variables)
– rules to form complex expressions from operators and operands, typically using
nesting with parentheses

– Example: Algebra of Arithmetic

– Operators for usual arithmetic operations: + - * /
– Operands: variables (x, y, z, …) and numerical constants
– Example arithmetic expression: 100 – ((x + y) / 2)

– In databases, operators and operands are finite relations, which leads to

the Relational Algebra
– We refer to expression as a query and the value produced as query result
DATA3404 ”Scalable Data Management" - 2023 (Roehm) 11
 Relational Algebra (RA)
– 1. Set Operations
– Union ( È ) tuples in relation 1 or in relation 2.
– Intersection ( Ç ) tuples in relation 1, as well as in relation 2.
– Difference (-) tuples in relation 1, but not in relation 2.
– 2. Operations that remove parts of a relation
– Selection ( s ) selects a subset of rows from relation.
– Projection ( p ) deletes unwanted columns from relation.
– 3. Operations that combine tuples from two relations
– Cross-product ( X ) allows us to fully combine two relations.
– Join ( ) to combine matching tuples from two relations.
– 4. A schema-level ‘rename’ operation
– Rename (r) allows us to rename one field or even whole relation
– Domain: set of relations
– operators take one/more relations as inputs and gives a new relation as result
=> RA queries by nesting of multiple operators

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 12

 Visualisation of Relational Algebra
Projection ( p ) Selection ( s ) Cross-product ( X ) Join ( )

X = =

Set Union ( È ) Set Minus ( - ) Set Intersection ( Ç )

È = - = Ç =

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 13

 Running Example
sid uos_code

name Student enrolled UnitOfStudy title

gender country semester creditPoints

Student Enrolled UnitOfStudy

sid name gender country sid uos_code semester uos_code title credit
Points
1001 Ian M AUS 1001 COMP5138 2005-S2
COMP5138 Relational DBMS 6
1002 Ha Tschi F ROK 1002 COMP5702 2005-S2
COMP5318 Data Mining 6
1003 Grant M AUS 1003 COMP5138 2005-S2
INFO6007 IT Project Management 6
1004 Simon M GBR 1006 COMP5318 2005-S2
SOFT1002 Algorithms 12
1005 Jesse F CHN 1001 INFS6014 2004-S1
ISYS3207 IS Project 4
1006 Franzisca F GER 1003 ISYS3207 2005-S2
COMP5702 MIT Research Project 18

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 14

 Set Operations
– These operations take two input relations R and S
– Set Union R È S
• Definition: R U S = { t | t Î R Ú t Î S }
– Set Intersection R Ç S
• Definition: R Ç S = { t | t Î R Ù t Î S }
– Set Difference R - S
• Definition: R - S = { t | t Î R Ù t Ï S }

– Important constraint: R and S have the same schema

– R, S have the same arity (same number of fields)
– `Corresponding’ fields must have the same names and domains
DATA3404 ”Scalable Data Management" - 2023 (Roehm) 15
 Projection
– ‘Deletes’ attributes that are not in projection list.
– Schema of result contains exactly the fields in the projection list, with the same names
that they had in the input relation.
– Examples:
Õ name, country (Student) Õ title (UnitOfStudy)
Student UnitOfStudy

name country title

Ian AUS Relational DBMS

Ha Tschi ROK Data Mining

Grant AUS IT Project Management

Simon GBR Algorithms

Jesse CHN IS Project

Franzisca GER MIT Research Project

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 16

 Selection
– Selects rows that satisfy a selection condition.
– Example: s country=‘AUS’ (Student)
Student

sid name gender country

1001 Ian M AUS

1003 Grant M AUS

– Result relation can be the input for another relational algebra operation!
(Operator composition)
– Example: Õ name ( s country=‘AUS’ (Student) )
Student

name

Ian

Grant

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 17

 Cross-Product
– Defined as: R x S = {t s | t Î R Ù s Î S}
– each tuple of R is paired with each tuple of S.
– Resulting schema has 1 field per field of R and S with field names ’inherited’ if possible.
• It might end in a conflict with two fields of the same name -> rename needed
– Sometimes also called Cartesian product
– Example:
R S result
A B C D E A B C D E
x = a 1 a 10 a
a 1 a 10 a
b 10 a a 1 b 10 a
b 2
b 20 b a 1 b 20 b
g 10 b a 1 g 10 b
b 2 a 10 a
b 2 b 10 a
b 2 b 20 b
b 2 g 10 b
DATA3404 ”Scalable Data Management" - 2023 (Roehm) 18
 Joins
– Conditional Join: R  c S = σ c (R×S)
– Example: Student  Lecturer
€ family_name=last _name
sid given family_name gender country empid first_name last_name room

1001
€ Cho Chung M AUS 47112344 Vera Chung 321

1004 Ciao Poon M CHN 12345678 Simon Poon 431

1004 Ciao Poon M CHN 99004400 Josiah Poon 482

1111 Alice Poon F AUS 12345678 Simon Poon 431

1111 Alice Poon F AUS 99004400 Josiah Poon 482

… … … … … … …

– Result schema same as the cross-product’s result schema.

– Sometimes called theta-join
– Equi-Join: Special case where the condition c contains only equalities.
DATA3404 ”Scalable Data Management" - 2023 (Roehm) 19
 Natural Join
– Natural Join: R S
– Equijoin on all shared-named fields.
€

– Result schema similar to cross-product, but only one copy of fields for
which equality is specified.
Enrolled UnitOfStudy result

sid uos_code uos_code title points sid uos_code title points

1001 COMP5138 COMP5138 Relational DBMS 6 1001 COMP5138 Relational DBMS 6

1002 COMP5702 COMP5318 Data Mining 6 1002 COMP5702 MIT Research Project 18

1003 COMP5138 INFO6007 IT Project Mgmt. 6 = 1003 COMP5138 Relational DBMS 6

1006 COMP5318 SOFT1002 Algorithms 12 1006 COMP5318 Data Mining 6

1001 INFO6007 ISYS3207 IS Project 4 1001 INFO6007 IT Project Mgmt. 6

1003 ISYS3207 COMP5702 MIT Research Project 18 1003 ISYS3207 IS Project 4

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 20

 Rename Operation
– Allows to name, and therefore to refer to, the results of relational-algebra
expressions.
– Allows to refer to a relation by more than one name.
– Notation 1: r x (E)
– returns the expression E under the name X (rename whole relation)
– Notation 1: r X (a1 à b1) (E)
– returns the result of expression E under the name X, and with the attributes a1
…. renamed to b1 … (rename individual attributes)
– (assumes that the relational-algebra expression E has arity n)

– Example: r Classlist( sid à student ) ( s uos_code=‘DATA3404' (Enrolled) )

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 21
 Basic versus Derived Operators
– We can distinguish between basic and derived RA operators
– Only 6 basic operators are required to express everything else:
– Union (È) tuples in relation 1 or in relation 2.
– Set Difference (-) tuples in relation 1, but not in relation 2.
– Selection (s) selects a subset of rows from relation.
– Projection (p) deletes unwanted columns from relation.
– Cross-product (X) allows us to fully combine two relations.
– Rename (r) allows us to rename one field to another name.

– Additional (derived) operations:

– intersection, join, division:
• Not essential, but (very!) useful.
– Cf. Join: R  S = σ (R×S)
c c
DATA3404 ”Scalable Data Management" - 2023 (Roehm) 22
€
 Relational Expressions
– A basic expression in relational algebra consists of either one of the following:
– Variable that refers to a relation of the same name in the database
– A constant relation
– Let E1 and E2 be relational-algebra expressions; then
the following are all relational-algebra expressions:
– E1 È E2
– E1 - E2
– E1 x E2
– E1  E2
– sP (E1), P is a (complex) predicate on attributes in E1
–€ πS (E1), S is a list consisting of some of the attributes in E1
– rX (E1), x is the new name for the result of E1
DATA3404 ”Scalable Data Management" - 2023 (Roehm) 23
 Exercise 1: Relational Algebra
Each of the following queries can be performed with a single Relational
Algebra operation. Which operator is required in each case?
1. SELECT name, age FROM Student
πname,age (Student)
2. SELECT * FROM Enrolled WHERE mark BETWEEN 85 AND 100
σmark=>85 ∧ mark<=100 (Enrolled)
3. SELECT * FROM Student, Enrolled
Student × Enrolled
4. SELECT * FROM Enrolled E, UosOffering U
WHERE U.uoscode=E.uoscode AND U.semester=E.semester
AND U.year=E.year
Enrolled⋈uoscode,semester,year UosOffering
DATA3404 ”Scalable Data Management" - 2023 (Roehm) 24
 Relational Algebra Recap: Projection and Selection
Assessment
sid uosCode sem year mark
316424328 INFO2120 S1 2012 72
305678453 INFO2120 S1 2012 86
316424328 INFO3005 S1 2010 63
305678453 COMP5138 S1 2012 94

RA: RA:
πuosCode, sem(Assessment) σuosCode=‘INFO2120’(Assessment)

uosCode sem uosCode sem sid uosCode sem year mark

INFO2120 S1 INFO2120 S1 316424328 INFO2120 S1 2012 72

INFO3005 S1 INFO2120 S1 305678453 INFO2120 S1 2012 86
COMP5138 S1 INFO3005 S1
Strict set-based COMP5138 S1

Example SQL (strict projection): Example SQL (selection):

SELECT DISTINCT uosCode, sem SELECT *
FROM Assessment; FROM Assessment
WHERE uosCode=‘INFO2120’;
DATA3404 ”Scalable Data Management" - 2023 (Roehm) 25
 RA Recap: Joins
Assessment UoSLecturer
sid uosCode sem year mark uosCode sem year lecturer
316424328 INFO2120 S1 2012 72 INFO2120 S1 2012 Uwe Roehm
305678453 INFO2120 S1 2012 86 INFO3005 S1 2010 Irena Koprinska
316424328 INFO3005 S1 2010 63
COMP5138 S2 2012 Bryn Jeffries
305678453 COMP5138 S1 2012 94

sid mark uosCode sem year lecturer

RA:
Assessment ⋈ UosLecturer
316424328 72 INFO2120 S1 2012 Uwe Roehm

305678453 86 INFO2120 S1 2012 Uwe Roehm

Example SQL: 316424328 63 INFO3005 S1 2010 Irena Koprinska

SELECT *
FROM Assessment A, UoSLecturer L
WHERE A.uosCode=L.uosCode
AND A.sem=L.sem
AND A.year=L.year;
OR
SELECT * FROM Assessment NATURAL JOIN UoSLecturer;
DATA3404 ”Scalable Data Management" - 2023 (Roehm) 26
 Concepts of Query Processing

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 27

 Overview of Query Translation and Execution
– SQL gets translated into relational algebra, which can be shown as logical
expression tree.
– Operators have one or more input sets and return one (or more) output set(s).
– Leaves correspond to (base) relations.
– Query Execution Plan: Tree of relational algebra operators, with choice of
algorithm for each operator.

– Two main issues in query optimization (cf. Week 8):

– For a given query, what plans are considered?
• Algorithm to search plan space for cheapest (estimated) plan.
– How is the cost of a plan estimated?
– Ideally:Want to find best plan.
– Practically: Avoid worst plans!
DATA3404 ”Scalable Data Management" - 2023 (Roehm) 28
 Parsing and Translation
SELECT name SQL query
FROM Student NATURAL JOIN Enrolled
WHERE uosCode=‘DATA3404’;

Projection Join Selection Relational Algebra (RA)

pname(Student ⋈ suosCode=‘DATA3404’(Enrolled))

RA Expression Tree Execution Plan

pname PROJECT(name)

⋈ NESTED LOOPS

suosCode=‘DATA3404’ Student INDEX FULL SCAN INDEX UNIQUE

(Enrolled_idx, SCAN (Student)
uosCode=‘DATA3404')
Enrolled

Logical Operators Physical Operators

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 29
 Exercise 2: Relational Algebra Expression Trees
Write a relational algebra expression, in the form of an
expression tree, equivalent to: pS.name, U.uoSname

SELECT S.name, U.uosName,

FROM Student S, Enrolled E, UosOffering U ⋈
WHERE S.sid = E.sid
ρU
AND U.uosCode = E.uosCode ⋈
AND U.semester = E.semester UosOffering
AND U.year = E.year ρS
sE.mark>=85
AND E.mark BETWEEN 85 AND 100; ∧ E.mark<=100

ρE Student
Rename operator (not essential)
Enrolled

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 30

 Expression Tree vs. Query Execution Plan

Õ balance Project balance Project balance

sbalance<2500 Index Range Scan

(account.balance,
Filterbalance<2500

balance<2500)

account Table Scan (account)

– Relational algebra operators == logical operators which represent the

intermediate results.
– Physical operators show how query is evaluated.
DATA3404 ”Scalable Data Management" - 2023 (Roehm) 31
 Physical Operations
Logical Operations RA Operators Physical Operations

• File Scan
• Access Paths
• Selections
σ • Index Scan
• Filter

• Projections π • Simple projection

• Merge-Sort

• Inner Joins, • Nested Loops

• Outer Joins, ⋈, × • Block-nested Loop
• Cross Products • Nested Index Loop
• …
Others
• Grouping Γ, ∩, ∪, -
• Set Operations
DATA3404 ”Scalable Data Management" - 2023 (Roehm) 32
 Common Techniques for Physical Operations

Indexing: Can use WHERE conditions to retrieve small

set of tuples (selections, joins)

Iteration: Sometimes, faster to scan all tuples even if

there is an index. (And sometimes, we can scan the
data entries in an index instead of the table itself.)

Partitioning: By using sorting or hashing, we can partition

the input tuples and replace an expensive operation by
similar operations on smaller inputs.

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 33

 Selections and Projections

suosCode=‘DATA3404’ πname

FILTER(uosCode=‘DATA3404’) PROJECT(name)

for each tuple r in R do begin for each tuple r in R do begin

if r.uosCode=‘DATA3404' then add (r.name) to result
add r to result end
end
(Often combined with Variations:
access path operation) § Sorted tuples (ORDER BY)
Ø Sort operation
§ Duplicate removal (DISTINCT, set operations)
Ø Partition tuples by hashing or sorting,
then remove duplicates
DATA3404 ”Scalable Data Management" - 2023 (Roehm) 34
 Access Paths and Selections
An access path is a method of retrieving tuples.
Examples:
SELECT * SELECT * SELECT name
Query FROM Student; FROM Enrolled FROM Student
WHERE uosCode=‘DATA3404’; WHERE sid=1234;

RA Student suosCode=‘DATA3404’ πname

Enrolled ssid=1234
Access Paths Student
Ø Table Scan: Ø Table Scan + Filter
Retrieve all pages (table scans typically include filtering)
of relation Ø Index Scan: Use index with matching Ø Index-only scan:
search key to find matching records Use a covering index
(if available) without accessing
- tree index? records (if it exists)
- hash Index?
DATA3404 ”Scalable Data Management" - 2023 (Roehm) 35
 Projections
SELECT R.sid, R.bid FROM Reserves R
– Trivial operation: just return subset of columns

SELECT DISTINCT R.sid, R.bid FROM Reserves R

– More expensive: need to remove duplicates.
– Sorting Approach: Sort on <sid, bid> and remove duplicates. (Can optimize
this by dropping unwanted information while sorting.)
– Hashing Approach: Hash on <sid, bid> to create partitions. Load partitions into
memory one at a time, build in-memory hash structure, and remove duplicates
– If there is an index with both R.sid and R.bid in the search key, may be
cheaper than to sort data entries!
DATA3404 ”Scalable Data Management" - 2023 (Roehm) 36
 Join Operations
– Several different algorithms to implement joins
– Nested-loop join
– Block nested-loop join
– Indexed nested-loop join
– Merge-join
– Hash-join
– Choice based on cost estimate

– More details on this next week…

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 37

 Measures of Query Costs
– Query Optimization: Amongst all equivalent evaluation plans choose the one with
lowest cost.
– Cost is generally measured as total elapsed time for answering query
– Many factors contribute to time cost
• disk accesses, CPU, or even network communication
– Typically disk access is the predominant cost, and is also relatively easy to estimate
– For simplicity we just use number of block transfers from disk as the cost measure
– We ignore the difference in cost between sequential and random I/O for simplicity
– We also ignore CPU costs for simplicity
– Cost of some operations (eg. joins or sort) depends on input data size
• reduction factor: # output tuples / # input tuples

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 38

 Passing records between operations
› Materialization (set-at-a-time): materialised
1. Evaluate whole operation Join results pname,uosName
2. Store (materialize) results in a temporary relation
3. Next operation reads in temporary relation. materialised
Join results

• Always applicable, expensive I/O ⋈

materialised set:

› Pipelining (tuple-at-a-time): Enrolled HD rows

⋈ Student
1. Evaluate one row of output of a relation
2. Pass (pipeline) each row to next operation sgrade=‘HD’
UnitOfStudy
• Much cheaper (all in memory) Enrolled
• Some operations not compatible with pipelining
(e.g., inner table of join, sorts, hash joins,
aggregations)

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 39

 Summary
– Parsing: convert a text-string which is the query, into a structured form (eg a
tree with logical operators in RA)
– What can be executed by the system, to find the result of the query, is a
physical plan (a tree of physical operations, which work on the physical
structure of the data eg index)
– There are often several different physical operations that perform the same
logical operation, and several different trees of logical operations that
produce the same result from a query
– The data management system has to choose between them (this is query
optimization)

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 40

 Looking ahead
– There are several alternative evaluation algorithms for each relational operator.
– We must understand query optimization in order to fully understand the
performance impact of a given database design (relations, indexes) on a
workload (set of queries).
– Two parts to optimizing a query:
– Consider a set of alternative physical plans (each corresponding to some logical
plan, with particular choices of the physical operators).
• Must prune search space; typically, left-deep plans only.
– Must estimate cost of each plan that is considered.
• Must estimate size of result and cost for each plan node.
• Key issues: Statistics, indexes, operator implementations.
– Choose one physical plan (with lowest estimated cost, among those considered);
this is what will be executed
DATA3404 ”Scalable Data Management" - 2023 (Roehm) 41
 External Sorting

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 42

 Motivation for Sorting
– A classic problem in computer science!
– Why is sorting important for DBMS?
• SQL queries can specify that the output be sorted (ORDER BY)
• Duplicate elimination
• SQL can be implemented efficiently if the input are sorted
• e.g. sort-merge-join algorithm involves sorting
• Sorting is first step in bulk loading B+ tree index.
– Why is it different in DBMSs to normal CS sorting?
• For relations that fit in memory, techniques like QuickSort can be used.
• Problem: sort 100 GB of data with 1 GB of RAM…
• why not virtual memory?
DATA3404 ”Scalable Data Management" - 2023 (Roehm) 43
 Sorting in Memory
SELECT * 2) Sort rows in Buffer in RAM
FROM Person memory (using,
Name Home Age
e.g., quicksort)
ORDER BY age; NeveTerry
Gail Daniels Alice Springs
Cairns 50
3
Ali Graves
Chastity Silva Adelaide
Melbourne 25
4
LionelIngram
Julie Hines Melbourne 14
7
Wyoming
Laurel Daniels
Campbell Adelaide
Darwin 59
9
SORT(age) Wade Kramer
Brendan
Meghan
Lionel
Wall
Hines
Frame 1
Patrick
Adelaide
Perth
Newcastle
Melbourne
36
13
25
14
Chastity Silva
Vaughan Nicholson Melbourne
Perth 16
4
AphroditeHolden
Leonard George Darwin 56
21
TABLE SCAN(Person) Vaughan
Ali GravesNicholson Perth
Adelaide 16
25
Julie Ingram
Meghan Patrick Melbourne
Newcastle 25
7
3) Output rows,
or save for next Name Home Age
operation. Clio
Wade Brooks
Kramer Newcastle
Adelaide 63
36
Melodie
Macy RuizStewart Melbourne
Brisbane 65
38
2 I/O Laurel
Aphrodite
Daniels
Maynard Darwin
Cairns 42
9

Person Macy
Neve Daniels
Ruiz Brisbane
Alice Springs 38
50
Sawyer
Aphrodite Frame 2
Holloway
George Sydney
Darwin 59
56

2 I/O Aphrodite
Esra Rasmussen
Esra
Wyoming
Maynard
Rasmussen
Campbell
Cairns
Cairns
Adelaide
42
56
56
59
Brendan
Sawyer Holloway
Wall Perth
Sydney 13
59
Gail
Clio Brooks
Terry Cairns
Newcastle 63
3
1) Read relation
Leonard
Melodie Holden
Stewart Darwin
Melbourne 21
65
Into memory
DATA3404 ”Scalable Data Management" - 2023 (Roehm) 44
 General External Merge Sort
– To sort a file with N pages using B buffer pages:
– Pass 0: use B buffer pages.
Produce éN / Bù sorted runs of B pages each.
– Pass 2, …, etc.: merge B-1 runs.

INPUT 1

... INPUT 2

...
OUTPUT ...
INPUT B-1
Disk Disk
B Main memory buffers

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 45

 External Merge Sort – Pass 0: Sort Phase

Buffer in RAM
3 I/O
Frame unsorted
unsorted Frame unsorted
Frame
1
Sorted Run
2 3
Buffer size B = 3
Person
3 I/O

Sort Phase:
Read all N pages into memory, B pages at a time.
Save back to disk as N/B = 12 sorted runs of B = 3 pages.
Total cost is N reads and N writes = 2N I/O
N=36 pages

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 46

 External Merge Sort – Pass i+1: Merging Runs

Buffer in RAM
Person Chastity Silva 4 Macy Ruiz
Julie Ingram 38
7 Chastity Silva
Brendan Wall 4
13

Input 1
Lionel Hines 14
Input 2
Laurel Daniels
Aphrodite Maynard 9
42
Output
Julie Ingram 7

Vaughan Nicholson 16 Esra

Brendan
Rasmussen
Wall 56
13 Laurel Daniels 9

2 runs (6 pages) read

1 merged run (6 pages) written

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 47

 External Merge Sort – Merge Phase
Merge pass 1:
12 sorted runs of 3 pages merged
to 6 sorted runs of 6 pages
Person N=36 (whole relation) pages read
N=36 (whole relation) pages written

Merge pass 2:
6 sorted runs of 6 pages merged
to 3 sorted runs of 12 pages
N=36 (whole relation) pages read
N=36 (whole relation) pages written

2 more merge passes required to

DATA3404 ”Scalable Data Management" - 2023 (Roehm)
make 1 sorted run of 36 pages 48
 Exercise 3: External Merge Sort Costs
A relation of 100,000 records is stored unsorted in 10,000 pages. It needs to
be sorted with an external merge sort, and the system has an available buffer
of 30 pages.
1) How many sorted runs will be initially produced?
334: Ceil(10000/30)
2) How much I/O is performed in the first merge pass?
20k: Every pass requires one read and one write of all 10,000 pages
3) How many merge passes are required?

4) (Harder) Can you derive a general formula for the amount of I/O?

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 49

 Scalability of External Merge Sort
Upper bound for total I/O: 2N ( élogB–1(N / B)ù + 1 )
number of passes
Number of passes:
N
B=3 B=5 B=9 B=17 B=129 B=257
100 7 4 3 2 1 1
1,000 10 5 4 3 2 2
10,000 13 7 5 4 2 2
100,000 17 9 6 5 3 3
1,000,000 20 10 7 5 3 3
10,000,000 23 12 8 6 4 3
100,000,000 26 14 9 7 4 4
1,000,000,000 30 15 10 8 5 4
DATA3404 ”Scalable Data Management" - 2023 (Roehm) 50
 Using B+ Trees for Sorting
– Scenario: Table to be sorted has B+ tree index on sorting
column(s).
– Idea: Can retrieve records in order by traversing leaf pages.
– Is this a good idea?
– Cases to consider:
– B+ tree is clustered Good idea!
– B+ tree is not clustered Could be a very bad idea!

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 51

 Merge Sort Summary
– External sorting is important; DBMS may dedicate part of
buffer pool for sorting!
– External merge sort minimizes disk I/O cost:
– Pass 0: Produces sorted runs of size B (# buffer pages).
Later passes: merge runs.
– # of runs merged at a time depends on B, and block size.
– Larger block size means less I/O cost per page.
– Larger block size means smaller # runs merged.
– In practice, # of runs rarely more than 2 or 3.
– The best sorts are wildly fast:
– Despite 40+ years of research, we’re still improving!
DATA3404 ”Scalable Data Management" - 2023 (Roehm) 52
 Measuring the State of the Art of Sorting
Metric: Sort rate (TBs / minute) achieved while sorting a very large amount of data
GraySort
(currently 100 TB minimum).
Metric: Minimum cost for sorting a very large amount of data on a public cloud.
CloudSort
(currently 100 TB). Complete rules in the CloudSort short paper.
Metric: Amount of data that can be sorted for a penny's worth of system time.
PennySort
Originally defined in AlphaSort paper. PennySort benchmark is now deprecated.
Metric: Amount of data that can be sorted in 60.00 seconds or less.
MinuteSort
Originally defined in AlphaSort paper.
Metric: Amount of energy required to sort either 108, 109 or 1010 records (10 GB, 100 GB or 1TB)
JouleSort
Originally defined in JouleSort paper. Only 1010 records measured nowadays.

Metric: Elapsed time to sort 1012 bytes of data.

TeraByte Sort
The TeraByte benchmark is now deprecated because it became essentially the same as MinuteSort.
Metric: Amount of time to sort one million records (100 MB).
This is the original sort benchmark, defined in A Measure of Transaction Processing Power With 25
Datamation Sort
others Datamation, V 31.7, April 1985, pp 112-118. Originally, winners took 1 hour, now 1 sec !
So the benchmark is deprecated.
DATA3404 ”Scalable Data Management" - 2023 (Roehm) http://sortbenchmark.org 53
 Summary

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 54

 Summary
– Concepts of Query Processing
– Parsing – Optimization – Execution
– We concentrate on parsing this week
– Relational Algebra
– Formal semantic of relational queries
– Logic of a query internally represented as expression tree
– Basis for later query optimization (Wk 8) due to different equivalent RA
expressions possible for the same query
– External Sorting
– Central operation, not only for ordering, also duplicate elimination and joins

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 55

 Next Week
– Query Execution
– Join Algorithms
– Set-Operations, Aggregation, Grouping etc

– Textbook:
– Garcia-Molina/Ullman/Widom: Chapter 15
– Ramakrishnan/Gehrke: Chapter 14
– Kifer/Bernstein/Lewis: Chapter 10.3-10.7

DATA3404 ”Scalable Data Management" - 2023 (Roehm) 56