Sorting &

10 Aggregations

Intro to Database Systems Andy Pavlo

15-445/15-645
Fall 2019 AP Computer Science
Carnegie Mellon University
 2

ADMINISTRIVIA

Homework #3 is due Wed Oct 9th @ 11:59pm

Mid-Term Exam is Wed Oct 16th @ 12:00pm

Project #2 is due Sun Oct 20th @ 11:59pm

CMU 15-445/645 (Fall 2019)

 3

C O U R S E S TAT U S

We are now going to talk about how Query Planning

to execute queries using table heaps
and indexes. Operator Execution

Next two weeks: Access Methods

→ Operator Algorithms Buffer Pool Manager
→ Query Processing Models
→ Runtime Architectures Disk Manager

CMU 15-445/645 (Fall 2019)

 4

QUERY PL AN
SELECT A.id, B.value
The operators are arranged in a tree. FROM A, B
WHERE A.id = B.id
Data flows from the leaves of the tree AND B.value > 100
up towards the root.
p A.id, B.value

The output of the root node is the

result of the query. ⨝ A.id=B.id

s value>100

A B
CMU 15-445/645 (Fall 2019)
 5

D I S K- O R I E N T E D D B M S

Just like it cannot assume that a table fits entirely

in memory, a disk-oriented DBMS cannot assume
that the results of a query fits in memory.

We are going use on the buffer pool to implement

algorithms that need to spill to disk.

We are also going to prefer algorithms that

maximize the amount of sequential access.

CMU 15-445/645 (Fall 2019)

 6

T O D AY ' S A G E N D A

External Merge Sort

Aggregations

CMU 15-445/645 (Fall 2019)

 7

WHY DO WE NEED TO SORT?

Tuples in a table have no specific order.

But queries often want to retrieve tuples in a

specific order.
→ Trivial to support duplicate elimination (DISTINCT).
→ Bulk loading sorted tuples into a B+Tree index is faster.
→ Aggregations (GROUP BY).

CMU 15-445/645 (Fall 2019)

 8

SORTING ALGORITHMS

If data fits in memory, then we can use a standard

sorting algorithm like quick-sort.

If data does not fit in memory, then we need to use

a technique that is aware of the cost of writing data
out to disk…

CMU 15-445/645 (Fall 2019)

 9

EXTERNAL MERGE SORT

Divide-and-conquer sorting algorithm that splits

the data set into separate runs and then sorts them
individually.

Phase #1 – Sorting
→ Sort blocks of data that fit in main-memory and then
write back the sorted blocks to a file on disk.

Phase #2 – Merging
→ Combine sorted sub-files into a single larger file.

CMU 15-445/645 (Fall 2019)

 10

2 -WAY E X T E R N A L M E R G E S O R T

We will start with a simple example of a 2-way

external merge sort.
→ "2" represents the number of runs that we are going to
merge into a new run for each pass.

Data set is broken up into N pages.

The DBMS has a finite number of B buffer pages

to hold input and output data.

CMU 15-445/645 (Fall 2019)

 11

2 -WAY E X T E R N A L M E R G E S O R T

Pass #0
→ Read every B pages of the table into memory
→ Sort pages into runs and write them back to disk.

Memory

Disk
Page #1 Page #2

CMU 15-445/645 (Fall 2019)

 11

2 -WAY E X T E R N A L M E R G E S O R T

Pass #0
→ Read every B pages of the table into memory
→ Sort pages into runs and write them back to disk.

Memory

Disk
Page #1 Page #2

CMU 15-445/645 (Fall 2019)

 11

2 -WAY E X T E R N A L M E R G E S O R T

Pass #0
→ Read every B pages of the table into memory
→ Sort pages into runs and write them back to disk.

Memory

Disk
Page #1 Page #2 Sorted Run

CMU 15-445/645 (Fall 2019)

 11

2 -WAY E X T E R N A L M E R G E S O R T

Pass #0
→ Read every B pages of the table into memory
→ Sort pages into runs and write them back to disk.

Memory Memory

Disk
Page #1 Page #2 Sorted Run

CMU 15-445/645 (Fall 2019)

 11

2 -WAY E X T E R N A L M E R G E S O R T

Pass #0
→ Read every B pages of the table into memory
→ Sort pages into runs and write them back to disk.

Memory Memory

Disk
Page #1 Page #2 Sorted Run

CMU 15-445/645 (Fall 2019)

 11

2 -WAY E X T E R N A L M E R G E S O R T

Pass #0
→ Read every B pages of the table into memory
→ Sort pages into runs and write them back to disk.

Pass #1,2,3,…
→ Recursively merges pairs of runs into runs twice as long.
→ Uses three buffer pages (2 for input pages, 1 for output).

Memory Memory Memory

Disk
Page #1 Page #2 Sorted Run

CMU 15-445/645 (Fall 2019)

 11

2 -WAY E X T E R N A L M E R G E S O R T

Pass #0
→ Read every B pages of the table into memory
→ Sort pages into runs and write them back to disk.

Pass #1,2,3,…
→ Recursively merges pairs of runs into runs twice as long.
→ Uses three buffer pages (2 for input pages, 1 for output).

Memory Memory Memory

Disk
Page #1 Page #2 Sorted Run

CMU 15-445/645 (Fall 2019)

 11

2 -WAY E X T E R N A L M E R G E S O R T

Pass #0
→ Read every B pages of the table into memory
→ Sort pages into runs and write them back to disk.

Pass #1,2,3,…
→ Recursively merges pairs of runs into runs twice as long.
→ Uses three buffer pages (2 for input pages, 1 for output).

Memory Memory Memory

Final Result
Disk
Page #1 Page #2 Sorted Run

CMU 15-445/645 (Fall 2019)

 12

2 -WAY E X T E R N A L M E R G E S O R T
EOF
3,4 6,2 9,4 8,7 5,6 3,1 2 ∅

In each pass, we read and PASS 1-PAGE

write each page in file.
3,4 2,6 4,9 7,8 5,6 1,3 2 ∅
#0 RUNS
PASS 2-PAGE
Number of passes
2,3 4,7 1,3 2
#1 4,6 8,9 5,6 ∅ RUNS

= 1 + ⌈ log2 N ⌉ PASS
2,3 1,2
4-PAGE
#2 RUNS
Total I/O cost 4,4
6,7
3,5
6

= 2N ∙ (# of passes) 8,9 ∅

PASS 1,2
8-PAGE
#3 2,3
RUNS
3,4
4,5
6,6
7,8
9
∅ CMU 15-445/645 (Fall 2019)
 13

2 -WAY E X T E R N A L M E R G E S O R T

This algorithm only requires three buffer pages to

perform the sorting (B=3).

But even if we have more buffer space available

(B>3), it does not effectively utilize them…

CMU 15-445/645 (Fall 2019)

 14

D O U B L E B U F F E R I N G O P T I M I Z AT I O N

Prefetch the next run in the background and store

it in a second buffer while the system is processing
the current run.
→ Reduces the wait time for I/O requests at each step by
continuously utilizing the disk.

Memory

Disk
Page #1 Page #2

CMU 15-445/645 (Fall 2019)

 14

D O U B L E B U F F E R I N G O P T I M I Z AT I O N

Prefetch the next run in the background and store

it in a second buffer while the system is processing
the current run.
→ Reduces the wait time for I/O requests at each step by
continuously utilizing the disk.

Memory

Disk
Page #1 Page #2

CMU 15-445/645 (Fall 2019)

 15

GENERAL EXTERNAL MERGE SORT

Pass #0
→ Use B buffer pages.
→ Produce ⌈N / B⌉ sorted runs of size B
Pass #1,2,3,…
→ Merge B-1 runs (i.e., K-way merge).

Number of passes = 1 + ⌈ logB-1 ⌈N / B⌉ ⌉

Total I/O Cost = 2N ∙ (# of passes)

CMU 15-445/645 (Fall 2019)

 15

GENERAL EXTERNAL MERGE SORT

Pass #0
→ Use B buffer pages.
→ Produce ⌈N / B⌉ sorted runs of size B
Pass #1,2,3,…
→ Merge B-1 runs (i.e., K-way merge).

Number of passes = 1 + ⌈ logB-1 ⌈N / B⌉ ⌉

Total I/O Cost = 2N ∙ (# of passes)

CMU 15-445/645 (Fall 2019)

 16

EXAMPLE

Sort 108 pages with 5 buffer pages: N=108, B=5

→ Pass #0: ⌈N / B⌉ = ⌈108 / 5⌉ = 22 sorted runs of 5 pages
each (last run is only 3 pages).
→ Pass #1: ⌈N’ / B-1⌉ = ⌈22 / 4⌉ = 6 sorted runs of 20 pages
each (last run is only 8 pages).
→ Pass #2: ⌈N’’ / B-1⌉ = ⌈6 / 4⌉ = 2 sorted runs, first one has
80 pages and second one has 28 pages.
→ Pass #3: Sorted file of 108 pages.

1+⌈ logB-1⌈N / B⌉ ⌉ = 1+⌈log4 22⌉ = 1+⌈2.229...⌉

= 4 passes
CMU 15-445/645 (Fall 2019)
 18

USING B+TREES FOR SORTING

If the table that must be sorted already has a

B+Tree index on the sort attribute(s), then we can
use that to accelerate sorting.

Retrieve tuples in desired sort order by simply

traversing the leaf pages of the tree.
Cases to consider:
→ Clustered B+Tree
→ Unclustered B+Tree

CMU 15-445/645 (Fall 2019)

 19

CASE #1 CLUSTERED B+TREE

B+Tree Index
Traverse to the left-most leaf page,
and then retrieve tuples from all leaf
pages.

This is always better than external 101 102 103 104

sorting because there is no Tuple Pages
computational cost and all disk access
is sequential.

CMU 15-445/645 (Fall 2019)

 20

CASE #2 UNCLUSTERED B+TREE

B+Tree Index
Chase each pointer to the page that
contains the data.

This is almost always a bad idea.

In general, one I/O per data record. 101 102 103 104
Tuple Pages

CMU 15-445/645 (Fall 2019)

 21

A G G R E G AT I O N S

Collapse multiple tuples into a single scalar value.

Two implementation choices:

→ Sorting
→ Hashing

CMU 15-445/645 (Fall 2019)

 22

S O R T I N G A G G R E G AT I O N
enrolled(sid,cid,grade)
SELECT DISTINCT cid sid cid grade
FROM enrolled 53666 15-445 C
WHERE grade IN ('B','C') 53688 15-721 A
53688 15-826 B
ORDER BY cid
53666 15-721 C
53655 15-445 C

sid cid grade cid cid

53666 15-445 C 15-445 15-445
53688 15-826 B 15-826 15-445
Filter 53666
53655
15-721
15-445
C
C Remove 15-721
15-445 Sort 15-721
15-826
Columns

CMU 15-445/645 (Fall 2019)

 22

S O R T I N G A G G R E G AT I O N
enrolled(sid,cid,grade)
SELECT DISTINCT cid sid cid grade
FROM enrolled 53666 15-445 C
WHERE grade IN ('B','C') 53688 15-721 A
53688 15-826 B
ORDER BY cid
53666 15-721 C
53655 15-445 C

sid cid grade cid cid

53666 15-445 C 15-445 15-445
53688 15-826 B 15-826 15-445
Filter 53666
53655
15-721
15-445
C
C Remove 15-721
15-445 Sort 15-721
15-826
Columns Eliminate
Dupes
CMU 15-445/645 (Fall 2019)
 22

S O R T I N G A G G R E G AT I O N
enrolled(sid,cid,grade)
SELECT DISTINCT cid sid cid grade
FROM enrolled 53666 15-445 C
WHERE grade IN ('B','C') 53688 15-721 A
53688 15-826 B
ORDER BY cid
53666 15-721 C
53655 15-445 C

sid cid grade cid cid

53666 15-445 C 15-445 15-445
53688 15-826 B 15-826 X
15-445
Filter 53666
53655
15-721
15-445
C
C Remove 15-721
15-445 Sort 15-721
15-826
Columns Eliminate
Dupes
CMU 15-445/645 (Fall 2019)
 23

A LT E R N AT I V E S T O S O R T I N G

What if we don’t need the data to be ordered?

→ Forming groups in GROUP BY (no ordering)
→ Removing duplicates in DISTINCT (no ordering)

Hashing is a better alternative in this scenario.

→ Only need to remove duplicates, no need for ordering.
→ Can be computationally cheaper than sorting.

CMU 15-445/645 (Fall 2019)

 24

H A S H I N G A G G R E G AT E

Populate an ephemeral hash table as the DBMS

scans the table. For each record, check whether
there is already an entry in the hash table:
→ DISTINCT: Discard duplicate.
→ GROUP BY: Perform aggregate computation.

If everything fits in memory, then it is easy.

If the DBMS must spill data to disk, then we need
to be smarter…

CMU 15-445/645 (Fall 2019)

 25

E X T E R N A L H A S H I N G A G G R E G AT E

Phase #1 – Partition
→ Divide tuples into buckets based on hash key.
→ Write them out to disk when they get full.

Phase #2 – ReHash
→ Build in-memory hash table for each partition and
compute the aggregation.

CMU 15-445/645 (Fall 2019)

 26

PHASE #1 PA R T I T I O N

Use a hash function h1 to split tuples into

partitions on disk.
→ We know that all matches live in the same partition.
→ Partitions are "spilled" to disk via output buffers.

Assume that we have B buffers.

We will use B-1 buffers for the partitions and 1
buffer for the input data.

CMU 15-445/645 (Fall 2019)

 27

PHASE #1 PA R T I T I O N
enrolled(sid,cid,grade)
SELECT DISTINCT cid sid cid grade
FROM enrolled 53666 15-445 C
WHERE grade IN ('B','C') 53688 15-721 A
53688 15-826 B
53666 15-721 C
53655 15-445 C
B-1 partitions
sid cid grade cid 15-445 15-445
15-445 15-445
53666 15-445 C 15-445
15-445

53688 15-826 B 15-826 h1 15-826

15-826
Filter 53666
53655
15-721
15-445
C
C Remove
15-721
15-445 ⋮
Columns ⋮
15-721

CMU 15-445/645 (Fall 2019)

 28

PHASE #2 REHASH

For each partition on disk:

→ Read it into memory and build an in-memory hash table
based on a second hash function h2.
→ Then go through each bucket of this hash table to bring
together matching tuples.

This assumes that each partition fits in memory.

CMU 15-445/645 (Fall 2019)

 29

PHASE #2 REHASH
enrolled(sid,cid,grade)
SELECT DISTINCT cid sid cid grade
FROM enrolled 53666 15-445 C
WHERE grade IN ('B','C') 53688 15-721 A
53688 15-826 B
Phase #1 Buckets 53666 15-721 C
53655 15-445 C
15-445 15-445
15-445 15-445
15-445

15-826
15-826

CMU 15-445/645 (Fall 2019)

 29

PHASE #2 REHASH
enrolled(sid,cid,grade)
SELECT DISTINCT cid sid cid grade
FROM enrolled 53666 15-445 C
WHERE grade IN ('B','C') 53688 15-721 A
53688 15-826 B
Phase #1 Buckets 53666 15-721 C
53655 15-445 C
15-445 15-445
15-445 15-445
15-445

15-826
15-826

CMU 15-445/645 (Fall 2019)

 29

PHASE #2 REHASH
enrolled(sid,cid,grade)
SELECT DISTINCT cid sid cid grade
FROM enrolled 53666 15-445 C
WHERE grade IN ('B','C') 53688 15-721 A
53688 15-826 B
Phase #1 Buckets 53666 15-721 C
Hash Table 53655 15-445 C
15-445 15-445
15-445 15-445
15-445
h2 cid
15-445
15-826
15-826

CMU 15-445/645 (Fall 2019)

 29

PHASE #2 REHASH
enrolled(sid,cid,grade)
SELECT DISTINCT cid sid cid grade
FROM enrolled 53666 15-445 C
WHERE grade IN ('B','C') 53688 15-721 A
53688 15-826 B
Phase #1 Buckets 53666 15-721 C
Hash Table 53655 15-445 C
15-445 15-445
15-445 15-445
15-445
h2 cid
15-445
15-826 15-826
15-826 h2
⋮

CMU 15-445/645 (Fall 2019)

 29

PHASE #2 REHASH
enrolled(sid,cid,grade)
SELECT DISTINCT cid sid cid grade
FROM enrolled 53666 15-445 C
WHERE grade IN ('B','C') 53688 15-721 A
53688 15-826 B
Phase #1 Buckets 53666 15-721 C
Hash Table 53655 15-445 C
15-445 15-445
15-445 15-445
15-445
h2 cid
15-445
15-826
15-826 h2 15-826 Final Result
⋮ cid
15-445
15-826

CMU 15-445/645 (Fall 2019)

 29

PHASE #2 REHASH
enrolled(sid,cid,grade)
SELECT DISTINCT cid sid cid grade
FROM enrolled 53666 15-445 C
WHERE grade IN ('B','C') 53688 15-721 A
53688 15-826 B
Phase #1 Buckets 53666 15-721 C
Hash Table 53655 15-445 C
15-445 15-445
15-445 15-445
15-445
h2 cid
15-445
15-826
15-826 h2 15-826 Final Result
⋮ cid
15-445
15-721 15-826

CMU 15-445/645 (Fall 2019)

 29

PHASE #2 REHASH
enrolled(sid,cid,grade)
SELECT DISTINCT cid sid cid grade
FROM enrolled 53666 15-445 C
WHERE grade IN ('B','C') 53688 15-721 A
53688 15-826 B
Phase #1 Buckets 53666 15-721 C
53655 15-445 C
15-445 15-445
15-445 15-445
15-445
h2
15-826
15-826 h2 Final Result
⋮ Hash Table cid
cid 15-445
15-721
h2 15-721
15-826
15-721

CMU 15-445/645 (Fall 2019)

 30

H A S H I N G S U M M A R I Z AT I O N

During the ReHash phase, store pairs of the form

(GroupKey→RunningVal)

When we want to insert a new tuple into the hash

table:
→ If we find a matching GroupKey, just update the
RunningVal appropriately
→ Else insert a new GroupKey→RunningVal

CMU 15-445/645 (Fall 2019)

 31

H A S H I N G S U M M A R I Z AT I O N

SELECT cid, AVG(s.gpa)

FROM student AS s, enrolled AS e
WHERE s.sid = e.sid
GROUP BY cid

15-445
15-445 h2 Hash Table
15-826 key value
Phase #1 h2 15-445 (2, 7.32)
Buckets
⋮ 15-826 (1, 3.33)
15-721 15-721 (1, 2.89)
h2

CMU 15-445/645 (Fall 2019)

 31

H A S H I N G S U M M A R I Z AT I O N
Running Totals
SELECT cid, AVG(s.gpa) AVG(col) → (COUNT,SUM)
FROM student AS s, enrolled AS e MIN(col) → (MIN)
WHERE s.sid = e.sid MAX(col) → (MAX)
GROUP BY cid SUM(col) → (SUM)
COUNT(col) → (COUNT)
15-445
15-445 h2 Hash Table
15-826 key value
Phase #1 h2 15-445 (2, 7.32)
Buckets
⋮ 15-826 (1, 3.33)
15-721 15-721 (1, 2.89)
h2

CMU 15-445/645 (Fall 2019)

 31

H A S H I N G S U M M A R I Z AT I O N
Running Totals
SELECT cid, AVG(s.gpa) AVG(col) → (COUNT,SUM)
FROM student AS s, enrolled AS e MIN(col) → (MIN)
WHERE s.sid = e.sid MAX(col) → (MAX)
GROUP BY cid SUM(col) → (SUM)
COUNT(col) → (COUNT)
15-445
15-445 h2 Hash Table Final Result
15-826 key value cid AVG(gpa)
Phase #1 h2 15-445 (2, 7.32) 15-445 3.66
Buckets
⋮ 15-826 (1, 3.33) 15-826 3.33
15-721 15-721 (1, 2.89) 15-721 2.89
h2

CMU 15-445/645 (Fall 2019)

 32

C O S T A N A LY S I S

How big of a table can we hash using this

approach?
→ B-1 "spill partitions" in Phase #1
→ Each should be no more than B blocks big

Answer: B ∙ (B-1)
→ A table of N pages needs about sqrt(N) buffers
→ Assumes hash distributes records evenly.
Use a "fudge factor" f>1 for that: we need
B ∙ sqrt(f ∙ N)

CMU 15-445/645 (Fall 2019)

 33

CONCLUSION

Choice of sorting vs. hashing is subtle and depends

on optimizations done in each case.

We already discussed the optimizations for

sorting:
→ Chunk I/O into large blocks to amortize seek+RD costs.
→ Double-buffering to overlap CPU and I/O.

CMU 15-445/645 (Fall 2019)

 34

PROJECT #2

You will build a thread-safe linear

probing hash table that supports
automatic resizing.

We define the API for you. You need

to provide the implementation.

https://15445.courses.cs.cmu.edu/fall2019/project2/

CMU 15-445/645 (Fall 2019)

 35

PROJECT #2 TA S K S

Page Layouts
Hash Table Implementation
Table Resizing
Concurrency Control Protocol

CMU 15-445/645 (Fall 2019)

 36

DEVELOPMENT HINTS

Follow the textbook semantics and algorithms.

You should make sure your page layout are

working correctly before switching to the actual
hash table itself.

Then focus on the single-threaded use case first.

Avoid premature optimizations.

→ Correctness first, performance second.

CMU 15-445/645 (Fall 2019)

 37

THINGS TO NOTE

Do not change any file other than the ones that

you submit to Gradescope.

Rebase on top of the latest BusTub master branch.

Post your questions on Piazza or come to TA

office hours.

CMU 15-445/645 (Fall 2019)

 38

P L A G I A R I S M WA R N I N G

Your project implementation must be

your own work.
→ You may not copy source code from other
groups or the web.
→ Do not publish your implementation on
Github.

Plagiarism will not be tolerated.

See CMU's Policy on Academic
Integrity for additional information.

CMU 15-445/645 (Fall 2019)

 39

NEXT CLASS

Nested Loop Join

Sort-Merge Join
Hash Join

CMU 15-445/645 (Fall 2019)