Distributed Database

Management Systems
 Outline
Introduction
Distributed DBMS Architecture
Distributed Database Design
Distributed Query Processing
Distributed Concurrency Control
Distributed Reliability Protocols

Distributed DBMS 2
 Outline
Introduction
What is a distributed DBMS
Problems
Current state-of-affairs
Distributed DBMS Architecture
Distributed Database Design
Distributed Query Processing
Distributed Concurrency Control
Distributed Reliability Protocols

Distributed DBMS 3
 Motivation

Database Computer
Technology Networks
integration distribution

D istr ibuted
Database
Systems
integration

integration ≠ centralization
Distributed DBMS 4
 What is a Distributed Database
System?
A distributed database (DDB) is a collection of multiple,
logically interrelated databases distributed over a computer
network.

A distributed database management system (D–DBMS) is

the software that manages the DDB and provides an
access mechanism that makes this distribution transparent
to the users.

Distributed database system (DDBS) = DDB + D–DBMS

Distributed DBMS 5
 What is not a DDBS?

A timesharing computer system

A loosely or tightly coupled multiprocessor system
A database system which resides at one of the
nodes of a network of computers - this is a
centralized database on a network node

Distributed DBMS 6
 Centralized DBMS on a Network

Site 1
Site 2

Site 5

Communication
Network

Site 4 Site 3

Distributed DBMS 7
 Distributed DBMS Environment

Site 1
Site 2

Site 5

Communication
Network

Site 4 Site 3

Distributed DBMS 8
 Implicit Assumptions
Data stored at a number of sites each site
logically consists of a single processor.
Processors at different sites are interconnected
by a computer network no multiprocessors
parallel database systems
Distributed database is a database, not a
collection of files data logically related as
exhibited in the users’ access patterns
relational data model
D-DBMS is a full-fledged DBMS
not remote file system, not a TP system

Distributed DBMS 9
 Distributed DBMS Promises

Transparent management of distributed,

fragmented, and replicated data

Improved reliability/availability through distributed

transactions

Improved performance

Easier and more economical system expansion

Distributed DBMS 10
 Transparency
Transparency is the separation of the higher level
semantics of a system from the lower level
implementation issues.
Fundamental issue is to provide
data independence
in the distributed environment
Network (distribution) transparency

Replication transparency

Fragmentation transparency
horizontal fragmentation: selection
vertical fragmentation: projection
hybrid

Distributed DBMS 11
Example
EMP ASG
ENO ENAME TITLE ENO PNO RESP DUR
E1 J. Doe Elect. Eng. E1 P1 Manager 12
E2 M. Smith Syst. Anal. E2 P1 Analyst 24
E3 A. Lee Mech. Eng. E2 P2 Analyst 6
E4 J. Miller Programmer E3 P3 Consultant 10
E5 B. Casey Syst. Anal. E3 P4 Engineer 48
E6 L. Chu Elect. Eng. E4 P2 Programmer 18
E7 R. Davis Mech. Eng. E5 P2 Manager 24
E8 J. Jones Syst. Anal. E6 P4 Manager 48
E7 P3 Engineer 36
E7 P5 Engineer 23
E8 P3 Manager 40

PROJ PAY
PNO PNAME BUDGET TITLE SAL
P1 Instrumentation 150000 Elect. Eng. 40000
P2 Database Develop. 135000 Syst. Anal. 34000
P3 CAD/CAM 250000 Mech. Eng. 27000
P4 Maintenance 310000 Programmer 24000

Distributed DBMS 12
 Transparent Access
SELECT ENAME,SAL
Tokyo
FROM EMP,ASG,PAY
WHERE DUR > 12
Boston Paris
AND EMP.ENO = ASG.ENO
Paris projects
AND PAY.TITLE = EMP.TITLE
Paris employees
Communication Paris assignments
Network Boston employees
Boston projects
Boston employees
Boston assignments
Montreal
New
Montreal projects
York
Paris projects
Boston projects New York projects
New York employees with budget > 200000
New York projects Montreal employees
New York assignments Montreal assignments
Distributed DBMS 13
 Distributed Database –
User View

Distributed Database

Distributed DBMS 14
 Distributed DBMS - Reality
User
Query
DBMS
Software
User
Application
DBMS
Software

DBMS Communication
Software Subsystem

User
DBMS User Application
Software Query DBMS
Software

User
Query
Distributed DBMS 15
 Potentially Improved
Performance

Proximity of data to its points of use

Requires some support for fragmentation and replication

Parallelism in execution
Inter-query parallelism

Intra-query parallelism

Distributed DBMS 16
 Parallelism Requirements

Have as much of the data required by each

application at the site where the application
executes
Full replication

How about updates?

Updates to replicated data requires implementation of
distributed concurrency control and commit protocols

Distributed DBMS 17
 System Expansion

Issue is database scaling

Emergence of microprocessor and workstation

technologies
Demise of Grosh's law

Client-server model of computing

Data communication cost vs telecommunication

cost

Distributed DBMS 18
Distributed DBMS Issues
Distributed Database Design
how to distribute the database
replicated & non-replicated database distribution
a related problem in directory management

Query Processing
convert user transactions to data manipulation instructions
optimization problem
min{cost = data transmission + local processing}
general formulation is NP-hard

Distributed DBMS 19
 Distributed DBMS Issues

Concurrency Control
synchronization of concurrent accesses
consistency and isolation of transactions' effects
deadlock management

Reliability
how to make the system resilient to failures
atomicity and durability

Distributed DBMS 20
 Relationship Between Issues
Directory
Management

Query Distribution
Reliabili ty
Processing Design

Concurrency
Control

Deadlock
Management
Distributed DBMS 21
 Outline
Introduction
Distributed DBMS Architecture
Distributed Database Design
Fragmentation
Data Placement

Distributed Query Processing

Distributed Concurrency Control
Distributed Reliability Protocols

Distributed DBMS 22
 Design Problem
In the general setting :
Making decisions about the placement of data and programs
across the sites of a computer network as well as possibly
designing the network itself.

In Distributed DBMS, the placement of

applications entails
placement of the distributed DBMS software; and
placement of the applications that run on the database

Distributed DBMS 23
 Distribution Design

Top-down
mostly in designing systems from scratch

mostly in homogeneous systems

Bottom-up
when the databases already exist at a number of sites

Distributed DBMS 24
 Top-Down Design
Requirements
Analysis

Objectives
User Input
Conceptual View Integration View Design
Design

Access
GCS Information ES’s

Distribution
Design User Input

LCS’s

Physical
Design

LIS’s

Distributed DBMS 25
 Distribution Design Issues
Why fragment at all?

How to fragment?

How much to fragment?

How to test correctness?

How to allocate?

Information requirements?

Distributed DBMS 26
 Fragmentation
Can't we just distribute relations?
What is a reasonable unit of distribution?
relation
views are subsets of relations ⇒ locality
extra communication
fragments of relations (sub-relations)
concurrent execution of a number of transactions that
access different portions of a relation
views that cannot be defined on a single fragment will
require extra processing
semantic data control (especially integrity enforcement)
more difficult

Distributed DBMS 27
 Fragmentation Alternatives –
Horizontal
PROJ
PROJ1 : projects with budgets less PNO PNAME BUDGET LOC

than $200,000 P1 Instrumentation 150000 Montreal

P2 Database Develop. 135000 New York
P3 CAD/CAM 250000 New
New York
PROJ2 : projects with budgets P4 Maintenance 310000 Paris
York

greater than or equal to P5 CAD/CAM 500000 Boston

$200,000
PROJ1 PROJ2

PNO PNAME BUDGET LOC PNO PNAME BUDGET LOC

P1 Instrumentation 150000 Montreal P3 CAD/CAM 250000 New York
P2 Database Develop. 135000 New York P4 Maintenance 310000 Paris
P5 CAD/CAM 500000 Boston

Distributed DBMS 28
 Fragmentation Alternatives –
Vertical
PROJ
PROJ1: information about PNO PNAME BUDGET LOC

project budgets P1 Instrumentation 150000 Montreal

P2 Database Develop. 135000 New York
P3 CAD/CAM 250000 New
New York
PROJ2: information about P4 Maintenance 310000 Paris
York

project names and P5 CAD/CAM 500000 Boston

locations

PROJ1 PROJ2

PNO BUDGET PNO PNAME LOC

P1 150000 P1 Instrumentation Montreal

P2 135000 P2 Database Develop. New York
P3 250000 P3 CAD/CAM New York
P4 310000 P4 Maintenance Paris
P5 500000 P5 CAD/CAM Boston

Distributed DBMS 29
 Degree of Fragmentation
finite number of alternat ives

tuples relations
or
attributes

Finding the suitable level of par ti tioning

within this range

Distributed DBMS 30
 Correctness of Fragmentation
Completeness
Decomposition of relation R into fragments R1, R2, ..., Rn is complete
iff each data item in R can also be found in some Ri

Reconstruction
If relation R is decomposed into fragments R1, R2, ..., Rn, then there
should exist some relational operator ∇ such that
R = ∇1≤i≤nRi
Disjointness
If relation R is decomposed into fragments R1, R2, ..., Rn, and data
item di is in Rj, then di should not be in any other fragment Rk (k ≠ j ).

Distributed DBMS 31
 Allocation Alternatives
Non-replicated
partitioned : each fragment resides at only one site
Replicated
fully replicated : each fragment at each site
partially replicated : each fragment at some of the sites

Rule of thumb:

read - only queries

If ≥1 replication is advantageous,
update queries
otherwise replication may cause problems

Distributed DBMS 32
 Fragmentation

Horizontal Fragmentation (HF)

Primary Horizontal Fragmentation (PHF)
Derived Horizontal Fragmentation (DHF)

Vertical Fragmentation (VF)

Hybrid Fragmentation (HF)

Distributed DBMS 33
 Primary Horizontal
Fragmentation
Definition :
Rj = σFj (R ), 1 ≤ j ≤ w
where Fj is a selection formula.
Therefore,
A horizontal fragment Ri of relation R consists of all
the tuples of R which satisfy a predicate pi.

⇓
Given a set of predicates M, there are as many
horizontal fragments of relation R as there are
predicates.
Distributed DBMS 34
PHF – Example
SKILL
TITLE, SAL

L1
EMP PROJ
ENO, ENAME, TITLE PNO, PNAME, BUDGET, LOC

L2 L3
ASG
ENO, PNO, RESP, DUR

Two candidate relations : PAY and PROJ

Distributed DBMS 35
 PHF – Example

PAY1 PAY2
TITLE SAL TITLE SAL
Mech. Eng. 27000 Elect. Eng. 40000
Programmer 24000 Syst. Anal. 34000

Distributed DBMS 36
 PHF – Example

PROJ1 PROJ2

PNO PNAME BUDGET LOC PNO PNAME BUDGET LOC

Database
P1 Instrumentation 150000 Montreal P2 135000 New York
Develop.

PROJ4 PROJ6

PNO PNAME BUDGET LOC PNO PNAME BUDGET LOC

P3 CAD/CAM 250000 New York P4 Maintenance 310000 Paris

Distributed DBMS 37
 PHF – Correctness
Completeness
Since the set of predicates is complete and minimal, the
selection predicates are complete

Reconstruction
If relation R is fragmented into FR = {R1,R2,…,Rr}

R = ∪∀Ri ∈FR Ri
Disjointness
Predicates that form the basis of fragmentation should be
mutually exclusive.

Distributed DBMS 38
 Derived Horizontal
Fragmentation
Defined on a member relation of a link according
to a selection operation specified on its owner.
Each link is an equijoin.
Equijoin can be implemented by means of semijoins.

TITLE, SAL

L1
EMP PROJ
ENO, ENAME, TITLE PNO, PNAME, BUDGET, LOC

L2 L3
ASG
ENO, PNO, RESP, DUR

Distributed DBMS 39
 DHF – Definition
Given a link L where owner(L)=S and member(L)=R, the
derived horizontal fragments of R are defined as
Ri = R F Si, 1≤i≤w

where w is the maximum number of fragments that will

be defined on R and
Si = σFi (S)

where Fi is the formula according to which the primary

horizontal fragment Si is defined.

Distributed DBMS 40
 DHF – Example
Given link L1 where owner(L1)=SKILL and
member(L1)=EMP
EMP1 = EMP SKILL1
EMP2 = EMP SKILL2
where
SKILL1 = σ SAL≤30000 (SKILL)
SKILL2 = σSAL>30000 (SKILL)
EMP1 EMP2
ENO ENAME TITLE ENO ENAME TITLE

E3 A. Lee Mech. Eng. E1 J. Doe Elect. Eng.

E4 J. Miller Programmer E2 M. Smith Syst. Anal.
E7 R. Davis Mech. Eng. E5 B. Casey Syst. Anal.
E6 L. Chu Elect. Eng.
E8 J. Jones Syst. Anal.
Distributed DBMS 41
 DHF – Correctness
Completeness
Referential integrity
Let R be the member relation of a link whose owner is relation
S which is fragmented as FS = {S1, S2, ..., Sn}. Furthermore,
let A be the join attribute between R and S. Then, for each
tuple t of R, there should be a tuple t' of S such that
t[A]=t'[A]
Reconstruction
Same as primary horizontal fragmentation.
Disjointness
Simple join graphs between the owner and the member
fragments.

Distributed DBMS 42
 Vertical Fragmentation
Has been studied within the centralized context
design methodology
physical clustering

More difficult than horizontal, because more

alternatives exist.
Two approaches :
grouping
attributes to fragments
splitting
relation to fragments

Distributed DBMS 43
 Vertical Fragmentation
Overlapping fragments
grouping
Non-overlapping fragments
splitting
We do not consider the replicated key attributes to be
overlapping.
Advantage:
Easier to enforce functional dependencies (for integrity
checking etc.)

Distributed DBMS 44
 VF – Correctness
A relation R, defined over attribute set A and key K,
generates the vertical partitioning FR = {R1, R2, …, Rr}.
Completeness
The following should be true for A:

A =∪ AR
i

Reconstruction
Reconstruction can be achieved by
R= K Ri ∀Ri ∈FR
Disjointness
TID's are not considered to be overlapping since they are maintained by
the system
Duplicated keys are not considered to be overlapping

Distributed DBMS 45
 Fragment Allocation
Problem Statement
Given
F = {F1, F2, …, Fn} fragments
S ={S1, S2, …, Sm} network sites
Q = {q1, q2,…, qq} applications
Find the "optimal" distribution of F to S.
Optimality
Minimal cost
Communication + storage + processing (read & update)
Cost in terms of time (usually)
Performance
Response time and/or throughput
Constraints
Per site constraints (storage & processing)
Distributed DBMS 46
 Allocation Model
General Form
min(Total Cost)
subject to
response time constraint
storage constraint
processing constraint

Decision Variable
1 if fragment F i is stored at site Sj
x ij = 
0 otherwise
Distributed DBMS 47
 Allocation Model
Total Cost

∑ query processing cost +

all queries

∑ all sites ∑ all fragments

cost of storing a fragment at a site

Storage Cost (of fragment Fj at Sk)

(unit storage cost at Sk) ∗ (size of F j) ∗x jk

Query Processing Cost (for one query)

processing component + transmission component

Distributed DBMS 48
 Allocation Model
Query Processing Cost
Processing component
access cost + integrity enforcement cost + concurrency control cost
Access cost

∑ all sites ∑ all fragments

(no. of update accesses+ no. of read accesses) ∗

x ij ∗local processing cost at a site

Integrity enforcement and concurrency control costs

Can be similarly calculated

Distributed DBMS 49
 Allocation Model
Query Processing Cost
Transmission component
cost of processing updates + cost of processing retrievals
Cost of updates

∑ ∑
all sites all fragments
update message cost +

∑ all sites ∑ all fragments

acknowledgment cost

Retrieval Cost

∑ all fragments
min all sites (cost of retrieval command +

cost of sending back the result)

Distributed DBMS 50
 Allocation Model
Constraints
Response Time
execution time of query ≤ max. allowable response time for
that query
Storage Constraint (for a site)

∑ all fragments
storage requirement of a fragment at that site ≤

storage capacity at that site

Processing constraint (for a site)

∑ all queries
processing load of a query at that site ≤

processing capacity of that site

Distributed DBMS 51
 Allocation Model
Solution Methods
FAP is NP-complete
DAP also NP-complete

Heuristics based on
single commodity warehouse location (for FAP)
knapsack problem
branch and bound techniques
network flow

Distributed DBMS 52
 Allocation Model
Attempts to reduce the solution space
assume all candidate partitionings known; select the “best”
partitioning

ignore replication at first

sliding window on fragments

Distributed DBMS 53
 Outline
Introduction
Distributed DBMS Architecture
Distributed Database Design
Distributed Query Processing
Query Processing Methodology
Distributed Query Optimization
Distributed Concurrency Control
Distributed Reliability Protocols

Distributed DBMS 54
 Query Processing
high level user query

query
processor

low level data manipulat ion

commands

Distributed DBMS 55
 Query Processing Components
Query language that is used
SQL: “intergalactic dataspeak”
Query execution methodology
The steps that one goes through in executing high-level
(declarative) user queries.

Query optimization
How do we determine the “best” execution plan?

Distributed DBMS 56
 Selecting Alternatives
SELECT ENAME
FROM EMP,ASG
WHERE EMP.ENO = ASG.ENO
AND DUR > 37
Strategy 1
ΠENAME(σDUR>37∧EMP.ENO=ASG.ENO (EMP × ASG))
Strategy 2
ΠENAME(EMP ENO (σDUR>37 (ASG)))

Strategy 2 avoids Cartesian product, so is “better”

Distributed DBMS 57
 What is the Problem?
Site 1 Site 2 Site 3 Site 4 Site 5
ASG1=σENO≤“E3”(ASG) ASG2=σENO>“E3”(ASG) EMP1=σENO≤“E3”(EMP) EMP2=σENO>“E3”(EMP) Result

Site 5 Site 5
result = EMP1’∪EMP2’ result2=(EMP1∪ EMP2) ENOσDUR>37(ASG1∪ ASG1)
EMP1’ EMP2’
ASG1 ASG2 EMP1 EMP2
Site 3 Site 4
EMP1’=EMP1 ENOASG1
’ EMP2’=EMP2 ENOASG2
’
Site 1 Site 2 Site 3 Site 4

ASG1’ ASG2’
Site 1 Site 2
ASG1’=σDUR>37(ASG1) ASG2’=σDUR>37(ASG2)

Distributed DBMS 58
 Cost of Alternatives
Assume:
size(EMP) = 400, size(ASG) = 1000
tuple access cost = 1 unit; tuple transfer cost = 10 units
Strategy 1
produce ASG': (10+10)∗tuple access cost 20
transfer ASG' to the sites of EMP: (10+10)∗tuple transfer cost 200
produce EMP': (10+10) ∗tuple access cost∗2 40
transfer EMP' to result site: (10+10) ∗tuple transfer cost 200
Total cost 460
Strategy 2
transfer EMP to site 5:400∗tuple transfer cost 4,000
transfer ASG to site 5 :1000∗tuple transfer cost 10,000
produce ASG':1000∗tuple access cost 1,000
join EMP and ASG':400∗20∗tuple access cost 8,000
Total cost 23,000
Distributed DBMS 59
 Query Optimization Objectives
Minimize a cost function
I/O cost + CPU cost + communication cost
These might have different weights in different
distributed environments
Wide area networks
communication cost will dominate
low bandwidth
low speed
high protocol overhead
most algorithms ignore all other cost components
Local area networks
communication cost not that dominant
total cost function should be considered
Can also maximize throughput
Distributed DBMS 60
 Query Optimization Issues –
Types of Optimizers
Exhaustive search
cost-based
optimal
combinatorial complexity in the number of relations
Heuristics
not optimal
regroup common sub-expressions
perform selection, projection first
replace a join by a series of semijoins
reorder operations to reduce intermediate relation size
optimize individual operations

Distributed DBMS 61
 Query Optimization Issues –
Optimization Granularity
Single query at a time
cannot use common intermediate results
Multiple queries at a time
efficient if many similar queries
decision space is much larger

Distributed DBMS 62
 Query Optimization Issues –
Optimization Timing
Static
compilation ⇒ optimize prior to the execution
difficult to estimate the size of the intermediate results ⇒
error propagation
can amortize over many executions
R*
Dynamic
run time optimization
exact information on the intermediate relation sizes
have to reoptimize for multiple executions
Distributed INGRES
Hybrid
compile using a static algorithm
if the error in estimate sizes > threshold, reoptimize at run
time
MERMAID
Distributed DBMS 63
 Query Optimization Issues –
Statistics
Relation
cardinality
size of a tuple
fraction of tuples participating in a join with another relation

Attribute
cardinality of domain
actual number of distinct values

Common assumptions
independence between different attribute values
uniform distribution of attribute values within their domain

Distributed DBMS 64
 Query Optimization
Issues – Decision Sites
Centralized
single site determines the “best” schedule
simple
need knowledge about the entire distributed database

Distributed
cooperation among sites to determine the schedule
need only local information
cost of cooperation

Hybrid
one site determines the global schedule
each site optimizes the local subqueries

Distributed DBMS 65
 Query Optimization Issues –
Network Topology
Wide area networks (WAN) – point-to-point
characteristics
low bandwidth
low speed
high protocol overhead
communication cost will dominate; ignore all other cost factors
global schedule to minimize communication cost
local schedules according to centralized query optimization

Local area networks (LAN)

communication cost not that dominant
total cost function should be considered
broadcasting can be exploited (joins)
special algorithms exist for star networks
Distributed DBMS 66
Distributed Query Processing
Methodology
Calculus Query on Distributed
Relations

Query GLOBAL
Query GLOBAL
Decomposition
Decomposition SCHE
SCHEMA
MA

Algebraic Query on Distributed

Relations
CONTROL
FRAG
SITE Data
Data FRAGMEN
MENTT
Localization SCHE MA
Localization SCHE MA

Fragment Query

STATS
Global
Global STATSON
ON
FRAG
Optimization
Optimization FRAGMEN
MENTS
TS

Optimized Fragment Query

with Communicat ion Operations

LOCAL Local LOCAL

Local LOCAL
SCHE
SITES Optimization
Optimization SCHEMAS
MAS

Optimized Local
Queries

Distributed DBMS 67
 Step 1 – Query Decomposition
Input : Calculus query on global relations
Normalization
manipulate query quantifiers and qualification
Analysis
detect and reject “incorrect” queries
possible for only a subset of relational calculus
Simplification
eliminate redundant predicates
Restructuring
calculus query ⇒ algebraic query
more than one translation is possible
use transformation rules
Distributed DBMS 68
 Normalization
Lexical and syntactic analysis
check validity (similar to compilers)
check for attributes and relations
type checking on the qualification
Put into normal form
Conjunctive normal form
(p11∨p12∨…∨p1n) ∧…∧ (pm1∨pm2∨…∨pmn)
Disjunctive normal form
(p11∧p12 ∧…∧p1n) ∨…∨ (pm1 ∧pm2∧…∧ pmn)
OR's mapped into union
AND's mapped into join or selection

Distributed DBMS 69
 Analysis
Refute incorrect queries
Type incorrect
If any of its attribute or relation names are not defined
in the global schema
If operations are applied to attributes of the wrong type
Semantically incorrect
Components do not contribute in any way to the
generation of the result
Only a subset of relational calculus queries can be
tested for correctness
Those that do not contain disjunction and negation
To detect
connection graph (query graph)
join graph

Distributed DBMS 70
 Simplification
Why simplify?
Remember the example
How? Use transformation rules
elimination of redundancy
idempotency rules
p1 ∧ ¬( p1) ⇔ false
p1 ∧ (p1 ∨ p2) ⇔ p1
p1 ∨ false ⇔ p1
…
application of transitivity
use of integrity rules

Distributed DBMS 71
 Simplification – Example
SELECT TITLE
FROM EMP
WHERE EMP.ENAME = “J. Doe”
OR (NOT(EMP.TITLE = “Programmer”)
AND (EMP.TITLE = “Programmer”
OR EMP.TITLE = “Elect. Eng.”)
AND NOT(EMP.TITLE = “Elect. Eng.”))

⇓
SELECT TITLE
FROM EMP
WHERE EMP.ENAME = “J. Doe”

Distributed DBMS 72
 Restructuring
Convert relational calculus to ΠENAME Project
relational algebra
Make use of query trees
σDUR=12 OR DUR=24
Example
Find the names of employees other than J.
Doe who worked on the CAD/CAM project σPNAME=“CAD/CAM” Select
for either 1 or 2 years.
σENAME≠“J. DOE”
SELECT ENAME
FROM EMP, ASG, PROJ
WHERE EMP.ENO = ASG.ENO PNO
AND ASG.PNO = PROJ.PNO
AND ENAME ≠ “J. Doe” ENO Join
AND PNAME = “CAD/CAM”
AND (DUR = 12 OR DUR = 24)
PROJ ASG EMP

Distributed DBMS 73
 Restructuring –Transformation
Rules
Commutativity of binary operations
R×S⇔S×R
R S⇔S R
R∪S⇔S∪R
Associativity of binary operations
( R × S ) × T ⇔ R × (S × T)
(R S) T⇔R (S T)
Idempotence of unary operations
ΠA’(ΠA’(R)) ⇔ ΠA’(R)
σp1(A1)(σp2(A2)(R)) = σp1(A1) ∧ p2(A2)(R)
where R[A] and A' ⊆ A, A" ⊆ A and A' ⊆ A"
Commuting selection with projection
Distributed DBMS 74
 Restructuring –
Transformation Rules
Commuting selection with binary operations
σp(A)(R × S) ⇔ (σp(A) (R)) × S
σp(Ai)(R (Aj,Bk) S) ⇔ (σp(Ai) (R)) (Aj,Bk) S
σp(Ai)(R ∪ T) ⇔ σp(Ai) (R) ∪ σp(Ai) (T)
where Ai belongs to R and T

Commuting projection with binary operations

ΠC(R × S) ⇔ Π A’(R) × Π B’(S)
ΠC(R (Aj,Bk) S) ⇔ Π A’(R) (Aj,Bk) Π B’(S)
ΠC(R ∪ S) ⇔ Π C (R) ∪ Π C (S)
where R[A] and S[B]; C = A' ∪ B' where A' ⊆ A, B' ⊆ B

Distributed DBMS 75
 Example
Recall the previous example: ΠENAME Project
Find the names of employees other than J.
Doe who worked on the CAD/CAM project
for either one or two years. σDUR=12 OR DUR=24

SELECT ENAME σPNAME=“CAD/CAM” Select

FROM PROJ, ASG, EMP
WHERE ASG.ENO=EMP.ENO
AND ASG.PNO=PROJ.PNO σENAME≠“J. DOE”
AND ENAME≠“J. Doe”
AND PROJ.PNAME=“CAD/CAM” PNO
AND (DUR=12 OR DUR=24)

ENO Join

PROJ ASG EMP

Distributed DBMS 76
 Equivalent Query
ΠENAME

σPNAME=“CAD/CAM” ∧(DUR=12 ∨ DUR=24) ∧ ENAME≠“J. DOE”

PNO ∧ENO

ASG PROJ EMP

Distributed DBMS 77
 Restructuring
ΠENAME

PNO

ΠPNO,ENAME

ENO

ΠPNO ΠPNO,ENO ΠPNO,ENAME

σPNAME = "CAD/CAM" σDUR =12 ∧ DUR=24 σENAME ≠ "J. Doe"

PROJ ASG EMP

Distributed DBMS 78
 Step 2 – Data Localization

Input: Algebraic query on distributed relations

Determine which fragments are involved
Localization program
substitute for each global query its materialization program
optimize

Distributed DBMS 79
 Example
ΠENAME
Assume
EMP is fragmented into EMP1, EMP2,
σDUR=12 OR DUR=24
EMP3 as follows:
EMP1=σENO≤“E3”(EMP)
σPNAME=“CAD/CAM”
EMP2= σ“E3”<ENO≤“E6”(EMP)
EMP3=σENO≥“E6”(EMP) σENAME≠“J. DOE”
ASG fragmented into ASG1 and ASG2
as follows:
PNO
ASG1=σENO≤“E3”(ASG)
ASG2=σENO>“E3”(ASG) ENO

Replace EMP by (EMP1∪EMP2∪EMP3 ) PROJ ∪ ∪

and ASG by (ASG1 ∪ ASG2) in any
query EMP1 EMP2 EMP3 ASG1 ASG2
Distributed DBMS 80
 Provides Parallellism

ENO ENO ENO ENO

EMP1 ASG1 EMP2 ASG2 EMP3 ASG1 EMP3 ASG2

Distributed DBMS 81
 Eliminates Unnecessary Work

ENO ENO ENO

EMP1 ASG1 EMP2 ASG2 EMP3 ASG2

Distributed DBMS 82
 Reduction for PHF
Reduction with selection
Relation R and FR={R1, R2, …, Rw} where Rj=σ p (R)
j

σ p (Rj)= φ if ∀x in R: ¬(pi(x) ∧ pj(x))

i
Example
SELECT *
FROM EMP
WHERE ENO=“E5”

σ ENO=“E5” σ ENO=“E5”

EMP1 EMP2 EMP3 EMP2

Distributed DBMS 83
 Reduction for PHF
Reduction with join
Possible if fragmentation is done on join attribute
Distribute join over union

(R1 ∪ R2) S ⇔ (R1 S) ∪ (R2 S)

Given Ri = σpi(R) and Rj = σpj(R)

Ri Rj = φ if ∀x in Ri, ∀y in Rj: ¬(pi(x) ∧ pj(y))

Distributed DBMS 84
 Reduction for PHF
Reduction with join - Example
Assume EMP is fragmented as before and
ASG1: σENO ≤ "E3"(ASG)
ASG2: σENO > "E3"(ASG)
Consider the query
SELECT*
FROM EMP, ASG
WHERE EMP.ENO=ASG.ENO
ENO

∪ ∪

EMP1 EMP2 EMP3 ASG1 ASG2

Distributed DBMS 85
 Reduction for PHF
Reduction with join - Example
Distribute join over unions
Apply the reduction rule

ENO ENO ENO

EMP1 ASG1 EMP2 ASG2 EMP3 ASG2

Distributed DBMS 86
 Reduction for VF
Find useless (not empty) intermediate relations
Relation R defined over attributes A = {A1, ..., An} vertically fragmented
as Ri = ΠA' (R) where A' ⊆ A:
ΠD,K(Ri) is useless if the set of projection attributes D is not in A'
Example: EMP1= ΠENO,ENAME (EMP); EMP2= ΠENO,TITLE (EMP)
SELECT ENAME
FROM EMP

ΠENAME ΠENAME

ENO ⇒
EMP1 EMP2 EMP1
Distributed DBMS 87
 Step 3 – Global Query
Optimization
Input: Fragment query
Find the best (not necessarily optimal) global
schedule
Minimize a cost function
Distributed join processing
Bushy vs. linear trees
Which relation to ship where?
Ship-whole vs ship-as-needed
Decide on the use of semijoins
Semijoin saves on communication at the expense of
more local processing.
Join methods
nested loop vs ordered joins (merge join or hash join)

Distributed DBMS 88
 Cost-Based Optimization
Solution space
The set of equivalent algebra expressions (query trees).
Cost function (in terms of time)
I/O cost + CPU cost + communication cost
These might have different weights in different distributed
environments (LAN vs WAN).
Can also maximize throughput

Search algorithm
How do we move inside the solution space?
Exhaustive search, heuristic algorithms (iterative
improvement, simulated annealing, genetic,…)

Distributed DBMS 89
 Query Optimization Process

Input Query

Search Space Transformation

Generation Rules

Equivalent QEP

Search Cost Model

Strategy

Best QEP

Distributed DBMS 90
 Search Space
Search space characterized by PNO
alternative execution plans
ENO
Focus on join trees PROJ

For N relations, there are O(N!) EMP ASG

equivalent join trees that can be
obtained by applying ENO
commutativity and associativity
rules PNO EMP
SELECT ENAME,RESP
FROM EMP, ASG, PROJ PROJ ASG
WHERE EMP.ENO=ASG.ENO
AND ASG.PNO=PROJ.PNO ENO,PNO

× ASG

PROJ EMP
Distributed DBMS 91
 Search Space
Restrict by means of heuristics
Perform unary operations before binary operations
…

Restrict the shape of the join tree

Consider only linear trees, ignore bushy ones

Linear Join Tree Bushy Join Tree

R4

R3

R1 R2 R1 R2 R3 R4

Distributed DBMS 92
 Search Strategy
How to “move” in the search space.
Deterministic
Start from base relations and build plans by adding one
relation at each step
Dynamic programming: breadth-first
Greedy: depth-first

Randomized
Search for optimalities around a particular starting point
Trade optimization time for execution time
Better when > 5-6 relations
Simulated annealing
Iterative improvement

Distributed DBMS 93
 Search Strategies
Deterministic

R4

R3 R3

R1 R2 R1 R2 R1 R2
Randomized

R3 R2

R1 R2 R1 R3

Distributed DBMS 94
 Cost Functions
Total Time (or Total Cost)
Reduce each cost (in terms of time) component individually
Do as little of each cost component as possible
Optimizes the utilization of the resources

Increases system throughput

Response Time
Do as many things as possible in parallel
May increase total time because of increased total activity

Distributed DBMS 95
 Total Cost
Summation of all cost factors

Total cost = CPU cost + I/O cost + communication cost

CPU cost = unit instruction cost ∗ no.of instructions

I/O cost = unit disk I/O cost ∗ no. of disk I/Os

communication cost = message initiation + transmission

Distributed DBMS 96
 Total Cost Factors
Wide area network
message initiation and transmission costs high
local processing cost is low (fast mainframes or
minicomputers)
ratio of communication to I/O costs = 20:1

Local area networks

communication and local processing costs are more or less
equal
ratio = 1:1.6

Distributed DBMS 97
 Response Time
Elapsed time between the initiation and the completion of a
query

Response time = CPU time + I/O time + communication time

CPU time = unit instruction time ∗ no. of sequential instructions

I/O time = unit I/O time ∗ no. of sequential I/Os

communication time = unit msg initiation time ∗ no. of

sequential msg + unit transmission time ∗ no. of
sequential bytes

Distributed DBMS 98
 Example
Site 1
x units

Site 3

Site 2 y units

Assume that only the communication cost is considered

Total time = 2 ∗ message initialization time + unit transmission
time ∗ (x+y)
Response time = max {time to send x from 1 to 3, time to send y
from 2 to 3}
time to send x from 1 to 3 = message initialization time + unit
transmission time ∗ x
time to send y from 2 to 3 = message initialization time + unit
transmission time ∗ y
Distributed DBMS 99
 Optimization Statistics
Primary cost factor: size of intermediate relations
Make them precise ⇒ more costly to maintain
For each relation R[A1, A2, …, An] fragmented as R1, …, Rr
length of each attribute: length(Ai)
the number of distinct values for each attribute in each fragment:
card(∏AiRj)
maximum and minimum values in the domain of each attribute:
min(Ai), max(Ai)
the cardinalities of each domain: card(dom[Ai])
the cardinalities of each fragment: card(Rj)
Selectivity factor of each operation for relations
For joins
card(R S)
SF (R,S) =
card(R) ∗ card(S)
Distributed DBMS 100
 Intermediate Relation
Sizes
Selection
size(R) = card(R) ∗ length(R)
card(σF (R)) = SFσ (F) ∗ card(R)
where
1
S Fσ(A = value) =
card(∏A(R))
max(A) – value
S Fσ(A > value) =
max(A) – min(A)
value – max(A)
S Fσ(A < value) =
max(A) – min(A)

SFσ(p(A i) ∧ p(A j)) = SFσ(p(A i )) ∗ SFσ(p(A j))

SFσ(p(A i) ∨ p(A j)) = SFσ(p(A i )) + SFσ(p(A j)) – (SFσ(p(A i)) ∗ SFσ(p(A j)))
SFσ(A ∈ value) = SFσ(A= value) ∗ card({values})

Distributed DBMS 101

 Intermediate Relation Sizes
Projection
card(ΠA(R))=card(R)

Cartesian Product
card(R × S) = card(R) ∗ card(S)

Union
upper bound: card(R ∪ S) = card(R) + card(S)
lower bound: card(R ∪ S) = max{card(R), card(S)}

Set Difference
upper bound: card(R–S) = card(R)
lower bound: 0

Distributed DBMS 102

 Intermediate Relation Size
Join
Special case: A is a key of R and B is a foreign key of S;
card(R A=B S) = card(S)
More general:
card(R S) = SF ∗ card(R) ∗ card(S)

Semijoin
card(R A S) = SF (S.A) ∗ card(R)
where
card(∏A(S))
SF (R A S)= SF (S.A) =
card(dom[A])

Distributed DBMS 103

 System R Algorithm

Simple (i.e., mono-relation) queries are executed

according to the best access path

Execute joins
2.1 Determine the possible ordering of joins

2.2 Determine the cost of each ordering

2.3 Choose the join ordering with minimal cost

Distributed DBMS 104

 System R Algorithm
For joins, two alternative algorithms :
Nested loops
for each tuple of external relation (cardinality n1)
for each tuple of internal relation (cardinality n2)
join two tuples if the join predicate is true
end
end
Complexity: n1∗n2
Merge join
sort relations
merge relations
Complexity: n1+ n2 if relations are previously sorted and
equijoin

Distributed DBMS 105

 System R Algorithm – Example
Names of employees working on the CAD/CAM
project
Assume
EMP has an index on ENO,
ASG has an index on PNO,
PROJ has an index on PNO and an index on PNAME

ASG
ENO PNO

EMP PROJ

Distributed DBMS 106

 System R Example (cont’d)
Choose the best access paths to each relation
EMP: sequential scan (no selection on EMP)
ASG: sequential scan (no selection on ASG)
PROJ: index on PNAME (there is a selection on
PROJ based on PNAME)

Determine the best join ordering

EMP ASG PROJ
ASG PROJ EMP
PROJ ASG EMP
ASG EMP PROJ
EMP × PROJ ASG
PROJ × EMP ASG
Select the best ordering based on the join costs evaluated
according to the two methods
Distributed DBMS 107
 System R Algorithm
Alternatives

EMP ASG PROJ

EMP ASG EMP × PROJASG EMP ASG PROJ PROJ ASG PROJ × EMP
pruned pruned pruned pruned

(ASG EMP) PROJ (PROJ ASG) EMP

Best total join order is one of

((ASG EMP) PROJ)
((PROJ ASG) EMP)

Distributed DBMS 108

 System R Algorithm
((PROJ ASG) EMP) has a useful index on
the select attribute and direct access to the join
attributes of ASG and EMP

Therefore, chose it with the following access

methods:
select PROJ using index on PNAME

then join with ASG using index on PNO

then join with EMP using index on ENO

Distributed DBMS 109

 Join Ordering in Fragment
Queries

Ordering joins
Distributed INGRES

System R*

Semijoin ordering
SDD-1

Distributed DBMS 110

 Join Ordering
Consider two relations only

if size (R) < size (S)

R S
if size (R) > size (S)

Multiple relations more difficult because too many

alternatives.
Compute the cost of all alternatives and select the
best one.
Necessary to compute the size of intermediate
relations which is difficult.
Use heuristics

Distributed DBMS 111

 Join Ordering – Example

Consider
PROJ PNO ASG ENO EMP

Site 2

ASG
ENO PNO

EMP PROJ
Site 1 Site 3

Distributed DBMS 112

 Join Ordering – Example
Execution alternatives:
1. EMP → Site 2 2. ASG → Site 1
Site 2 computes EMP'=EMP ASG Site 1 computes EMP'=EMP ASG
EMP' → Site 3 EMP' → Site 3
Site 3 computes EMP’ PROJ Site 3 computes EMP’ PROJ

3. ASG → Site 3 4. PROJ → Site 2

Site 3 computes ASG'=ASG PROJ Site 2 computes PROJ'=PROJ ASG
ASG' → Site 1 PROJ' → Site 1
Site 1 computes ASG' EMP Site 1 computes PROJ' EMP

5. EMP → Site 2
PROJ → Site 2
Site 2 computes EMP PROJ ASG

Distributed DBMS 113

 Semijoin Algorithms
Consider the join of two relations:
R[A] (located at site 1)
S[A] (located at site 2)

Alternatives:
1 Do the join R A S
2 Perform one of the semijoin equivalents
R A S ⇔ (R A S) A S
⇔ R A (S A R)
⇔ (R A S) A (S A R)

Distributed DBMS 114

 Semijoin Algorithms
Perform the join
send R to Site 2
Site 2 computes R A S

Consider semijoin (R A S) A S
S' ← ∏A(S)
S' → Site 1
Site 1 computes R' = R A S'
R' → Site 2
Site 2 computes R' A S

Semijoin is better if
size(ΠA(S)) + size(R A S)) < size(R)

Distributed DBMS 115

 Distributed Query
Processing
Algorithms Opt. Objective Opt. Network Semijoin Stats Fragments
Timing Function Factors Topology

Dist. Dynamic Resp. Msg. Size, General or No 1 Horizontal

INGRES time or Proc. Cost Broadcast
Total time
R* Static Total time No. Msg., General or No 1, 2 No
Msg. Size, Local
IO, CPU
SDD-1 Static Total time Msg. Size General Yes 1,3,4, No
5

1: relation cardinality; 2: number of unique values per attribute; 3: join selectivity factor; 4: size
of projection on each join attribute; 5: attribute size and tuple size

Distributed DBMS 116

 R* Algorithm
Cost function includes local processing as well as
transmission
Considers only joins
Exhaustive search
Compilation
Published papers provide solutions to handling
horizontal and vertical fragmentations but the
implemented prototype does not

Distributed DBMS 117

 R* Algorithm
Performing joins
Ship whole
larger data transfer
smaller number of messages
better if relations are small

Fetch as needed
number of messages = O(cardinality of external relation)
data transfer per message is minimal
better if relations are large and the selectivity is good

Distributed DBMS 118

 R* Algorithm –
Vertical Partitioning & Joins
1. Move outer relation tuples to the site of the inner
relation
(a) Retrieve outer tuples
(b) Send them to the inner relation site
(c) Join them as they arrive

Total Cost = cost(retrieving qualified outer tuples)

+ no. of outer tuples fetched ∗
cost(retrieving qualified inner tuples)

+ msg. cost ∗ (no. outer tuples fetched ∗

avg. outer tuple size) / msg. size

Distributed DBMS 119

 R* Algorithm –
Vertical Partitioning & Joins
2. Move inner relation to the site of outer relation
cannot join as they arrive; they need to be stored

Total Cost = cost(retrieving qualified outer tuples)

+ no. of outer tuples fetched ∗
cost(retrieving matching inner tuples
from temporary storage)
+ cost(retrieving qualified inner tuples)
+ cost(storing all qualified inner tuples
in temporary storage)
+ msg. cost ∗ (no. of inner tuples fetched ∗
avg. inner tuple size) / msg. size

Distributed DBMS 120

 R* Algorithm –
Vertical Partitioning & Joins
3. Move both inner and outer relations to another site
Total cost = cost(retrieving qualified outer tuples)
+ cost(retrieving qualified inner tuples)
+ cost(storing inner tuples in storage)
+ msg. cost ∗ (no. of outer tuples fetched ∗
avg. outer tuple size) / msg. size
+ msg. cost ∗ (no. of inner tuples fetched ∗
avg. inner tuple size) / msg. size
+ no. of outer tuples fetched ∗
cost(retrieving inner tuples from
temporary storage)

Distributed DBMS 121

 R* Algorithm –
Vertical Partitioning & Joins
4. Fetch inner tuples as needed
(a) Retrieve qualified tuples at outer relation site
(b) Send request containing join column value(s) for outer tuples to
inner relation site
(c) Retrieve matching inner tuples at inner relation site
(d) Send the matching inner tuples to outer relation site
(e) Join as they arrive
Total Cost = cost(retrieving qualified outer tuples)
+ msg. cost ∗ (no. of outer tuples fetched)
+ no. of outer tuples fetched ∗ (no. of
inner tuples fetched ∗ avg. inner tuple
size ∗ msg. cost / msg. size)
+ no. of outer tuples fetched ∗
cost(retrieving matching inner tuples
for one outer value)

Distributed DBMS 122

 Step 4 – Local Optimization

Input: Best global execution schedule

Select the best access path
Use the centralized optimization techniques

Distributed DBMS 123