Overview

• Distributed Database Concepts

• Advantages of Distributed Databases
• Data Fragmentation, Replication, and Allocation Techniques

Distributed Databases • Concurrency Control and Recovery

• Distributed Transactions
CO527 Advanced Database Systems • Two-phase commit protocol
• Three-phase commit protocol
Dr. (Eng.) Sampath Deegalla • Distributed Query Processing
• Types of Distributed Database Systems

Distributed Database Concepts Distributed Database Concepts (cont.)

• A Distributed Database (DDB) is a collection of • Recently distributed and database technologies are being developed
multiple logically related database distributed over for dealing with vast amount of data which is known as big data
technologies.
a computer network, and a distributed database
management system as a software system that • Then data mining and machine learning algorithms are used to
extract needed knowledge.
manages a distributed database while making the
distribution transparent to the user.

• Distributed Database is a combined result of the

two technologies
• Database technology
• Network and Data Communication Technology
 Advantages of Distributed Database System Advantages of Distributed Database System
• Management of distributed data with different levels of
transparency: • Example
• This refers to the physical placement of data (files, relations, etc.) which • The EMPLOYEE, PROJECT, and WORKS_ON tables may be
is not known to the user (distribution transparency/data organization fragmented horizontally and stored with possible
transparency). replication as shown below.

Advantages of Distributed Database System Advantages of Distributed Database System

• Distribution and Network transparency: • Other Advantages

• Users do not have to worry about operational details of the network. • Improved ease and flexibility of application development
• There is Location transparency, which refers to freedom of issuing command from any • Increased reliability and availability:
location without affecting its working. • Reliability refers to system live time, that is, system is running efficiently most of the
• Then there is Naming transparency, which allows access to any names object (files, time.
relations, etc.) from any location. • Availability is the probability that the system is continuously available (usable or
accessible) during a time interval.
• Replication transparency: • A distributed database system has multiple nodes (computers) and if one fails then
others are available to do the job.
• It allows to store copies of a data at multiple sites as shown in the above
• Improved performance:
diagram.
• A distributed DBMS fragments the database to keep data closer to where it is needed
• This is done to minimize access time to the required data. most.
• This reduces data management (access and modification) time significantly.
• Fragmentation transparency: • Easier expansion (scalability):
• Allows to fragment a relation horizontally (create a subset of tuples of a • Horizontal scalability: Allows new nodes (computers) to be added anytime without
relation) or vertically (create a subset of columns of a relation). chaining the entire configuration.
• Vertical scalability: Expanding capacity of individual nodes of the system or processing
power.
Data Fragmentation, Replication and Allocation Data Fragmentation, Replication and Allocation

• Data Fragmentation • Horizontal fragmentation (Sharding)

• Split a relation into logically related and correct parts. A relation can be • It is a horizontal subset of a relation which contain those of
fragmented in two ways:
• Horizontal Fragmentation
tuples which satisfy selection conditions.
• Vertical Fragmentation • Consider the Employee relation with selection condition (DNO
= 5). All tuples satisfy this condition will create a subset which
will be a horizontal fragment of Employee relation.
• A selection condition may be composed of several conditions
connected by AND or OR.
• Derived horizontal fragmentation: It is the partitioning of a
primary relation to other secondary relations which are related
with Foreign keys.

Data Fragmentation, Replication and Allocation Data Fragmentation, Replication and Allocation

• Vertical fragmentation • Data Replication

• It is a subset of a relation which is created by a subset of • Database is replicated to all sites.
columns. Thus a vertical fragment of a relation will contain • In full replication the entire database is replicated (called fully
values of selected columns. There is no selection condition replicated distributed database) and in partial replication some
used in vertical fragmentation. selected part is replicated to some of the sites.
• Consider the Employee relation. A vertical fragment of can be • Data Distribution (Data Allocation)
created by keeping the values of Name, Bdate, Sex, and
Address. • This is relevant only in the case of partial replication or
partition.
• Because there is no condition for creating a vertical fragment,
each fragment must include the primary key attribute of the • The selected portion of the database is distributed to the
parent relation Employee. In this way all vertical fragments of database sites.
a relation are connected.
Data Fragmentation, Replication and Allocation Data Fragmentation, Replication and Allocation

Concurrency Control and Recovery Concurrency Control and Recovery

• Distributed Databases encounter a number of concurrency control • Dealing with multiple copies of data items:
and recovery problems which are not present in centralized • The concurrency control must maintain global consistency. Likewise the recovery
databases. mechanism must recover all copies and maintain consistency after recovery.
• Dealing with multiple copies of data items • Failure of individual sites:
• Failure of individual sites • Database availability must not be affected due to the failure of one or two sites
• Communication link failure and the recovery scheme must recover them before they are available for use.
• Distributed commit • Communication link failure:
• Distributed deadlock • This failure may create network partition which would affect database availability
even though all database sites may be running.
• Distributed commit:
• A transaction may be fragmented and they may be executed by a number of sites.
This require a two or three-phase commit approach for transaction commit.
• Distributed deadlock:
• Since transactions are processed at multiple sites, two or more sites may get
involved in deadlock. This must be resolved in a distributed manner.
 Concurrency Control and Recovery Concurrency Control and Recovery
• Distributed Concurrency control based on a distributed copy of a data
item • Transaction management:
• Primary site technique: A single site is designated as a primary site which serves • Concurrency control and commit are managed by this site.
as a coordinator for transaction management. • In two phase locking, this site manages locking and releasing data items. If
all transactions follow two-phase policy at all sites, then serializability is
guaranteed.
Primary site
Site 5
Site 1

Site 4 Communications neteork

Site 3 Site 2

Concurrency Control and Recovery Concurrency Control and Recovery

• Transaction Management • Primary Copy Technique:

• Advantages: • In this approach, instead of a site, a data item partition is
• An extension to the centralized two phase locking so designated as primary copy. To lock a data item just the
implementation and management is simple. primary copy of the data item is locked.
• Data items are locked only at one site but they can be accessed at • Advantages:
any site.
• Since primary copies are distributed at various sites, a single
• Disadvantages:
site is not overloaded with locking and unlocking requests.
• All transaction management activities go to primary site which is
likely to overload the site. • Disadvantages:
• If the primary site fails, the entire system is inaccessible. • Identification of a primary copy is complex. A distributed
• To aid recovery a backup site is designated which behaves as a directory must be maintained, possibly at all sites.
shadow of primary site. In case of primary site failure, backup
site can act as primary site.
Concurrency Control and Recovery Concurrency Control and Recovery

• Recovery from a coordinator failure • Concurrency control based on voting:

• In both approaches a coordinator site or copy may become • There is no primary copy of coordinator.
unavailable. This will require the selection of a new • Send lock request to sites that have data item.
coordinator.
• If majority of sites grant lock then the requesting transaction gets the data
• Primary site approach with no backup site: item.
• Aborts and restarts all active transactions at all sites. Elects a • Locking information (grant or denied) is sent to all these sites.
new coordinator and initiates transaction processing. • To avoid unacceptably long wait, a time-out period is defined. If the
• Primary site approach with backup site: requesting transaction does not get any vote information then the
• Suspends all active transactions, designates the backup site as transaction is aborted.
the primary site and identifies a new back up site. Primary site
receives all transaction management information to resume
processing.
• Primary and backup sites fail or no backup site:
• Use election process to select a new coordinator site.

Distributed Transactions Distributed Transactions

• Transaction that updates data on two or more systems • Each computer runs a transaction manager
• Responsible for subtransactions on that system
• Transaction managers communicate with other transaction managers
• Challenge • Performs prepare, commit, and abort calls for subtransactions
• Handle machine, software, & network failures while preserving transaction
integrity • Every subtransaction must agree to commit changes before the
transaction can complete
Transactional Commits are Consensus Two-Phase Commit Protocol

• Remember consensus?
• Agree on a value proposed by at least one process
• The coordinator proposes to commit a transaction
• Participants will agree ⇒ all participants then commit
• Participants will not agree ⇒ all participants then abort
• Here, we need unanimous agreement to commit

Two-phase commit protocol Transaction States

• Goal:
• Reliably agree to commit or abort a collection of sub-transactions
• All processes in the transaction will agree to commit or abort
• One transaction manager is elected as a coordinator – the rest are
participants
• Assume:
• write-ahead log in stable storage
• No system dies forever
• Systems can always communicate with each other

When a participant enters the prepared state, it contacts the coordinator to

start the commit protocol to commit the entire transaction
Two-Phase Commit Protocol Two-Phase Commit Protocol

• Phase 1: Voting Phase • Phase 1: Voting Phase

• Get commit agreement from every participant • Get commit agreement from every participant

A single “no” response means that we will have to abort the transaction

Two-Phase Commit Protocol Two-Phase Commit Protocol

• Phase 2: Commit Phase • Phase 2: Commit Phase

• Send the results of the vote to every participant • Get “I have committed” acknowledgements from every participant

Send abort if any participant voted “no”

Dealing with failure Two-Phase Commit Protocol: Phase 1

• 2PC assumes a fail-recover model • Voting Phase

• Any failed system will eventually recover
• A recovered system cannot change its mind
• If a node agreed to commit and then crashed, it must be willing and able to
commit upon recovery
• Each system will use a writeahead (transaction) log
• Keep track of where it is in the protocol (and what it agreed to)
• As well as values to enable commit or abort (rollback)

• Get distributed agreement: the coordinator asked each participant if it

will commit or abort and received replies from each participant.

Two-Phase Commit Protocol: Phase 2 Dealing with failure

• Commit Phase • Failure during Phase 1 (voting)

• Coordinator dies
• Some participants may have responded; others have no clue
• Coordinator restarts; checks log; sees that voting was in progress
• Coordinator restarts voting
• Participant dies
• The participant may have died before or after sending its vote to the coordinator
• If the coordinator received the vote, wait for other votes and go to phase 2
• Otherwise: wait for the participant to recover and respond (keep querying it)

• Tell all participants to commit or abort

• Get everyone’s response that they’re done
Dealing with failure Adding a recovery coordinator

• Failure during Phase 2 (commit/abort) • Another system can take over for the coordinator
• Coordinator dies • Could be a participant that detected a timeout to the coordinator
• Some participants may have been given commit/abort instructions
• Coordinator restarts; checks log; informs all participants of chosen action
• Recovery node needs to find the state of the protocol
• Participant dies • Contact ALL participants to see how they voted
• The participant may have died before or after getting the commit/abort request • If we get voting results from all participants
• Coordinator keeps trying to contact the participant with the request • We know that Phase 1 has completed
• Participant recovers; checks log; gets request from coordinator • If all participants voted to commit ⇒ send commit request
• If it committed/aborted, acknowledge the request • Otherwise send abort request
• Otherwise, process the commit/abort request and send back the acknowledgement • If ANY participant has not voted
• We know that Phase 1 has not completed
• Restart the protocol

• But … if a participant node also crashes, we’re stuck!

• Have to wait for recovery

What’s wrong with the 2PC protocol? Three-Phase Commit Protocol

• Biggest problem: it’s a blocking protocol • Same setup as the two-phase commit protocol:
• If the coordinator crashes, participants have no idea whether to commit or • Coordinator & Participants
abort
• Add timeouts to each phase that result in an abort
• A recovery coordinator helps in some cases
• A non-responding participant will also result in blocking • Propagate the result of the commit/abort vote to each participant
• When a participant gets a commit/abort message, it does not know if before telling them to act on it
every other participant was informed of the result • This will allow us to recover the state if any participant dies
Three-Phase Commit Protocol Three-Phase Commit Protocol

• Split the second phase of 2PC into two parts: • Phase 1: voting phase
• Coordinator sends canCommit? queries to participants & et responses
• 2a. “Precommit” (prepare to commit) phase
• Purpose: Find out if everyone agrees to commit
• Send Prepare message to all participants when it received a yes from all participants in • It the coordinator gets a timeout from any participant, or any NO replies are received
phase 1
• Send an abort to all participants
• Participants can prepare to commit but cannot do anything that cannot be undone • If a participant times out waiting for a request from the coordinator
• Participants reply with an acknowledgement • It aborts itself (assume coordinator crashed)
• Purpose: let every participant know the state of the result of the vote so that state can • Else continue to phase 2
be recovered if anyone dies
• Phase 2: Prepare to commit phase
• 2b. “Commit” phase (same as in 2PC) • Send a Prepare message to all participants. Get OK messages from all participants
• If coordinator gets ACKs for all “precommit” messages • Purpose: let all participants know the decision to commit
• It will send a commit message • If coordinator times out: assume participant crashed, send abort to all participants
• Else it will abort – send an abort message to all participants • The coordinator cannot count on every participant having received the Prepare message
• Phase 3: finalize
• Send commit messages to participants and get responses from all
• If participant times out: contact any other participant and move to that state (commit or
abort)
• If coordinator times out: that’s ok

3PC Recovery 3PC Weaknesses

• If the coordinator crashes • Main weakness of 3PC

• A recovery node can query the state from any available participants • May have problems when the network gets partitioned
• Possible states that the participant may report: • Partition A: nodes that received Prepare message
• Recovery coordinator for A: allows commit
• Already committed
• That means that every other participant has received a Prepare to Commit
• Partition B: nodes that did not receive Prepare message
• Some participants may have committed • Recovery coordinator for B: aborts
• Send Commit message to all participants (just in case they didn’t get it) • Either of these actions are legitimate as a whole
• Not committed but received a Prepare message • But when the network merges back, the system is inconsistent
• That means that all participants agreed to commit; some may have committed • Not good when a crashed coordinator recovers
• Send Prepare to Commit message to all participants (just in case they didn’t get it) • It needs to find out that someone took over and stay quiet
• Wait for everyone to acknowledge; then commit
• Otherwise it will mess up the protocol, leading to an inconsistent state
• Not yet received a Prepare message
• This means no participant has committed; some may have agreed
• Transaction can be aborted or the commit protocol can be restarted
3PC coordinator recovery problem Query Processing in Distributed Databases
• Issues
• Suppose a coordinator sent a Prepare messages to all participants
• Cost of transferring data (files and results) over the network.
• Suppose all participants acknowledged these message • This cost is usually high so some optimization is necessary.
• BUT the coordinator died before it got all acknowledgements
• Example relations: Employee at site 1 and Department at Site 2
• A recovery coordinator queries a participant • Employee at site 1. 10,000 rows. Row size = 100 bytes. Table size = 106
• Continues with the commit: Sends Prepare, gets ACKs, sends Commit bytes.
Fname Minit Lname SSN Bdate Address Sex Salary Superssn Dno
• At the same time...
• The original coordinator recovers • Department at Site 2. 100 rows. Row size = 35 bytes. Table size = 3,500
bytes. Dname Dnumber Mgrssn Mgrstartdate
• Realizes it is still missing some replies from the Prepare
• Q: For each employee, retrieve employee name and department
• Times out and decides to send an Abort to all participants
name Where the employee works.
• Some processes may commit while others abort! • Q: Fname,Lname,Dname (Employee Dno = Dnumber Department)
• 3PC is not resilient against asynchronous fail-recover failures

Query Processing in Distributed Databases Query Processing in Distributed Databases

• Result • Strategies:
• The result of this query will have 10,000 tuples, assuming that every 1. Transfer Employee and Department to site 3.
employee is related to a department.
• Total transfer bytes = 1,000,000 + 3500 = 1,003,500 bytes.
• Suppose each result tuple is 40 bytes long. The query is submitted at site 3
and the result is sent to this site. 2. Transfer Employee to site 2, execute join at site 2 and send
• Problem: Employee and Department relations are not present at site 3. the result to site 3.
• Query result size = 40 * 10,000 = 400,000 bytes. Total transfer
size = 400,000 + 1,000,000 = 1,400,000 bytes.
3. Transfer Department relation to site 1, execute the join at
site 1, and send the result to site 3.
• Total bytes transferred = 400,000 + 3500 = 403,500 bytes.
• Optimization criteria: minimizing data transfer.
• Preferred approach: strategy 3.
Query Processing in Distributed Databases Query Processing in Distributed Databases

• Consider the query • The result of this query will have 100 tuples, assuming
• Q’: For each department, retrieve the department name and the name of that every department has a manager, the execution
the department manager strategies are:
• Relational Algebra expression: 1. Transfer Employee and Department to the result site and
• Fname,Lname,Dname (Employee Mgrssn = SSN Department) perform the join at site 3.
• Total bytes transferred = 1,000,000 + 3500 = 1,003,500 bytes.
2. Transfer Employee to site 2, execute join at site 2 and send
the result to site 3. Query result size = 40 * 100 = 4000
bytes.
• Total transfer size = 4000 + 1,000,000 = 1,004,000 bytes.
3. Transfer Department relation to site 1, execute join at site 1
and send the result to site 3.
• Total transfer size = 4000 + 3500 = 7500 bytes.
• Preferred strategy: Choose strategy 3.

Query Processing in Distributed Databases Query Processing in Distributed Databases

• Now suppose the result site is 2. Possible strategies : • Semijoin:

1. Transfer Employee relation to site 2, execute the query and present the • Objective is to reduce the number of tuples in a relation
result to the user at site 2. before transferring it to another site.
• Total transfer size = 1,000,000 bytes for both queries Q and Q’.
2. Transfer Department relation to site 1, execute join at site 1 and send
• Example execution of Q or Q’:
the result back to site 2. 1. Project the join attributes of Department at site 2, and
• Total transfer size for Q = 400,000 + 3500 = 403,500 bytes and for Q’ = 4000 + transfer them to site 1. For Q, 4 * 100 = 400 bytes are
3500 = 7500 bytes. transferred and for Q’, 9 * 100 = 900 bytes are transferred.
2. Join the transferred file with the Employee relation at site 1,
and transfer the required attributes from the resulting file
to site 2. For Q, 34 * 10,000 = 340,000 bytes are transferred
and for Q’, 39 * 100 = 3900 bytes are transferred.
3. Execute the query by joining the transferred file with
Department and present the result to the user at site 2.
Types of Distributed Database Systems Types of Distributed Database Systems
• Heterogeneous
• Homogeneous
• Federated: Each site may run different database system but the data
• All sites of the database access is managed through a single conceptual schema.
system have identical setup,
• This implies that the degree of local autonomy is minimum. Each site must
i.e., same database system adhere to a centralized access policy. There may be a global schema.
software.
• Multidatabase: There is no one conceptual global schema. For data
• The underlying operating access a schema is constructed dynamically as needed by the application
system may be different. software.
• For example, all sites run
Oracle or DB2, or Sybase or
some other database system.
• The underlying operating
systems can be a mixture of
Linux, Window, Unix, etc.

Types of Distributed Database Systems

• Federated Database Management Systems Issues

• Differences in data models:
• Relational, Objected oriented, hierarchical, network, etc.
• Differences in constraints:
• Each site may have their own data accessing and processing constraints.
• Differences in query language:
• Some site may use SQL, some may use SQL-89, some may use SQL-92, and so on.