CS 382: Network-Centric Computing

Replication, Failures, Consistency

Zartash Afzal Uzmi

Spring 2023-24

ACK: Slides use some material from Scott Shenker (UC Berkeley) and Jim Kurose (UMass)
 Agenda
⚫ Replication (of?)

⚫ Failures and Correctness

⚫ Primary-backup Protocols
⚫ Asynchronous protocols
⚫ Synchronous protocols

⚫ Two-phase Commit (2PC) Protocol

⚫ Example
2
Replication

3
 Replication
⚫ When we replicate an object, we create copies of the object and store
them on different servers

⚫ Each copy is called a replica

4
 Why Replication?
⚫ Fault tolerance
⚫ If one replica crashes, simply switch to another replica
⚫ With k replicas of each object, it can tolerate the failure of any (k-1) servers in the
system

⚫ Performance
⚫ Helps when scaling for size or geographical area
⚫ Load balancing: divide the workload among multiple servers
⚫ Placing the copy of the object in proximity of the client, e.g., CDNs

5
 Nature of Replicated Data
⚫ Read-only data
⚫ Easy to replicate; we just make multiple copies
⚫ Read-write data
⚫ Writes result in different replicas. Any challenge?
⚫ Replicas must be kept consistent; modifications need to be propagated to all
other copies!
⚫ What do applications need: Read-only or Read-write?
⚫ Want the distributed system with multiple replicas to appear as if there was one
copy on a single machine. Want Read-write data replication!
⚫ Challenge:
⚫ When and how to propagate write updates?
6
 Depends on Application Requirements
⚫ What do applications require?
⚫ From replicated/distributed systems

⚫ Availability
⚫ The application is operational and instantly processes requests
⚫ Some server failures do not prevent surviving servers from continuing to operate

⚫ Partition Tolerance:
⚫ The application continues to operate despite message loss due to network partition

7
Network Partitions Divide Systems

8
Network Partitions Divide Systems

9
 Fundamental Tradeoff
⚫ Replicas appear to be a single [consistent] machine but lose availability
during a network partition

⚫ OR

⚫ All replicas remain available during a network partition but do not

appear to be a single machine (inconsistent data!!!)

10
 CAP Theorem Preview
⚫ You cannot achieve all three of:
1. Consistency Consistency Availability
2. Availability
3. Partition-Tolerance
Partition
tolerance

⚫ Consistency ➙ Replicas Act Like Single Machine

⚫ Availability ➙ All Sides of Partition Continue
⚫ Partition Tolerance ➙ Partitions Can Happen
11
 CAP Conjecture [Brewer 00]
⚫ From a keynote lecture by Eric Brewer (2000)
⚫ History: Eric started Inktomi, early Internet search site based around
“commodity” clusters of computers

⚫ Popular interpretation: 2-out-of-3

⚫ Consistency
⚫ Availability
⚫ Partition Tolerance

12
 CAP Theorem [Gilbert Lynch 02]
Assume that an algorithm provides all of CAP (to contradict!)
Let us start with a variable x=0 consistently stored at A and B

Client 1 Client 1

A B

13
 CAP Theorem [Gilbert Lynch 02]
Assume that an algorithm provides all of CAP (to contradict!)

w(x=1)
Client 1 Client 1
ok
A B

Write eventually returns

(from A)

Partition Possible
14
 CAP Theorem [Gilbert Lynch 02]
Assume to contradict that an algorithm provides all of CAP

w(x=1) r(x)
Client 1 Client 1
ok x=0
A B

Read begins after write

Write eventually returns completes
Read eventually returns

Partition Possible
15
 CAP Theorem [Gilbert Lynch 02]
Assume to contradict that an algorithm provides all of CAP
Not consistent (C) => contradiction!

w(x=1) r(x)
Client 1 Client 1
ok x=0
A B

Read begins after write

Write eventually returns completes
Read eventually returns

Partition Possible
16
 CAP Interpretation Part 1
⚫ Cannot “choose” no partitions
⚫ 2-out-of-3 interpretation doesn’t make sense
⚫ Instead, availability OR consistency?

⚫ i.e., a fundamental tradeoff between availability and consistency

⚫ When designing a system, you must choose one or the other; both are not possible
simultaneously

17
 CAP Interpretation Part 2
⚫ It is a theorem, with proof, that you understand!

⚫ Cannot “beat” CAP Theorem

⚫ Can engineer systems to make partitions extremely rare, and then just
take the rare hit to availability (or consistency)

18
Some Real Distributed Systems Relax
Consistency Constraints …

Memcache at
Facebook

19
Questions?

20
Node Failures and Correctness

21
 Failure Model: Fail-Stop
Node Fails!

Nodes fail by crashing

A machine is either working correctly or it is doing nothing

22
Failure Model: Byzantine Failures
Node Fails

Node operates arbitrarily after a failure

(this includes not sending messages at all or sending different and wrong
messages to different servers or lying about a value)

Can be caused by
• Malicious attacks
• Software errors

23
 Failures in Distributed Systems (DS)
⚫ What distinguishes DS from single-machine systems:
⚫ Some nodes might still be working correctly while others are experiencing failures
⚫ DS may continue to operate even if part of it is failing

⚫ A design goal for distributed systems:

⚫ “Correctly” operate even when failures occur

24
 Correctness for Strong Consistency
⚫ Replicas act like a single machine

⚫ Specifically
⚫ If one node commits an update, no other replica rejects it
⚫ If one replica rejects it, no one commits the update

25
 We will assume Fail-Stop model
⚫ Many consistency protocols assume fail-stop model

⚫ Reason: Byzantine failures are very hard to deal with and usually add
substantial performance overheads

⚫ However, Byzantine fault-tolerant solutions are important in certain

contexts
⚫ Covered in the advanced course “CS 582: Distributed Systems”
⚫ Also covered in “CS3812: Into to Blockchain”

26
Primary-backup protocols

27
 Primary-backup protocols
⚫ One special node (primary) orders requests

⚫ Route all updates through the primary node

⚫ Assigns an order to the updates
⚫ All replicas commit updates in the assigned order
⚫ Simple to implement

28
 Two Types of Primary-backup Schemes
⚫ Asynchronous primary-backup protocol

⚫ Synchronous primary-backup protocol

29
 Primary-backup protocol

Client

Replica Primary Replica

30
 Primary-backup protocol

Client

Write-request

Replica Primary Replica

31
 Primary-backup protocol
• The primary sends an ACK to
the client once it has performed
the write locally
Client OR
• It waits for all replicas first to
perform the update and then
sends the ACK
Write-request

Replica Primary Replica

32
Primary-backup protocol[asynchronous version]

Client

Step 1
Write-request

Replica Primary Replica

33
Primary-backup protocol[asynchronous version]

Client

Step 1 ACK write

Write-request completed
Step 2

Replica Primary Replica

34
Primary-backup protocol[asynchronous version]

Client

Step 1 ACK write

Write-request completed
Step 2

Replica Primary Replica

Tell replicas Tell replicas

Step 3
to update to update
35
Primary-backup protocol[asynchronous version]

Client

Step 1 ACK write

Write-request completed
Step 2
Step 4 Update Update
ACK ACK
Replica Primary Replica

Tell replicas Tell replicas

Step 3
to update to update
36
 Analyzing the Asynchronous Version
⚫ Performance (w/o failures):
⚫ The client does not need to spend any additional time waiting for the internals of
the system to do their work
⚫ The system is also more tolerant of network latency since fluctuations in internal
latency do not cause additional waiting on the client-side

⚫ What about correctness (with failures)?

37
 What could go wrong if there are failures?
⚫ If the primary fails before the updates are sent to the backups,
⚫ Then updates may be lost

⚫ Correctness can get violated under failures

⚫ If one server commits (update), no one rejects it
⚫ If one rejects it, no one commits

⚫ From the client’s perspective

⚫ There are no guarantees that you can read back what you wrote if there are any
failures in the system – no “Read Your Writes” consistency!

38
Primary-backup protocol[synchronous version]

Client

Step 1
Write-request

Replica Primary Replica

39
Primary-backup protocol[synchronous version]

Client

Step 1
Write-request

Replica Primary Replica

Tell replicas Tell replicas

Step 2
to update to update
40
Primary-backup protocol[synchronous version]

Client

Step 1
Write-request

Step 3 Update Update

ACK ACK
Replica Primary Replica

Tell replicas Tell replicas

Step 2
to update to update
41
Primary-backup protocol[synchronous version]

Client

Step 1 ACK write

Write-request completed
Step 4
Step 3 Update Update
ACK ACK
Replica Primary Replica

Tell replicas Tell replicas

Step 2
to update to update
42
 Analyzing the Synchronized Version
⚫ Performance (w/o failures):
⚫ Clients have to wait for additional time – used for synchronizing replicas
⚫ Network delay fluctuations inside the system impact the client

⚫ Can this version guarantee correctness with failures?

43
What could go wrong if there are failures?

44
 Failure Scenario

Client

Write-request

Replica Primary Replica

45
 Failure Scenario

Client

Write-request

Replica Primary Replica

Tell replicas Tell replicas

to update to update
46
 Failure Scenario

Client

Write-request
This replica
crashes
Update
ACK

Replica Primary Replica

Write update Tell replicas Tell replicas

committed to update to update
47
 Failure Scenario

The client assumes the

Client
write update failed

Write-request
This replica
crashes
Update
ACK

Replica Primary Replica

Write update Tell replicas Tell replicas

committed to update to update
48
 Failure Scenario

Replicas do not agree

The client assumes the
We need a way to roll Client
write update failed
back updates

Write-request
This replica
crashes
Update
ACK

Replica Primary Replica

Write update Tell replicas Tell replicas

committed to update to update
49
 Two Phase Commit (2-PC)
⚫ Consists of two distinct phases

⚫ Used in several distributed systems

50
Two Phase Commit (2-PC)

51
 Key idea
⚫ Allow the system to roll back updates on failures
⚫ By using two phases

⚫ This is in contrast to single-phase primary-based protocols

⚫ Where there is no step for rolling back an operation that has failed on some nodes
and succeeded on other nodes

52
 Terminology
⚫ Primary Node
⚫ Coordinator

⚫ Replicas
⚫ Cohort, worker, participant

⚫ Prepare phase (also voting phase OR commit-request phase)

⚫ Replicas become prepared
⚫ Commit phase
⚫ Transactions are committed across the system
53
 2-PC

Coordinator Replica 1 … Replica N-1

54
 2-PC

Coordinator Replica 1 … Replica N-1

Prepare Save update to

disk

Respond with
yes or No

55
 2-PC

Coordinator Replica 1 … Replica N-1

Save update to
Prepare disk

Respond with
Yes from all yes or No
replicas within
the timeout

56
 2-PC

Coordinator Replica 1 … Replica N-1

Save update to
Prepare disk

Respond with
Yes from all yes or No
replicas within
the timeout
Commit
Commit
updates from
disk to store
ACK
57
 2-PC

Coordinator Replica 1 … Replica N-1

Save update to
Prepare disk

Respond with
If any “No” or yes or No
timeout before
all votes
Abort

ACK
58
 Failures in 2-PC: Food for Thought
⚫ If a server votes yes, can it commit unilaterally before receiving commit
message?

⚫ If a server voted No, can it abort right away without waiting for an abort
message?

59
 Failures in 2-PC
⚫ To deal with replica crashes
⚫ Each replica saves tentative updates into permanent storage, right before replying
Yes/No in the first phase
⚫ Retrievable after crash recovery

⚫ To deal with coordinator crashes

⚫ The coordinator logs all decisions and received/sent messages on disk
⚫ After recovery or new election ➙ new coordinator takes over

60
 Correctness and Performance of 2-PC
⚫ Correctness: All hosts that decide reach the same decision
⚫ No commit unless everyone says “yes”

⚫ Performance: If failures, then 2PC might block

⚫ Failure of any process can result in non-progress
⚫ Doesn’t tolerate failures well: must wait for repair

61
 2-PC Summary
⚫ Primary-backup schemes (1 Phase protocols)
⚫ Safety may be violated ➙ Can’t roll back updates

⚫ Two-Phase Commit
⚫ Allow for rolling back updates
⚫ Sensitive to coordinator failure ➙ blocking

62
 Summing it up …
⚫ Replication is needed for fault tolerance and performance

⚫ But it introduces challenges

⚫ CAP theorem shows we cannot provide all three properties: strong consistency,
availability, and partition tolerance

⚫ How to design a consistency protocol for a replicated system?

⚫ Primary-backup protocols: asynchronous and synchronous
⚫ Need more than one phase to provide consistency under failures

63
 Advanced Topics on Consistency
⚫ Paxos

⚫ Raft

⚫ PBFT

⚫ Blockchain

64
Questions?

66