Topics of Distributed System Section

• Distributed deadlock detection

– Global state and distributed snapshot
– Logical clock and vector clock, …
• Distributed mutual exclusion
• Distributed transaction execution
• Fault tolerance
– Quorum-based approach
– Leader-based approach: Paxos and PBFT
• Consistency
 Model
• A number of machines connected by network.
• Message transfer has non-negligible delay:
– Synchronous model: delay has an upper bound
– Asynchronous model: no upper bound
 Problem
• No single machine has all state

• Sometimes need to operate on global state

– Is the system in a deadlock?
– How much money does a bank have?
– Take a checkpoint for future recovery
 Problem
• Does the following simple approach work?

Monitor

Give me your state

 Counter example: deadlock detection
• Check if there is a cycle in wait-for graph

p0 p0

p1 p2 p1 p2

Deadlock No deadlock
Counter example: deadlock detection
• Suppose a monitor uses the simple approach
Monitor
p0

Who is waiting
for you?

p1 p2
Counter example: deadlock detection
• Suppose a monitor uses the simple approach
Monitor
p0

p1 p2

Initial state: p0 is waiting for p1

Counter example: deadlock detection
• Suppose a monitor uses the simple approach
Monitor
p0

Who is waiting
for you?

p1 p2

Monitor’s query arrives at p1

Counter example: deadlock detection
• Suppose a monitor uses the simple approach
Monitor
p0

p0 is waiting for p1

p1 p2

p1 responds.
Counter example: deadlock detection
• Suppose a monitor uses the simple approach
Monitor
p0

p1 p2

p1 grants the resource to p0

Counter example: deadlock detection
• Suppose a monitor uses the simple approach
Monitor
p0

p1 p2

p1 executes a job that needs to wait for p2

Counter example: deadlock detection
• Suppose a monitor uses the simple approach
Monitor
p0

Who is waiting
for you?

p1 p2

Monitor’s query arrives at p2.

Counter example: deadlock detection
• Suppose a monitor uses the simple approach
Monitor
p0

p1 is waiting for p2.

p1 p2

p2 responds.
Counter example: deadlock detection
• Suppose a monitor uses the simple approach
Monitor
p0

p1 p2

p2 executes a job that needs to wait for p0.

Counter example: deadlock detection
• Suppose a monitor uses the simple approach
Monitor
p0
Who is waiting
for you?

p1 p2

Monitor’s query arrives at p0.

Counter example: deadlock detection
• Suppose a monitor uses the simple approach
Monitor
p0

p2 is waiting for p0.

p1 p2

p0 responds.
Counter example: deadlock detection
• Suppose a monitor uses the simple approach
Monitor
p0

p0 is waiting for p1.

p1 is waiting for p2.
p2 is waiting for p0.

Deadlock!

p1 p2

No deadlock.
 Problem
• Simply combining state of each node does not work
– Need to somehow get each one’s state at the right timing

• Timing is hard in a distributed system

– No global clock
– Message transfer has non-negligible delays

• Before that, we need to answer the following

question: what is a “good” global state?
 Events and histories
• Each process executes a sequence of events
• Events can be of three types: local, send, and receive
• eip is the i-th event of process p
• The local history hp of process p is the sequence of
events executed by process p
– hkp: prefix that contains first k events
• The history H is the set hp0 hp1 … hp(n-1)
 Happened-before
• Leslie Lamport, 1978

• 2013 ACM Turing Award

– Named ACM Fellow afterwards ……

• Also initial developer of Latex

 Happened-before
• A binary rela on → defined over events

1. If eki , eli hi and k < l, then eki → eli

– Events in a local history are totally ordered
2. If ei = send(m) and ej = receive(m), then ei → ej
– Sending a message precedes receiving it.
3. If e → e’ and e’ → e’’ then e →e’’
– Transitivity
 Consistent global state
• A global state G is consistent if
– ei, ej: ej G ei →ej ei G
 Space-Time diagram
• A graphic representation of a distributed
execution

p0 p0
Request

p1

p2

p1 p2
 Space-Time diagram
• A graphic representation of a distributed
execution

p0 p0
Grant
p1

p2

p1 p2
 Space-Time diagram
• A graphic representation of a distributed
execution

p0 p0

p1
Request

p2

p1 p2
 Space-Time diagram
• A graphic representation of a distributed
execution

p0 p0

Request
p1

p2

p1 p2
 Space-Time diagram
• A graphic representation of a distributed
execution
States received by the monitor

p0 p0

p1

p2

p1 p2 Is this a consistent global state?

 Go back to the problem
• Naïve approach obtains a global state
• We want to obtain a consistent global state

Monitor

Give me your state

DISCUSSION
---TRY IT YOURSELF
 Distributed snapshot
• Local state of a process
– Registers, memory state, disk state, …
– Incoming and outgoing messages
• If we simply combine multiple local states
✔If eki , eli hi and k < l, and eli C, then eki C
✕If ei = send(m) and ej = receive(m), and ej C,
then ei C
 Distributed snapshot
• Someone needs to send a “take snapshot”
command to everyone to start
• It may arrive at different nodes at arbitrary
time. How to coordinate local snapshots at
the right timing?
p0

p1

p2
 Chandy Lamport protocol
• Global state is consisted of state of each
node and state of each channel
– State of a node: registers, memory, disk, etc
– State of a channel: messages
• Assume we record messages at the receiver side

• We can ask a node to start to take snapshot

– It will record its own state and its incoming
messages
 Chandy Lamport protocol
• When a node pi receives “take snapshot” for
the first time, it starts taking snapshot and
relays the message to every one
• Two purposes
– It notifies pj to start taking snapshot if pj has not
started yet
– It tells pj that messages sent by pi to pj after
“take snapshot” should not be included in pj’s
snapshot
• Assuming channel is FIFO
 Chandy Lamport protocol
• If p1 takes snapshot at red arrow, p0’s
snapshot should not include the last two
received messages.
• How to let p0 know?

p0

p1

p2
 Chandy Lamport protocol
• If p1 takes snapshot at red arrow, p0’s
snapshot should not include the last two
received messages.
• How to let p0 know? p1 will relay the “take
snapshot” message to p0 as a barrier
p0

p1

p2
 Chandy Lamport protocol
• Start: a node sends “take snapshot” to itself
• When a node receives “take snapshot” for the
first time:
– It records application state
– It starts recording all incoming messages
– It relays “take snapshot” to all other nodes
• When to stop recording
– When pj receives “take snapshot” from pi, pj stops
recording incoming messages from pi.
– Recording completes when pj receives “take
snapshot” from every node
Chandy Lamport protocol

p0

p1

p2
Chandy Lamport protocol

p0

p1

p2
Chandy Lamport protocol

p0

p1

p2
Chandy Lamport protocol

p0

p1

p2
Chandy Lamport protocol

p0

p1

p2

What is the recorded state of each node?

Chandy Lamport protocol

p0

p1

p2

What is the recorded state of each node?

 Exercise

p0

p1

p2

What happens if starting at another position?

 Exercise

Prove the correctness of Chandy Lamport protocol

✔If eki , eli hi and k < l, and eli C, then eki C

? If ei = send(m) and ej = receive(m), and ej C, then
ei C
 Proof

p0

p1
 Proof

p0

p1

Since p1’s recorded state includes the receive event, p1 must

receive p0’s “take snapshot” after the receive event.
(because p1 records incoming messages from p0 until it receives
“take snapshot” from p0)
 Proof

p0

p1

Since p1’s recorded state includes the receive event, p1 must

receive p0’s “take snapshot” after the receive event.

P0 relays “take snapshot” after it receives “take snapshot” for

the first time and records its own state at that moment.
 Proof

p0

p1

Since p1’s recorded state includes the receive event, p1 must

receive p0’s “take snapshot” after the receive event.

P0 relays “take snapshot” after it receives “take snapshot” for

the first time and records its own state at that moment.

Therefore, p0’s snapshot must include the send event.

 Consistent global state

It may not be a state that actually happened in real

time.
It is a state that could happen.
p0

p1

p2

Then why do we care?

 Stable properties
• If a predicate Φ holds in state ∑, then it holds
in all states that are reachable from ∑
– Once true, always true
• Examples:
– Deadlock
– Termination
– Unreachability for garbage collection
• Counter examples:
– Number of online users
– Money in the bank
 Stable properties
• Consistent global state is useful when
evaluating stable properties.
• How about unstable properties?
– Rough estimate
– Stronger assumption
– Block the system