D ISTRIBUTED S YSTEMS [COMP9243]

S YNCHRONOUS VS A SYNCHRONOUS D ISTRIBUTED S YSTEMS

Lecture 5: Synchronisation and Coordination Timing model of a distributed system

(Part 1) Affected by:

Slide 1 Slide 3
➜ Execution speed/time of processes
➀ Distributed Algorithms ➜ Communication delay
➁ Time and Clocks ➜ Clocks & clock drift
➂ Global State ➜ (Partial) failure
➃ Concurrency Control

D ISTRIBUTED A LGORITHMS Synchronous Distributed System:

Algorithms that are intended to work in a distributed Time variance is bounded

environment
Execution : bounded execution speed and time
Used to accomplish tasks such as:
Communication : bounded transmission delay
➜ Communication
➜ Accessing resources Clocks : bounded clock drift (and differences in clocks)
Slide 2 ➜ Allocating resources Slide 4
Effect:
➜ Consensus
➜ Can rely on timeouts to detect failure
➜ etc.
V Easier to design distributed algorithms
Synchronisation and coordination inextricably linked to X Very restrictive requirements
distributed algorithms • Limit concurrent processes per processor
➜ Achieved using distributed algorithms • Limit concurrent use of network
➜ Required by distributed algorithms • Require precise clocks and synchronisation

S YNCHRONOUS VS A SYNCHRONOUS D ISTRIBUTED S YSTEMS 1 S YNCHRONOUS VS A SYNCHRONOUS D ISTRIBUTED S YSTEMS 2

 Asynchronous Distributed System:

Time variance is not bounded

Execution : different steps can have varying duration S YNCHRONISATION AND C OORDINATION
Communication : transmission delays vary widely Important:

Clocks : arbitrary clock drift Doing the right thing at the right time.
Slide 5 Slide 7
Effect: Two fundamental issues:
➜ Allows no assumption about time intervals ➜ Coordination (the right thing)
X Cannot rely on timeouts to detect failure ➜ Synchronisation (the right time)
X Most asynch DS problems hard to solve
V Solution for asynch DS is also a solution for synch DS
➜ Most real distributed systems are hybrid synch and asynch

E VALUATING D ISTRIBUTED A LGORITHMS

General Properties:
➜ Performance
S YNCHRONISATION
• number of messages exchanged
• response/wait time Ordering of all actions
• delay ➜ Total ordering of events
Slide 6 • throughput: 1/(delay + executiontime) Slide 8 ➜ Total ordering of instructions
• complexity: O()
➜ Total ordering of communication
➜ Efficiency ➜ Ordering of access to resources
• resource usage: memory, CPU, etc. ➜ Requires some concept of time
➜ Scalability
➜ Reliability
• number of points of failure (low is good)

S YNCHRONISATION AND C OORDINATION 3 C OORDINATION 4

 C OORDINATION
Coordinate actions and agree on values.

Coordinate Actions:
➜ What actions will occur
Slide 9 ➜ Who will perform actions Slide 11 T IME AND C LOCKS

Agree on Values:
➜ Agree on global value
➜ Agree on environment
➜ Agree on state

T IME
Global Time:
M AIN I SSUES ➜ ’Absolute’ time
• Einstein says no absolute time
Time and Clocks: synchronising clocks and using time in • Absolute enough for our purposes
distributed algorithms ➜ Astronomical time
Slide 10 Global State: how to acquire knowledge of the system’s Slide 12 • Based on earth’s rotation
global state • Not stable
➜ International Atomic Time (IAT)
Concurrency Control: coordinating concurrent access to
• Based on oscillations of Cesium-133
resources
➜ Coordinated Universal Time (UTC)
• Leap seconds
• Signals broadcast over the world

T IME AND C LOCKS 5 T IME 6

 P HYSICAL C LOCKS
Synchronisation Using Physical Clocks:
Local Time: Examples:
Slide 13 ➜ Not synchronised to Global source Slide 15 ➜ Performing events at an exact time (turn lights on/off,
➜ Relative not ’absolute’ lock/unlock gates)
➜ Logging of events (for security, for profiling, for debugging)
➜ Tracking (tracking a moving object with separate cameras)
➜ Make (edit on one computer build on another)

U SING C LOCKS IN C OMPUTERS

Computer Clocks:
➜ Crystal oscillates at known frequency
➜ Oscillations cause timer interrupts
Based on actual time
➜ Timer interrupts update clock
➜ Cp (t): current time (at UTC time t) on machine p
Slide 14 Clock Skew: Slide 16 ➜ Ideally Cp (t) = t
➜ Crystals in different computers run at slightly different rates X Clock differences causes clocks to drift
➜ Clocks get out of sync ➜ Must regularly synchronise with UTC
➜ Skew: instantaneous difference
➜ Drift: rate of change of skew

Timestamps:
➜ Used to denote at which time an event occurred

P HYSICAL C LOCKS 7 S YNCHRONISING P HYSICAL C LOCKS 8

 S YNCHRONISING P HYSICAL C LOCKS C RISTIAN ’ S A LGORITHM
Time Server:
Internal Synchronisation:
➜ Has UTC receiver
➜ Clocks synchronise locally
➜ Passive
➜ Only synchronised with each other
Algorithm:
Slide 17 External Synchronisation: Slide 19
➜ Clients periodically request the time
➜ Clocks synchronise to an external time source
➜ Don’t set time backward
➜ Synchronise with UTC every δ seconds
➜ Take propagation and interrupt handling delay into account
Time Server: • (T 1 − T 0)/2
➜ Server that has the correct time • Or take a series of measurements and average the delay
➜ Server that calculates the correct time ➜ Accuracy: 1-10 millisec (RTT in LAN)

B ERKELEY A LGORITHM
N ETWORK T IME P ROTOCOL (NTP)
Hierarchy of Servers:
Time daemon
3:00 3:00 3:00 0 3:05 +5 ➜ Primary Server: has UTC clock

3:00 -10 +15

➜ Secondary Server: connected to primary
3:00 +25 -20 ➜ etc.
Slide 18 Network Slide 20
Synchronisation Modes:

2:50 3:25 2:50 3:25 3:05 3:05 Multicast: for LAN, low accuracy
(a) (b) (c)
Procedure Call: clients poll, reasonable accuracy

Symmetric: Between peer servers. highest accuracy

Accuracy: 20-25 milliseconds

C RISTIAN ’ S A LGORITHM 9 N ETWORK T IME P ROTOCOL (NTP) 10

 The relation → is a partial order:
➜ If a → b, then a causally affects b
➜ We consider unordered events to be concurrent:

Synchronisation: Example:a 6→ b and b 6→ a implies a k b

➜ Estimate clock offsets and transmission delays between two E11 E12 E13 E14
P1
nodes
Slide 21 ➜ Keep estimates for past communication Slide 23
➜ Choose offset estimate for lowest transmission delay P2
➜ Also determine unreliable servers E21 E22 E23 E24
Real Time
➜ Accuracy 1 - 50 msec
➜ Causally related: E11 → E12 , E13 , E14 , E23 , E24 , . . .
E21 → E22 , E23 , E24 , E13 , E14 , . . .
➜ Concurrent: E11 kE21 , E12 kE22 , E13 kE23 , E11 kE22 , E13 kE24 ,
E14 kE23 , . . .

L OGICAL C LOCKS Lamport’s logical clocks:

➜ Software counter to locally compute the happened-before
Event ordering is more important than physical time: relation →
➜ Events (e.g., state changes) in a single process are ordered ➜ Each process pi maintains a logical clock Li
➜ Processes need to agree on ordering of causally related events ➜ Lamport timestamp:
(e.g., message send and receive)
• Li (e): timestamp of event e at pi
Local ordering: • L(e): timestamp of event e at process it occurred at
➜ System consists of N processes pi , i ∈ {1, . . . , N }
Implementation:
Slide 22 ➜ Local event ordering →i : Slide 24
➀ Before timestamping a local event pi executes Li := Li + 1
If pi observes e before e′ , we have e →i e′ ➁ Whenever a message m is sent from pi to pj :
Global ordering: • pi executes Li := Li + 1 and sends Li with m
➜ Leslie Lamport’s happened before relation → • pj receives Li with m and executes Lj := max(Lj , Li ) + 1
➜ Smallest relation, such that (receive(m) is annotated with the new Lj )
1. e →i e′ implies e → e′ Properties:
2. For every message m, send (m) → receive(m) ➜ a → b implies L(a) < L(b)
3. Transitivity: e → e′ and e′ → e′′ implies e → e′′ ➜ L(a) < L(b) does not necessarily imply a → b

L OGICAL C LOCKS 11 L OGICAL C LOCKS 12

 Example:
Vector clocks:
E11 E12 E13 E14 E15 E16 E17
P1 ➜ At each process, maintain a clock for every other process
1 2 3 4 5 6 7 ➜ I.e., each clock Vi is a vector of size N
➜ Vi [j] contains i’s knowledge about j’s clock
1 2 3 4 7
P2 Implementation:
E21 E22 E23 E24 E25 ➀ Initially, Vi [j] := 0 for i, j ∈ {1, . . . , N }
Real Time ➁ Before pi timestamps an event: Vi [i] := Vi [i] + 1
Slide 25 Slide 27
➂ Whenever a message m is sent from pi to pj :
Total event ordering: • pi executes Vi [i] := Vi [i] + 1 and sends Vi with m
➜ Complete partial to total order by including process identifiers • pj receives Vi with m and merges the vector clocks Vi and
➜ Given local time stamps Li (e) and Lj (e′ ), we define global time Vj :
stamps hLi (e), ii and hLj (e′ ), ji 
➜ Lexicographical ordering: hLi (e), ii < hLj (e′ ), ji iff  max(Vj [k], Vi [k]) + 1 , if j = k
Vj [k] :=
• Li (e) < Lj (e′ ) or  max(Vj [k], Vi [k]) , otherwise
• Li (e) = Lj (e′ ) and i < j

V ECTOR C LOCKS Properties:

➜ For all i, j, Vi [i] ≥ Vj [i]
Main shortcoming of Lamport’s clocks: ➜ a → b iff V (a) < V (b) where
➜ L(a) < L(b) does not imply a → b • V = V ′ iff V [i] = V ′ [i] for i ∈ {1, . . . , N }
➜ We cannot deduce causal dependencies from time stamps: • V ≥ V ′ iff V [i] ≥ V ′ [i] for i ∈ {1, . . . , N }
E11 E12
• V > V ′ iff V ≥ V ′ ∧ V 6= V ′
P1 • V kV ′ iff V 6> V ′ ∧ V ′ 6> V
1 2
E21 E22
P2
Example:
Slide 26 1 3 Slide 28
E31 E32 E33
P3 E11 1 2 E12 E13 6
1 2 3
P1
Real Time (1,0,0) (2,0,0) (3,4,1)
E21 1 3 E22 E23 4 E24 5
➜ We have L1 (E11 ) < L3 (E33 ), but E11 6→ E33 P2
(0,1,0) (2,2,0) (2,3,1) (2,4,1)
➜ Why?
1 E31 E32 2
• Clocks advance independently or via messages P3
(0,0,1) (0,0,2)
• There is no history as to where advances come from Real Time
➜ For L1 (E12 ) and L3 (E32 ), 2 = 2 versus (2, 0, 0) 6= (0, 0, 2)

V ECTOR C LOCKS 13 G LOBAL S TATE 14

 C ONSISTENT C UTS
Determining global properties:
➜ We need to combine information from multiple nodes
➜ Without global time, how do we know whether collected local
Slide 29 G LOBAL S TATE Slide 31 information is consistent?
➜ Local state sampled at arbitrary points in time surely is not
consistent
➜ We need a criterion for what constitutes a globally consistent
collection of local information

Local history:
➜ N processes pi , i ∈ {1, . . . , N }
G LOBAL S TATE ➜ For each pi ,
Determining global properties: • event series he0i , e1i , e2i , . . .i
➜ Distributed garbage collection: • is called pi ’s history denoted by hi .
Do any references exist to a given object? • May be finite or infinite
Slide 30 ➜ Distributed deadlock detection: Slide 32 ➜ We denote by hki a k-prefix of hi .
Do processes wait in a cycle for each other? ➜ Each event eji is either a local event or a communication event
➜ Distributed termination detection:
Process state:
Did a set of processes cease all activity? (Consider messages in
transit!) ➜ State of process pi immediately before event eki denoted ski
➜ State ski records all events included in the history hk−1
i

➜ Hence, s0i refers to pi ’s initial state

C ONSISTENT C UTS 15 C ONSISTENT C UTS 16

 Global history and state: cut 1 cut 2
➜ Using a total event ordering, we can merge all local histories
0 0 1 1
into a global history: P3 s
3
r
3
s
3
r
3

N
[
H= hi
i=1 0 1 0 1
Slide 33 Slide 35 P2 r
2 r
2
s
2
s
2 s
2
2
➜ Similarly, we can combine a set of local states s1 , . . . , sN into a
global state:
0 0 1
P1 s r
S = (s1 , . . . , sN ) 1 1 r
1

➜ Which combination of local state is consistent?

Consistent cut:
Cuts: ➜ We call a cut consistent iff,
➜ Similar to the global history, we can define cuts based on
for all events e′ ∈ C, e → e′ implies e ∈ C
k-prefixes:
N
[ ➜ A global state is consistent if it corresponds to a consistent cut
C= hci i ➜ Note: we can characterise the execution of a system as a
i=1
sequence of consistent global states
➜ hci i is history of pi up to and including event eci i
Slide 34 Slide 36 S0 → S1 → S2 → · · ·
➜ The cut C corresponds to the state

S = (sc11 +1 , . . . , scNn +1 )
Linearisation:
➜ The final events in a cut are its frontier: ➜ A global history that is consistent with the happened-before
relation → is also called a linearisation or consistent run
{eci i | i ∈ {1, . . . , N }}
➜ A linearisation only passes through consistent global states
➜ A state S ′ is reachable from state S if there is a linearisation that
passes thorough S and then S ′

C ONSISTENT C UTS 17 C HANDY & L AMPORT ’ S S NAPSHOTS 18

 C HANDY & L AMPOR T ’ S S NAPSHOTS
➜ Determines a consistent global state
➜ Takes care of messages that are in transit P1
*
➜ Useful for evaluating stable global properties
m3
m1
Properties: *
P2
Slide 37 ➜ Reliable communication and failure-free processes Slide 39
➜ Point-to-point message delivery is ordered m2
➜ Process/channel graph must be strongly connected
P3
➜ On termination, *
• processes hold only their local state components and
• a set of messages that were in transit during the snapshot.

Outline of the algorithm:

➀ One process initiates the algorithm by
• recording its local state and
• sending a marker message over each outgoing channel
➁ On receipt of a marker message over incoming channel c,
• if local state not yet saved, save local state and send marker
messages, or
• if local state already saved, channel snapshot for c is
Slide 38 Slide 40 C ONCURRENCY
complete
➂ Local contribution complete after markers received on all
incoming channels

Result for each process:

➜ One local state snapshot
➜ For each incoming channel, a set of messages received after
performing the local snapshot and before the marker came
down that channel

C HANDY & L AMPORT ’ S S NAPSHOTS 19 C ONCURRENCY 20

 C ONCURRENCY
Concurrency in a Non-Distributed System: D ISTRIBUTED M UTUAL E XCLUSION
Typical OS and multithreaded programming problems ➜ Concurrent access to distributed resources
➜ Prevent race conditions ➜ Must prevent race conditions during critical regions
➜ Critical sections
Requirements:
Slide 41 ➜ Mutual exclusion Slide 43 ➀ Safety: At most one process may execute the critical section at
• Locks a time
• Semaphores ➁ Liveness: Requests to enter and exit the critical section
• Monitors eventually succeed
➜ Must apply mechanisms correctly ➂ Ordering: Requests are processed in happened-before ordering
• Deadlock
• Starvation

M ETHOD 1: C ENTRAL S ERVER

Simplest approach:
➜ Requests to enter and exit a critical section are sent to a lock
Concurrency in a Distributed System: server
➜ Permission to enter is granted by receiving a token
Distributed System introduces more challenges
➜ When critical section left, token is returned to the server
➜ No directly shared resources (e.g., memory)
Slide 42 ➜ No global state Slide 44
➜ No global clock 0 1 2 0 1 2 0 1 2
Request Release
➜ No centralised algorithms Request OK
OK
No reply
➜ More concurrency 3
Queue is
3 3
2
empty
Coordinator

(a) (b) (c)

D ISTRIBUTED M UTUAL E XCLUSION 21 M ETHOD 1: C ENTRAL S ERVER 22

 Properties:
Properties:
➜ Ring imposes an average delay of N/2 hops (limits scalability)
➜ Easy to implement
Slide 45 Slide 47 ➜ Token messages consume bandwidth
➜ Does not scale well
➜ Failing nodes or channels can break the ring (token might be
➜ Central server may fail
lost)

M ETHOD 2: T OKEN R ING

Implementation:
➜ All processes are organised in a logical ring structure
➜ A token message is forwarded along the ring M ETHOD 3: U SING M ULTICASTS AND L OGICAL C LOCKS
➜ Before entering the critical section, a process has to wait until
the token comes by Algorithm by Ricart & Agrawala:
➜ Must retain the token until the critical section is left ➜ Processes pi maintain a Lamport clock and can communicate
pairwise
Slide 46 Slide 48
2
➜ Processes are in one of three states:
1 3
1. Released: Outside of critical section
0 4
2. Wanted: Waiting to enter critical section
0 2 4 9 7 1 6 5 8 3
3. Held: Inside critical section
7 5
6

(a) (b)

M ETHOD 2: TOKEN R ING 23 M ETHOD 3: U SING M ULTICASTS AND L OGICAL C LOCKS 24

 E VALUATING D ISTRIBUTED A LGORITHMS
Process behaviour: General Properties:
➀ If a process wants to enter, it ➜ Performance
• multicasts a message hLi , pi i and • number of messages exchanged
• waits until it has received a reply from every process • response/wait time
➁ If a process is in Released, it immediately replies to any request • delay
Slide 49 to enter the critical section Slide 51 • throughput: 1/(delay + executiontime)
➂ If a process is in Held, it delays replying until it is finished with the • complexity: O()
critical section ➜ Efficiency
➃ If a process is in Wanted, it replies to a request immediately only • resource usage: memory, CPU, etc.
if the requesting timestamp is smaller than the one in its own
➜ Scalability
request
➜ Reliability
• number of points of failure (low is good)

M UTUAL E XCLUSION : A C OMPARISON

Enters
Messages Exchanged:
critical
region ➜ Messages per entry/exit of critical section
8
• Centralised: 3
0 0 0 • Ring: 1 → ∞
8 OK OK OK
12
8 Enters
• Multicast: 2(n − 1)
1 2 1 2 1 2 critical
12 OK region Delay:
12
Slide 50 (a) (b) (c)
Slide 52 ➜ Delay before entering critical section
• Centralised: 2
• Ring: 0 → n − 1
Properties: • Multicast: 2(n − 1)

➜ Multicast leads to increasing overhead Reliability:

(more sophisticated algorithm using only subsets of peer ➜ Problems that may occur
processes exists) • Centralised: coordinator crashes
➜ Susceptible to faults • Ring: lost token, process crashes
• Multicast: any process crashes

E VALUATING D ISTRIBUTED A LGORITHMS 25 M UTUAL E XCLUSION : A C OMPARISON 26