Distributed Systems

05. Clock Synchronization

Paul Krzyzanowski

Rutgers University
Fall 2018

September 24, 2018 © 2014-2018 Paul Krzyzanowski 1

Synchronization
Synchronization covers interactions among distributed processes

Clocks Identifying when something happened

Mutual exclusion Only one entity can do an operation at a time

Leader election Who coordinates activity?

Message consistency Does everyone have the same view of events?

Agreement Can everyone agree on a proposed value?

All of these are trivial in non-distributed systems

All of these are tricky in distributed systems

September 24, 2018 © 2014-2018 Paul Krzyzanowski 2

 Clock Synchronization

September 24, 2018 © 2014-2018 Paul Krzyzanowski 3

Why?
• Allow a process to identify "now" in a way that's
consistent with other processes on other systems

• Temporal ordering of events from concurrent processes

– Example: replication & identifying latest versions
• Last writer wins or latest version wins

September 24, 2018 © 2014-2018 Paul Krzyzanowski 4

Simple approach to replication

Data store
x=4
P0 x=1
x=4
x=8
x=8 Replica data store
P1
x=1

September 24, 2018 © 2014-2018 Paul Krzyzanowski 5

Simple approach to replication

Arrives first

Data store
x=4
P0 x=8
x=4
Inconsistent replicas!
x=8
x=8 Replica data store
P1
x=4
Arrives first

September 24, 2018 © 2014-2018 Paul Krzyzanowski 6

 Simple approach to replication
Send a time stamp with each modification request
Only newer timestamps can override older data

x=8 overrides x=4

Data store
x = 4, T=6:58
P0 x = 8, T = 7:02
x = 4, T
=6:58
=7:0 2
x = 8, T x=4 does NOT
Replica data store override x=8
x = 8, T=7:02
P1
x = 8, T = 7:02
Arrives first

There are problems with this … but physical clocks help this work most of the time

September 24, 2018 © 2014-2018 Paul Krzyzanowski 7

Logical vs. physical clocks
• Physical clocks keep time of day
– Consistent across systems

• Logical clock keeps track of event ordering

– among related (causal) events

September 24, 2018 © 2014-2018 Paul Krzyzanowski 8

 Physical clocks

September 24, 2018 © 2014-2018 Paul Krzyzanowski 9

Problem: Get two systems to agree on time
• Why is it hard?
– Two clocks hardly ever agree
– Quartz oscillators oscillate at slightly different frequencies

• Clocks tick at different rates

– Create ever-widening gap in perceived time
– Clock Drift
• Difference between two clocks at one point in time
– Relative offset
• Short-term variation in frequency
– Jitter

• Also note: astronomical time vs. relative time

– Time of day vs. count of seconds from epoch

September 24, 2018 © 2014-2018 Paul Krzyzanowski 10

Dealing with drift
Not good idea to set a clock back
– Illusion of time moving backwards can confuse message ordering
and software development environments

Go for gradual clock correction

If fast:
Make the clock run slower until it synchronizes

If slow:
Make the clock run faster until it synchronizes

September 24, 2018 © 2014-2018 Paul Krzyzanowski 11

Dealing with drift
The OS can do this:

1. Redefine the rate at which system time is advanced with each

interrupt

or

2. Read the counter but compensate for drift

Adjustment changes slope of system time:

Linear compensation function

September 24, 2018 © 2014-2018 Paul Krzyzanowski 12

Compensating for a fast clock

Computer’s time, C

Clock synchronized
offset
Linear compensation
function applied
e
ti m
ct
rfe
pe

UTC time, t

September 24, 2018 © 2014-2018 Paul Krzyzanowski 13

Compensating for a fast clock

Computer’s time, C

Now we’re drifting again!

e
ti m
ct
rfe
pe

UTC time, t

September 24, 2018 © 2014-2018 Paul Krzyzanowski 14

Resynchronizing
After synchronization period is reached
– Resynchronize periodically
– Successive application of a second linear compensating function
can bring us closer to true slope

Long-term clock stability is not guaranteed

The system clock can still drift based on changes in temperature,
pressure, humidity, and age of the crystal

Keep track of adjustments and apply continuously

– e.g., BSD adjtimex & Linux adjtimex system calls
and hwclock command

September 24, 2018 © 2014-2018 Paul Krzyzanowski 15

Going to sleep
• RTC keeps on ticking when the system is off
(or sleeping)

• OS cannot apply correction continually

• Estimate drift on wake-up and apply a correction factor

September 24, 2018 © 2014-2018 Paul Krzyzanowski 16

Getting accurate time
• Attach GPS receiver to each computer
– Accurate to ~ 40 ns

• Not practical solution for every machine

– Cost, power, convenience, environment
– Accuracy gets worse near buildings, bridges, trees, …

September 24, 2018 © 2014-2018 Paul Krzyzanowski 17

Synchronize from a time server
Simplest synchronization technique
– Send a network request to obtain the time
– Set the time to the returned value

what’s the time?

time
client
server
3:42:19

Does not account for network or processing latency

September 24, 2018 © 2014-2018 Paul Krzyzanowski 18

Cristian’s algorithm
Compensate for delays
– Note times:
• request sent: T0
• reply received: T1
– Assume network delays are symmetric

Tserver
server

request reply
client
time
T0 T1

September 24, 2018 © 2014-2018 Paul Krzyzanowski 19

Cristian’s algorithm
Client sets time to:
Tserver
server

request reply
client
time
T0 T1
!" − !$
= estimated overhead
2 in each direction

T1 −T0
T new =Tserver +
2

September 24, 2018 © 2014-2018 Paul Krzyzanowski 20

Error bounds
If the minimum message transit time (Tmin) is known:

Place bounds on accuracy of result

September 24, 2018 © 2014-2018 Paul Krzyzanowski 21

Error bounds
Tserver
server

request reply
client
T0 T1 time
Tmin Tmin
Earliest time message
Latest time message
arrives
leaves

range = T1 - T0 - 2Tmin

T1 −T0
accuracy of result = ± −T min
2

September 24, 2018 © 2014-2018 Paul Krzyzanowski 22

Cristian’s algorithm: example
• Send request at 5:08:15.100 (T0)

• Receive response at 5:08:15.900 (T1)

– Response contains 5:09:25.300 (Tserver)

• Elapsed time is T1 -T0 Note:

1 000 ms = 1 s
5:08:15.900 - 5:08:15.100 = 800 ms 1 000 000 µs = 1s

• Best guess: timestamp was generated 400 ms ago

• Set time to Tserver+ elapsed time

5:09:25.300 + 400 = 5:09.25.700
September 24, 2018 © 2014-2018 Paul Krzyzanowski 23
Cristian’s algorithm: example
If best-case message time=200 ms T0 = 5:08:15.100
T1 = 5:08:15.900
Ts = 5:09:25:300
Tserver Tmin = 200 ms
server

request reply
client
time
T0 T1
200 200

800

900 −100 800

Error = ± − 200 = ± − 200 = ±200
2 2
September 24, 2018 © 2014-2018 Paul Krzyzanowski 24
Berkeley Algorithm
• Gusella & Zatti, 1989

• Assumes no machine has an accurate time source

• Obtains average from participating computers
• Synchronizes all clocks to a fault-tolerant average

September 24, 2018 © 2014-2018 Paul Krzyzanowski 25

Berkeley Algorithm: example

master
3:00
9:
:25 2:50
10
3

3:25 2:50 9:10

1. Request timestamps from all slaves

September 24, 2018 © 2014-2018 Paul Krzyzanowski 26

Berkeley Algorithm: example

master
3:00
9:
:25 2:50
10
3

3:25 2:50 9:10

Suppose
2. Compute fault-tolerant average: max ∂=0:45

3 : 25 + 2 : 50 + 3 : 00
= 3 : 05
3

September 24, 2018 © 2014-2018 Paul Krzyzanowski 27

Berkeley Algorithm: example

master
3:00
-6
:20 +0:15
:0
-0 5

3:25 2:50 9:10

3. Send offset to each client

September 24, 2018 © 2014-2018 Paul Krzyzanowski 28

Network Time Protocol, NTP
• 1991, 1992
– Internet Standard, version 3: RFC 1305

• June 2010
– Internet Standard, version 4: RFC 5905-5908
– IPv6 support
– Improve accuracy to tens of microseconds
– Dynamic server discovery

September 24, 2018 © 2014-2018 Paul Krzyzanowski 29

NTP Goals
• Enable clients across Internet to be accurately synchronized to UTC
despite message delays
– Use statistical techniques to filter data and gauge quality of results

• Provide reliable service

– Survive lengthy losses of connectivity
– Redundant paths
– Redundant servers

• Provide scalable service

– Enable huge numbers of clients to synchronize frequently
– Offset effects of clock drift

• Provide protection against interference

– Authenticate source of data

September 24, 2018 © 2014-2018 Paul Krzyzanowski 30

NTP servers
Arranged in strata
0
– Stratum 0 = master clock
1
– 1ststratum: systems connected directly
to accurate time source 2
– 2nd stratum: systems synchronized
from 1st stratum systems 3
–…
4
– 15thstratum: systems synchronized
from 14th stratum systems

Synchronization Subnet

September 24, 2018 © 2014-2018 Paul Krzyzanowski 31

NTP Synchronization Modes
Multicast mode
– for high speed LANs
– Lower accuracy but efficient

Procedure call mode

– Cristian’s algorithm

Symmetric mode
– Peer servers can synchronize with each other to provide mutual
backup
• Usually used with stratum 1 & 2 servers
• Pair of servers retain data to improve synchronization over time

All messages are delivered unreliably with UDP (port 123)

September 24, 2018 © 2014-2018 Paul Krzyzanowski 32

NTP Clock Quality
• Precision
– Smallest increase of time that can be read from the clock
• Jitter
– Difference in successive measurements
– Due to network delays, OS delays, and clock oscillator instability

• Accuracy
– How close is the clock to UTC?

September 24, 2018 © 2014-2018 Paul Krzyzanowski 33

NTP messages
• Procedure call and symmetric mode
– Messages exchanged in pairs: request & response
• Time encoded as a 64 bit value:
– Divide by 232 to get the number of seconds since Jan 1 1900 UTC

• NTP calculates:
– Offset for each pair of messages (θ)
• Estimate of time offset between two clocks
– Delay (δ)
• Travel time: ½ of total delay minus remote processing time
– Dispersion
• Maximum offset error relative to reference clock

• Use this data to find preferred server:

– Probe multiple servers – each several times
– Pick lowest dispersion … at the lowest stratum if tied

September 24, 2018 © 2014-2018 Paul Krzyzanowski 34

SNTP
Simple Network Time Protocol
– Based on Unicast mode of NTP – subset of NTP, not new protocol
– Operates in multicast or procedure call mode
– Recommended for environments where server is root node and
client is leaf of synchronization subnet
– Root delay, root dispersion, reference timestamp ignored

v3 RFC 2030, October 1996

v4 RFC 5905, June 2010

September 24, 2018 © 2014-2018 Paul Krzyzanowski 35

SNTP Example
T2 T3
server

request reply
client
time
T1 T4

Round-trip network delay: Time offset:

($% −$' ) + ($* − $+ )
- = $+ − $' − ($* − $% ) !=
2

September 24, 2018 © 2014-2018 Paul Krzyzanowski 36

SNTP Example
T2 T3
server

request reply
client
time
T1 T4

Round-trip network delay: Time offset:

($% −$' ) + ($* − $+ )
- = $+ − $' − ($* − $% ) !=
2

September 24, 2018 © 2014-2018 Paul Krzyzanowski 37

SNTP example
T2=800 T3=850
server

request reply
time
client
T1=1100 T4=1200
Offset =
((800 - 1100) + (850 - 1200)) / 2
= ((-300) + (-350)) / 2
= -650 / 2 = -325 Time offset:
($% −$' ) + ($* − $+ )
!=
2
Set time to T4 + t
= 1200 - 325 = 875

September 24, 2018 © 2014-2018 Paul Krzyzanowski 38

SNTP = Cristian’s algorithm
T2 T3
server

request reply
Ts time
client
T1 T4

!$ + !&
Just define !" =
2

September 24, 2018 © 2014-2018 Paul Krzyzanowski 39

Key Points: Physical Clocks
• Cristian’s algorithm & SNTP
– Set clock from server
– But account for network delays
– Error: uncertainty due to network/processor latency
• Errors are additive
• Example: ±10 ms and ±20 ms = ±30 ms

• Adjust for local clock drift

– Linear compensating function

September 24, 2018 © 2014-2018 Paul Krzyzanowski 40

 Precision Time Protocol

September 24, 2018 © 2014-2018 Paul Krzyzanowski 41

PTP: IEEE 1588 Precision Time Protocol
• Designed to synchronize clocks on a LAN to sub-
microsecond precision
– Designed for LANs, not global: low jitter, low latency
– Timestamps ideally generated at the MAC or PHY layers to minimize
delay and jitter

• Determine master clock

– Use a Best Master Clock algorithm to determine which clock in the
network is most precise
– Other clocks become slaves

• Two phases in synchronization

1. Offset correction
2. Delay correction

September 24, 2018 © 2014-2018 Paul Krzyzanowski 42

PTP: Choose the “best” clock
Best Master Clock
• Distributed election based on properties of clocks
• Criteria from highest to lowest:
– Priority 1 (admin-defined hint)
– Clock class
– Clock accuracy
– Clock variance: estimate of stability based on past syncs
– Priority 2 (admin-defined hint #2)
– Unique ID (tie-breaker)

September 24, 2018 © 2014-2018 Paul Krzyzanowski 43

PTP: Master initiates sync

master T1
syn time
c

slave
T2 time

Master initiates the protocol by sending a sync message containing a timestamp

Slave timestamps arrival with a timestamp from its local clock

Offset + Delay = T2 - T1

September 24, 2018 © 2014-2018 Paul Krzyzanowski 44

PTP: Send delay request

master T1 T4
time

st
ue
syn

req
c

lay
slave de
T2 T3 time

Slave needs to figure out the network delay. Send a delay request

Note the time it was sent.

September 24, 2018 © 2014-2018 Paul Krzyzanowski 45

PTP: Receive delay response

master T1 T4
time

de
st

lay
ue
syn

res
req
c

po
lay

nse
slave de
T2 T3 time

Master marks the time of arrival and returns it in a delay response

Delay response = Delay - Offset = T4 – T3

September 24, 2018 © 2014-2018 Paul Krzyzanowski 46

PTP: Slave computes offset

master T1 T4
time

de
st

lay
ue
syn

res
req
c

po
lay

nse
slave de
T2 T3 time

master_slave_difference = T2 – T1 = delay + offset

slave_master_difference = T4 – T3 = delay – offset
master_slave_difference – slave_master_difference = 2(offset)
T2 – T1 – T4 + T3 = 2 (offset)
offset = (T2 – T1 – T4 + T3) / 2

September 24, 2018 © 2014-2018 Paul Krzyzanowski 47

NTP vs. PTP
• Range
– NTP: nodes widely spread out on the Internet
– PTP: local area networks

• Accuracy
– NTP usually several milliseconds on WAN
– PTP usually sub-microsecond on LAN

September 24, 2018 © 2014-2018 Paul Krzyzanowski 48

 The end

September 24, 2018 © 2014-2018 Paul Krzyzanowski 49