Clock Synchronization

Embedded Systems
 2

Outline
Ø Time and Clocks
Ø Time Measurement
Ø DenseTime versus SparseTime
Ø Internal Clock Synchronization
Ø External Clock Synchronization

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 3
Instants are
Temporally Ordered
The continuum of real time can be modelled by a directed
timeline consisting of an infinite set {𝑇} of instants with the
following properties:
1. {𝑇} is an ordered set, i.e., if 𝑝 and 𝑞 are two instants, then
(1) 𝑝 is simultaneous with 𝑞 or
(2) 𝑝 precedes 𝑞 or relations are mutually
exclusive
(3) 𝑞 precedes 𝑝
The order of instants on the timeline is called the temporal order.
2. {𝑇} is a dense set. This means that, if 𝑝 ≠ 𝑟, there is at least
one 𝑞 between 𝑝 and 𝑟.

RealTime
p q r
Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]
 4
Durations
and Events
Ø A section of the time line is called a duration.
Ø An event is a happening at an instant of time.
Ø An event does not have a duration.
Ø If two events occur at an identical instant, then the two
events are said to occur simultaneously.
Ø Instants are totally ordered, while events are only partially
ordered, since simultaneous events are not in the order
relation.
Ø Events can be totally ordered if another criterion is
introduced to order events that occur simultaneously (e.g.,
in a distributed computer system the node number can be
used to order simultaneous events).

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 5

Causal Order
Ø Definition of causality by a mark method without
reference to time: "If event 𝑒1 is a cause of event 𝑒2,
then a small variation (a mark) in 𝑒1 is associated with
small variation in 𝑒2, whereas small variations in 𝑒2 are
not necessarily associated with small variations in 𝑒1."
Ø Example: Suppose there are two events 𝑒1 and 𝑒2:
Ø 𝑒1 Somebody enters a room.
Ø 𝑒2 The telephone starts to ring.
Ø Consider the following two cases
Ø (i) 𝑒2 occurs after𝑒1
Ø (ii) 𝑒1 occurs after 𝑒2

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 6

Alarm Analysis
Ø A primary alarm event leads to a shower of secondary
alarm events (alarm shower).
Ø If the (partial) temporal order between alarm events
has been established, it is possible to exclude an alarm
event that definitely occurred later than other alarm
events from being the primary event.
Ø A precise global time-base helps to determine the
event set that is in this definitely-occurred-later
relation.
Ø Delivery order of messages in computer networks:
Ø Temporal?
Ø Causal?
Ø Consistent?
Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]
 7
Clocks and
Timestamps
Ø A clock is a device that contains a counter and increments this
counter periodically (microticks)
Ø Assumption of an external observer with an atomic clock that
has a granularity that is much smaller than the duration of any
intervals of interest:
Ø Reference clock 𝑧
Ø High precision, e.g., 1 femto second(10−15sec)
Ø The granularity of a clock 𝑐 is the number of microticks of the
reference clock between any two consecutive microticks of 𝑐.
Ø Given a clock and an event, a timestamp of the event is the
state of clock immediately after the event occurrence,
denoted by
Ø 𝑐𝑙𝑜𝑐𝑘 (𝑒𝑣𝑒𝑛𝑡)
Ø We assume that relativistic effects can be neglected.
Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]
 8

Clock Drift
Ø Clock Drift:
" "
𝑧(𝑚𝑖𝑐𝑟𝑜𝑡𝑖𝑐𝑘!#$ ) − 𝑧(𝑚𝑖𝑐𝑟𝑜𝑡𝑖𝑐𝑘! )
𝐷𝑟𝑖𝑓𝑡!" =
𝑛"
Ø where:
Ø 𝑘 the clock we are looking at
Ø 𝑛𝑘 nominal number of ticks of the reference clock within a
granule of clock 𝑘
Ø 𝑧 timestamp of the reference clock

Ø Drift Rate: " "

" 𝑧(𝑚𝑖𝑐𝑟𝑜𝑡𝑖𝑐𝑘!#$ ) − 𝑧(𝑚𝑖𝑐𝑟𝑜𝑡𝑖𝑐𝑘! )
r! = −1
𝑛"

Ø Perfect clock has drift rate of 0

Ø Real clocks have drift rates from 10−2 to 10−8
Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]
 9
Failure Modes
of a Clock

Time of the Perfect Clock

Reference Error in Drift
Clock A good clock with a
(rate error) bounded driftrate r stays
in the shaded area

Error in Counter
(state error)

Time of the LocalClock

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 10

Precision
Ø Offset between two clocks j,k at tick i:
%" %
𝑜𝑓𝑓𝑠𝑒𝑡! = 𝑧(𝑚𝑖𝑐𝑟𝑜𝑡𝑖𝑐𝑘! ) − 𝑧(𝑚𝑖𝑐𝑟𝑜𝑡𝑖𝑐𝑘!" )

Ø Given an ensemble of clocks {1, 2, . . . , 𝑛}, the

maximum offset between any two clocks of the
ensemble is called the precision of the ensemble at
microtick 𝑖:
%"
Π! = max 𝑜𝑓𝑓𝑠𝑒𝑡!
∀1≤𝑗,𝑘≤n

Ø The process of mutual resynchronization of an

ensemble of clocks in order to maintain a bounded
precision is called internal synchronization.

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 11

Accuracy
Ø The offset of clock 𝑘 w.r.t. the reference
clock 𝑧 at tick 𝑖 is called the accuracy.
Ø Accuracy denotes the maximum offset
of a given clock from the external time
reference.
Ø This process of resynchronization of a
clock with the reference clock is called
external synchronization.
Ø If all clocks are externally synchronized
with an accuracy 𝐴, then the ensemble
is also internally synchronized with a
precision of at most 2 · 𝐴.
Ø The opposite is not true.
Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]
 12

Time Standards
Ø International AtomicTime (TAI):
Ø TAI is a physical time standard that defines the second as the
duration of 9 192 631 770 periods of the radiation of a specified
transition of the cesium atom 133.
Ø TAI is a chronoscopic timescale, i.e., a timescale without any
discontinuities.
Ø Defines the epoch, the origin of time measurement, as January 1,
1958 at 00:00 hours, and continuously increases the counter.
Ø Universal Time Coordinated(UTC):
Ø UTC is an astronomical time standard that is the basis for the time
on the "wall clock".
Ø Duration of the second conforms to TAIstandard.
Ø Number of seconds in an hour is occasionally modified by inserting
leap seconds to maintain synchrony between the wall-clock time and
the astronomical phenomena, like day and night.
Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]
 13

A Problem with the Leap Second

Ø Software Engineering Notes of March 1996 (p.16) reports on a
problem that occurred when a leap second was added at
midnight on New Year's Eve 1995. The leap second was added,
but the date inadvertently advanced to Jan. 2.
Ø The synchronization of AP radio broadcast network depends on
the official time signal, and this glitch affected their operation
for several hours until the problem was corrected.
Ø Making corrections at midnight is obviously risky:
1. The day increments to January 1, 1996, 00:00:00.
2. You reset the clock to 23:59:59, back one second.
3. The clock continues running.
4. The day changes again, and it's suddenly, January 2, 1996,
00:00:00.

R. Obermaisser,Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 14

Global Time
Ø If there is a single reference clock available,all time
measurements can be performed by this single clock that
acts as a common “global”time.
Ø In a distributed system we synchronize clocks in order to
generate a common notion of time, a “global time” in the
distributed system.
Ø However, such a global time is an abstract notion thatcan
only be approximated by the clocks in the nodes.
Ø It is possible to select a subset of the microticks of each local
clock k for the generation of the local implementation of a
global notion of time. We call such a selected local microtick
a macrotick (or a tick) of the global time.

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 15

If no global time-base is available, then

Ø There are 𝑛 independent local time references and the
timestamps can only be related if they originate from the
same clock.
Ø Interval measurements between events observed at
different nodes are limited by the end-to-end delay jitter.
Ø The delay jitter of (ET) communication system
determines the jitter in the control loops – this may be
unacceptable for many real-time control applications.
Ø State estimation is very difficult, since the precise point
in time of measurement of a process variable is not
known.

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 16
Requirements of a
Global Time Base
Ø Chronoscopic behavior, i.e. no discontinuities, even at the
points of resynchronization
Ø Known precision
Ø High dependability
Ø Metric of the physical second

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 17
Reasonableness
Condition

Ø The global time t is called reasonable, if all local implementations

of the global time satisfy the following reasonableness condition
for the global granularity 𝑔 of amacrotick:
𝑔> Π
Ø This reasonableness condition ensures that the synchronization
error is bounded to less than one macrogranule, i.e., the
duration between two macroticks.
Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]
 18
One Tick Difference:
What Does it Mean?

Ø Because of the accumulation of the synchronization error

and the digitalization error, it is not possible to reconstruct
the temporal order of two events from the knowledge that
the global timestamps differ by one.

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 19
Interval
Measurement

It follows: (𝑑𝑜𝑏𝑠 – 2𝑔) < 𝑑𝑡𝑟𝑢𝑒 < (𝑑𝑜𝑏𝑠 + 2𝑔)

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 20

Π/Δ Precedence
Ø Given a set of events {𝐸} and two durations 𝜋 and Δ
where 𝜋 ≪ Δ , such that for any two elements 𝑒𝑖
and 𝑒𝑗 of this set the following condition holds:
[ z(e i ) - z(e j ) £ p ] Ú [ z(e i ) - z(e j ) > D ]

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 21
Fundamental Limits
to Time Measurement
Ø Given a distributed system with a reasonable global
timebase with granularity 𝑔. Then the following fundamental
limits to time measurement must be observed:
Ø If a single event is observed by two nodes, there is always the
possibility that the timestamps will differ by one tick
Ø Let us assume that 𝑑𝑜𝑏𝑠 is the observed duration of an interval.
Then the true duration 𝑑𝑡𝑟𝑢𝑒 is
𝑑&'( − 2𝑔 < 𝑑)*+, < 𝑑&'( + 2𝑔

Ø The temporal order of events can only be recovered, if

the observed time difference 𝑑𝑜𝑏𝑠 ≥ 2𝑔
Ø The temporal order of events can always be recovered,
if the event set is 0/3𝑔 precedent.

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 22

Dense Time
Ø A timebase is dense if events can occur at any point of
the timeline.
Ø Consequences of these fundamental limits of time
measurement in distributed systems:
Ø If a single event occurring on a dense timebase is
observed by two nodes of the distributed system (e.g., to
achieve redundancy in the observations), then an explicit
agreement protocol is needed to establish a consistent
view of the temporal point of event occurrence.
Ø If two events occur on a dense timebase, then it is
impossible to always recover the temporal order of the
events if they occur within an interval of 3𝑔.

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 23
Inconsistent Order
on Dense Time Base
Ø Event 𝑒1 is observed by node 𝑗 at time 2 and by node 𝑚 at time 1
Ø Event 𝑒2 is only observed by node 𝑘 that reports its observation
"𝑒2 occurred at 3" to node 𝑗 and node 𝑚.
Ø Node 𝑗 calculates timestamp difference of 1 and concludes the
events occurred at about the same time and cannot be ordered.
Ø Node 𝑚 calculates a timestamp difference of 2 and concludes that
𝑒1 has definitely occurred before 𝑒2.

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 24

Sparse Time Base

Ø If the occurrence of events is restricted to some active
intervals with duration p with an interval of silence of
duration D between any two active intervals, then we
call the timebase p/D-sparse, or sparse for short.

0 1 2 3 4 5 6 7 8 9

Time
p D p D p

Ø Events are only allowed to occur at subintrevals of the

timeline

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 25

Space/Time Lattice

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 26

Time and State

Ø In abstract system theory (Mesarovic, p.45), the notion of state
is introduced in order to separate the past from the future:
Ø “The state enables the determination of a future output solely
on the basis of the future input and the state the system is in. In
other word, the state enables a “decoupling” of the past from
the present and future. The state embodies all past history of a
system. Knowing the state “supplants” knowledge of the past.
Apparently, for this role to be meaningful, the notion of past
and future mustbe relevant for the system considered.”

A precise concept of time is a prerequisite for a

precise concept of state.

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 27
Consistent Distributed
State based on Sparse Time Base
Ø Interval of silence with clearly defined distributed state of the system.
Ø Interval of activity with communication or computational activities.

Past Future

state is only defined

during intervall Δ Time
Δ
π π

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 28
The Synchronization
Condition

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 29
Malicious
(Byzantine) Clocks
11:00 13:00

good good
A B

C to A: 9:00 C to B 15:00
C
two faced malicious

Ø Total Number N of clocks must be N ≥ (3k +1), where k is

the number of malicious (Byzantine) faults.

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 30

Central Master Algorithm

Ø Unique node (central master) periodically sends its time
counter in synchronization messages to the slave nodes
Ø As soon as a slave receives a new time value from the
master, the slave records the state of its local time counter
as the time of message arrival
Ø Difference between the master's time contained in the
synchronization message and the recorded slave's time of
message arrival, corrected by the latency of the message
transport, is measure of deviation between clocks
Ø Slave corrects its clock by this deviation to bring it into
agreement with the master's clock.
Ø Precision of central Master Algorithm: 𝚷𝒄𝒆𝒏𝒕𝒓𝒂𝒍 = 𝜺 + 𝚪

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 31

Delay Jitter
Ø The difference dmax − dmin is called the delay jitter 𝜀.
Ø Typical protocol execution time in standard OSI
protocols (with time redundant transmissions)

dmin delay jitter dmax

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]
 32
Distributed
Clock Synchronization
Ø Typically, distributed fault-tolerant clock resynchronization
proceeds in three distinct phases:
1. Every node acquires knowledge about the state of the global
time counters in all other nodes by message exchanges among
the nodes.
2. Every node analyzes the collected information to detect errors
and executes the convergence function to calculate a
correction value for the node's local global time counter.
3. The local time counter of the node is adjusted by the
calculated correction value.
Ø The algorithms differ in the way in which they collect
the time values from the other nodes, in the type of
convergence function used, and in the way in which the
correction value is applied to the time counter.
Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]
 How well can we
33

synchronize clocks?

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 34

Convergence Function
Ø Examples:
Ø Average Algorithm
Ø Fault-Tolerant Average (FTA)
Ø Fault-Tolerant Midpoint
Ø Interactive Consistency Algorithms

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 35
Fault-Tolerant
Algorithm
Every node measures the time differences between its own
clock and all other clocks and rejects the 𝑘 extreme
differences, where 𝑘 is the number of Byzantine faults that
are to be tolerated.

If 𝑘 = 1,then

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 36
Fault Tolerant
Average Algorithm
Worst scenario happens, if the Byzantine clock sets its
(faulty) time values at different nodes at a different
corner of the
Precision window:

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 37

Precision of the FTA

Convergence Function
𝑘Π
Φ 𝑁, 𝑘, 𝜀 = + 𝜀
𝑁 − 2𝑘
Precision
𝑁 − 2𝑘
Π 𝑁, 𝑘, 𝜀, Γ = 𝜀 + Γ = 𝜀 + Γ 𝜇(𝑁, 𝑘)
𝑁 − 3𝑘

Where 𝜇(𝑁, 𝐾) is called the Byzantine error factor and is

tabulated in the following table:

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 38

Fault Tolerant
Midpoint
1. Measured values are ordered according to size.
2. k largest and k smallest values are deleted from this list.
3. Smallest and largest values are selected from the
remaining values.
4. These two values are added and divided by two.

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 39
State Correction
Versus Rate Correction
Ø State correction
Ø Correction term calculated by the convergence function is immediately applied
Ø Simple to apply
Ø Disadvantage of generating a discontinuity in the time base (e.g.,clocks are set
backwards and the same nominal time value is reached twice)
Ø Rate correction
Ø Modify clock rate to slow down or speed up during the next resynchronization
interval
Ø Changing the number of microticks in some of the macroticks
Ø Average of rate correction terms among all clocks should be close to zero for
avoiding common-mode drift
Correction Correction
Deviation

Deviation

Embedded
R. Obermaisser,Embedded
Systems Systems [Ref.:H.Kopetz,Real-Time Systems 2011]
 40
Interactive
Consistency Algorithms
Ø To overcome the problem of the Byzantine error factor,
every node sends its view of the ensemble to all other
nodes in order that every node has the global view of
the situation, i.e., it can find out which node has been
cheating.
Ø Every node takes this consistent global view, i.e., the
matrix of time vectors, as the basis of the correction
factor calculation.
Ø Pro: 𝜇(𝑁, 𝑘) = 1
Ø Con: Extra round of communication

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 41
Limit to Internal
Clock Synchronization
Ø Lundelius and Lynch have shown that in a system with
𝑁 clocks and a delay jitter 𝜀 it is impossible to
synchronize clocks better

1
∆ABC = 𝜀 1 −
𝑁

Ø The proof assumes that all clocks have perfect

oscillators.

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 42

Critical Parameters
Ø What are the critical parameters that determine the
quality of the global time base?
Ø Drift offset Γ = 2 · 𝑅𝑠𝑦𝑛𝑐 · 𝜌
Ø Delay jitter 𝜀 = d max −𝑑𝑚𝑖𝑛
Ø The occurrence of Byzantine failures is a rare event
Ø The delay jitter is smallest, if the clock synchronization
is performed very close to the physical level – by the
hardware.
Ø Compared to the delay jitter, the algorithmic effects
are small.

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 43
Quality Attributes
of a Global Time Base
Ø Precision: Maximum difference between the respective
ticks of the clocks in an ensemble.
Ø Accuracy: Maximum difference between the tick of a
clock and the corresponding tick of the external
reference clock.
Ø Fault-Tolerance: Number and types of faults the clocking
system can tolerate.
Ø Blackout Survivability: Blackout duration that can be
tolerated without losing synchronism.

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]

 44
Blackout
Survivability Interval
Ø What is the maximum duration of a total communication
blackout that can be tolerated without loss of synchronism?

Ø How long will it take until the clocks drift so far from each other
that they will leave the slack interval?
𝑔 − Π 𝑔 Granularity
𝑑 ()*+,-./ = P Precision
2𝜌 r Drift rate
Ø (Assume Byzantine error factor 𝜇 = 1)
Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]
 45

Summary
and Key Takeaways
Ø Instants, Durations,Events and Clocks
Ø Causal Order and TemporalOrder
Ø Clock Drift
Ø GlobalTime, Precision and Accuracy
Ø Reasonableness Condition
Ø Fundamental Limits to TimeMeasurement
Ø Dense and SparseTime
Ø Central Master and Distributed Clock Synchronization
Ø Fault-Tolerant Algorithms (FTA, FTM, Interactive Consistency)
Ø State and Rate Correction
Ø Blackout Survivability

Embedded Systems [Ref.:H.Kopetz,Real-Time Systems 2011]