Unit II LOGICAL TIME AND GLOBAL

STATE
Part 1 : LOGICAL TIME
i. Physical Clock Synchronization: NTP
ii. Logical Time
iii. A Framework for a System of Logical Clocks
iv. Scalar Time
v. Vector Time

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 1 / 67

 Distributed Computing: Principles, Algorithms, and
Systems

Physical Clock Synchronization:

NTP

Motivation
In centralized systems, there is only single clock. A process gets
the time by simply issuing a system call to the kernel.
In distributed systems, there is no global clock or common
memory. Each processor has its own internal clock and its own
notion of time.
These clocks can easily drift seconds per day, accumulating
significant errors over time.
Also, because different clocks tick at different rates, they may
not remain always synchronized although they might be
synchronized when they start.
This clearly poses serious problems to applications that
depend on a synchronized notion of time.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 2 / 67

 Distributed Computing: Principles, Algorithms, and
Systems

Physical Clock Synchronization:

NTP
Motivation

For most applications and algorithms that run in a distributed

system, we need to know time in one or more of the following
contexts:
◮ The time of the day at which an event happened on a specific
machine in the network.
◮ The time interval between two events that happened on different
machines in the network.
◮ The relative ordering of events that happened on different
machines in the network.
Unless the clocks in each machine have a common notion of time,
time-based queries cannot be answered.
Clock synchronization has a significant effect on many problems
like secure systems, fault diagnosis and recovery, scheduled
operations, database systems, and real-world clock values.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 3 / 67

 Distributed Computing: Principles, Algorithms, and
Systems

Physical Clock Synchronization:

NTP

Clock synchronization is the process of ensuring that physically

distributed processors have a common notion of time.
Due to different clocks rates, the clocks at various sites may
diverge with time and periodically a clock synchronization must
be performed to correct this clock skew in distributed systems.
Clocks are synchronized to an accurate real-time standard
like UTC (Universal Coordinated Time).
Clocks that must not only be synchronized with each other but
also have to adhere to physical time are termed physical clocks.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 4 / 67

 Distributed Computing: Principles, Algorithms, and
Systems

Physical Clock Synchronization:

NTP
Definitions and Terminology
Let Ca and Cb be any two clocks.
Time: The time of a clock in a machine p is given by the
function Cp (t), where Cp (t) = t for a perfect clock.
Frequency: Frequency is the rate at which a clock progresses.
The ′
frequency at time t of clock Ca is Ca (t ).
Offset: Clock offset is the difference between the time reported
by a clock and the real time. The offset of the clock Ca is given by
Ca (t) − t. The offset of clock Ca relative to Cb at time t ≥ 0 is
given by Ca(t) − Cb (t).
is (C ′ (t)The
Skew: − Cskew
′ (t)).
of a clock is the difference in the frequencies of
a b
Drift
the (rate):
clock and the The drift ofclock.
perfect clockThe
Ca isskew
the second derivative
of a clock of the
Ca relative
clock
to
with value
clock Cb attotime
respect time,t namely, C ′′ (t). The drift of clock Ca relative
C at time t is Ca (t ) − Cb
tob clock
′′ ′′
(t ).

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 5 / 67

 Distributed Computing: Principles, Algorithms, and
Systems

Physical Clock Synchronization:

NTP

Clock Inaccuracies
Physical clocks are synchronized to an accurate real-time
standard like UTC (Universal Coordinated Time).
However, due to the clock inaccuracy discussed above, a timer
(clock) is said to be working within its specification if (where
constant ρ is the maximum skew rate specified by the
manufacturer.)
d
1−ρ≤ ≤1+ρ
C
dt
(1)
Figure 3.5 illustrates the behavior of fast, slow, and perfect
clocks with respect to UTC.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 6 / 67

 Distributed Computing: Principles, Algorithms, and
Systems

Physical Clock Synchronization:

NTP

Fast Clock
dC/dt >
1
Perfect Clock

Clock time, C
dC/dt = 1

Slow Clock
dC/dt < 1

UTC, t

Figure 3.5: The behavior of fast, slow, and perfect clocks with
respect to UTC.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 7 / 67

 Distributed Computing: Principles, Algorithms, and
Systems

Physical Clock Synchronization:

NTP

Offset delay estimation method

The Network Time Protocol (NTP) which is widely used for
clock synchronization on the Internet uses the The Offset
Delay Estimation method.
The design of NTP involves a hierarchical tree of time
servers.
◮ The primary server at the root synchronizes with the UTC.
◮ The next level contains secondary servers, which act as a
backup to the primary server.
◮ At the lowest level is the synchronization subnet which has the
clients.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 8 / 67

 Distributed Computing: Principles, Algorithms, and
Systems

Physical Clock Synchronization:

NTP

Clock offset and delay estimation:

In practice, a source node cannot accurately estimate the local time
on the target node due to varying message or network delays between
the nodes.
This protocol employs a common practice of performing several
trials and chooses the trial with the minimum delay.
Figure 3.6 shows how NTP timestamps are numbered and
exchanged between peers A and B.
Let T1, T2, T3, T4 be the values of the four most recent timestamps
as shown. Assume clocks A and B are stable and running at the
same speed.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 9 / 67

 Distributed Computing: Principles, Algorithms, and
Systems

T1 T2
B

A
T3

T4

Figure 3.6: Offset and delay

estimation.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 10 / 67

 Distributed Computing: Principles, Algorithms, and
Systems

Let a = T 1 − T 3 and b = T 2 −
. network delay difference from A to B and from B to A,
T 4the
If
called differential delay, is small, the clock offset θ and
roundtrip delay δ of B relative to A at time T4 are
approximately given by the following.
a+
θ , δ= a−b
b2
= (2)
Each NTP message includes the latest three timestamps T1, T2
and T3, while T4 is determined upon arrival.
Thus, both peers A and B can independently calculate delay and
offset using a single bidirectional message stream as shown in
Figure 3.7.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 11 / 67

 Distributed Computing: Principles, Algorithms, and
Systems

Server A T i-2 T i-1

T i-3 Ti
Server B

Figure 3.7: Timing diagram for the two

servers.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 12 / 67

 Distributed Computing: Principles, Algorithms, and
Systems

The Network Time Protocol synchronization

protocol.

A pair of servers in symmetric mode exchange pairs of timing

messages.
A store of data is then built up about the relationship
between the two servers (pairs of offset and delay).
Specifically, assume that each peer maintains pairs (Oi ,Di ),
where
Oi - measure of offset (θ)
Di - transmission delay of two messages (δ).
The offset corresponding to the minimum delay is chosen.
Specifically, the delay and offset are calculated as follows.
Assume that message m takes time t to transfer and m′ takes
t ′ to transfer.

(Continued on the next slide . . . .)

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 13 / 67
 Distributed Computing: Principles, Algorithms, and
Systems

The Network Time Protocol synchronization

protocol.

The offset between A’s clock and B’s clock is O. If A’s local
clock time is
A(t ) and B’s local clock time is B(t ), we have
A(t ) = B(t ) + (3
O )
The
n, Ti − 2 = Ti − 3 + t + O (4
Ti = Ti − 1 − O + t
′ )
(5
Assuming t = t , the offset Oi can be
′
)
estimated as:
(6
Oi = (T i − 2 − T i − 3 + T i )
− 1 − T i )/2
Di = (T i − T i − 3) − (T i − 1 − T i (7
The round-trip delay )is estimated as: )
−2

The eight of
The value most recent
Oi that pairs of (O
corresponds toi ,minimum
Di ) are D i s chosen to
i
retained. Logical
A. Kshemkalyani and M. Singhal (Distributed CUP 14 / 67
 Distributed Computing: Principles, Algorithms, and
Systems

Introductio
n

The concept of causality between events is fundamental to the

design and analysis of parallel and distributed computing and
operating systems.
Usually causality is tracked using physical time.
In distributed systems, it is not possible to have a global
physical time.
As asynchronous distributed computations make progress in
spurts, the logical time is sufficient to capture the fundamental
monotonicity property associated with causality in distributed
systems.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 15 /

 Distributed Computing: Principles, Algorithms, and
Systems

Introductio
n
Logical Time:

In distributed systems, it is not possible to have global physical time. It is possible

to realize only an approximation of physical time which is called logical time. The
logical time which advances in jumps is sufficient to capture the fundamental
monotonicity property associated with causality in distributed systems. The three
ways to implement logical time are scalar time, vector time, and matrix time.

Causality among events in a distributed system is a powerful concept in reasoning,

analyzing, and drawing inferences about a computation. The knowledge of the causal
precedence relation among the events of processes helps solve a variety of problems in
distributed systems, such as distributed algorithms design, tracking of dependent
events, knowledge about the progress of a computation, and concurrency measures.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 16 /

 Distributed Computing: Principles, Algorithms, and
Systems

A Framework for a System of Logical

Clocks
Definition
A system of logical clocks consists of a time domain T and a
logical clock C . Elements of T form a partially ordered set over a
relation <.
Relation < is called the happened before or causal precedence.
Intuitively, this relation is analogous to the earlier than relation
provided by the physical time.
The logical clock C is a function that maps an event e in a
distributed system to an element in the time domain T , denoted
as C(e) and called the timestamp of e, and is defined as follows:
C : H ›→ T
such that the following property is satisfied:
for two events ei and ej , ei → ej =⇒ C(ei ) < C(ej ).

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 17 /

 Distributed Computing: Principles, Algorithms, and
Systems

A Framework for a System of Logical

Clocks
This monotonicity property is called the clock consistency
condition. When T and C satisfy the following condition,
for two events ei and ej , ei → ej ⇔ C(ei ) <
C(ej ) the system of clocks is said to be strongly
consistent.
Implementing Logical Clocks
Implementation of logical clocks requires addressing
two issues: data structures local to every process to represent
logical time and a protocol to update the data structures to
ensure the consistency condition.
Each process pi maintains data structures that
allow it the following two capabilities:
◮ A local logical clock, denoted by lci , that helps process pi measure its own
progress.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 18 /

 Distributed Computing: Principles, Algorithms, and
Systems

Implementing Logical
Clocks

◮ A logical global clock, denoted by gci , that is a representation of process pi ’s

local view of the logical global time. Typically, lci is a part of gci .

The protocol ensures that a process’s logical clock, and thus its view of
the global time, is managed consistently. The protocol consists of the
following two rules:
R1: This rule governs how the local logical clock is updated by a
process when it executes an event.
R2: This rule governs how a process updates its global logical
clock to update its view of the global time and global
progress.
Systems of logical clocks differ in their representation of logical time
and also in the protocol to update the logical clocks.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 19 /

 Distributed Computing: Principles, Algorithms, and
Systems

Scalar
Time

Proposed by Lamport in 1978 as an attempt to totally order

events in a distributed system.
Time domain is the set of non-negative integers.
The logical local clock of a process pi and its local view of the
global time are squashed into one integer variable Ci .
Rules R1 and R2 to update the clocks are as follows:
R1: Before executing an event (send, receive, or internal), process pi
executes the following:
Ci := Ci + d (d > 0)
In general, every time R1 is executed, d can have a different value;
however, typically d is kept at 1.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 20 /

 Distributed Computing: Principles, Algorithms, and
Systems

Scalar
Time

R2: Each message piggybacks the clock value of its sender at

sending time. When a process pi receives a message with
timestamp Cmsg , it executes the following actions:
◮ Ci := max (Ci , Cmsg )
◮ Execute R1.
◮ Deliver the message.
Figure 3.1 shows evolution of scalar time.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 21 /

 Distributed Computing: Principles, Algorithms, and
Systems

Scalar Time
Evolution of scalar
time:

1 2 3 8 9
p
1
9
2
1 4 5 7 11
p
2 3 10
4
1 b
p
3
5 6 7

Figure 3.1: The space-time diagram of a distributed

execution.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 22 /

 Distributed Computing: Principles, Algorithms, and
Systems

Basic
Properties

Consistency Property
Scalar clocks satisfy the monotonicity and hence the
consistency property: for two events ei and ej , ei → ej
=⇒ C(ei ) < C(ej ).
Total Ordering
Scalar clocks can be used to totally order events in a distributed
system.
The main problem in totally ordering events is that two or more
events at different processes may have identical timestamp.
For example in Figure 3.1, the third event of process P1 and the second
event of process P2 have identical scalar timestamp.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 23 / 67

 Distributed Computing: Principles, Algorithms, and
Systems

Total
Ordering
A tie-breaking mechanism is needed to order such events. A tie is
broken as follows:
Process identifiers are linearly ordered and tie among events
with identical scalar timestamp is broken on the basis of their
process identifiers.
The lower the process identifier in the ranking, the higher the
priority.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 24 / 67

 Distributed Computing: Principles, Algorithms, and
Systems

Properties
...

Event counting
If the increment value d is always 1, the scalar time has the
following interesting property: if event e has a timestamp h,
then h-1 represents the minimum logical duration, counted in
units of events, required before producing the event e;
We call it the height of the event e.
In other words, h-1 events have been produced sequentially
before the event e
regardless of the processes that produced these events.
For example, in Figure 3.1, five events precede event b on the
longest causal path ending at b.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 25 / 67

 Distributed Computing: Principles, Algorithms, and
Systems

Properties
...
No Strong Consistency
The system of scalar clocks is not strongly consistent; that is, for
two events
ei and ej , C(ei ) < C(ej ) =/⇒ ei → ej .
For example, in Figure 3.1, the third event of process P1 has
smaller scalar timestamp than the third event of process P2.
However, the former did not happen before the latter.
The reason that scalar clocks are not strongly consistent is that
the logical local clock and logical global clock of a process are
squashed into one, resulting in the loss causal dependency
information among events at different processes.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 26 / 67

 Distributed Computing: Principles, Algorithms, and
Systems

Vector Time

The system of vector clocks was developed independently by

Fidge, Mattern and Schmuck.
In the system of vector clocks, the time domain is represented by
a set of
n-dimensional non-negative integer vectors.
Each process pi maintains a vector vti [1..n], where vti [i ] is the
local logical clock of pi and describes the logical time progress at
process pi .
vti [j] represents process pi ’s latest knowledge of process pj local
time.
If vti [j]=x , then process pi knows that local time at
process pj has progressed till x .
The entire vector vti constitutes pi ’s view of the global logical
time and is used to timestamp events.
A. Kshemkalyani and M. Singhal (Distributed Logical CUP 27 / 67
 Distributed Computing: Principles, Algorithms, and
Systems

Vector
Time
Process pi uses the following two rules R1 and R2 to update its clock:
R1: Before executing an event, process pi updates its local
logical time as follows:
vt i [i ] := vt i [i ] + d

(d > 0)
R2: Each message m is piggybacked with the vector clock vt of the
sender process at sending time. On the receipt of such a message
(m,vt), process pi executes the following sequence of actions:
◮ Update its global logical time as follows:

1 ≤ k ≤ n : vt i [k ] := max (vt i [k ], vt [k ])

◮ Execute R1.
◮ Deliver the message m.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 28 / 67

 Distributed Computing: Principles, Algorithms, and
Systems

Vector Time

The timestamp of an event is the value of the vector clock of

its process when the event is executed.
Figure 3.2 shows an example of vector clocks progress with the
increment value d=1.
Initially, a vector clock is [0, 0, 0, ...., 0].

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 29 / 67

 Distributed Computing: Principles, Algorithms, and
Systems

Vector Time
An Example of Vector
Clocks

1 2 3 4 5
0 0 0 3 3
0 0 0 4 4
p
1 5
2 2 3
0 3 4
0 0 2 2 2 4 5
1 2 3 4 6
0 0 0 0 4
p
2 5
2
3 5
0 4
0 2 2 2
0 3 3 3
1 2 3 4
p
3

Figure 3.2: Evolution of vector

time.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 30 / 67

 Distributed Computing: Principles, Algorithms, and
Systems

Vector Time
Comparing Vector
Timestamps
The following relations are defined to compare two vector
timestamps, vh
and vk :
vh = vk ⇔ ∀x : vh[x ] = vk [x ]
vh ≤ vk ⇔ ∀x : vh[x ] ≤ vk [x ]
vh < vk ⇔ vh ≤ vk and ∃x : vh[x ] <
vk [x ]
vh ǁ vk ⇔ ¬(vh < vk ) ∧ ¬(vk < vh)
If the process at which an event occurred is known, the test to
compare two timestamps can be simplified as follows: If events x
and y respectively occurred at processes pi and pj and are assigned
timestamps vh and vk, respectively, then

x→y ⇔ vh[i ] ≤ vk [i ]
xǁy ⇔ vh[i ] > vk [i ] ∧ vh[j ]
< vk [j ]
A. Kshemkalyani and M. Singhal (Distributed Logical CUP 31 / 67
 Distributed Computing: Principles, Algorithms, and
Systems

Vector Time
Properties of Vector
Time

Isomorphism
If events in a distributed system are timestamped using a
system of vector clocks, we have the following property.
If two events x and y have timestamps vh and vk, respectively,
then

x→y ⇔
vh < vk x ǁ y ⇔
vh ǁ vk .

Thus, there is an isomorphism between the set of partially

ordered events produced by a distributed computation and
their vector timestamps.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 32 / 67

 Distributed Computing: Principles, Algorithms, and
Systems

Vector
Time
Strong Consistency
The system of vector clocks is strongly consistent; thus, by
examining the vector timestamp of two events, we can
determine if the events are causally related.
However, Charron-Bost showed that the dimension of vector clocks
cannot be less than n, the total number of processes in the
distributed computation, for this property to hold.
Event Counting
If d=1 (in rule R1), then the i th component of vector clock at
process pi , vti [i ], denotes the number of events that have
occurred at pi until that instant.
So, if an event e has timestamp vh, vh[j] denotes the number
of events executed by process pj that causally precede e.
Clearly, Σ vh[j ] − 1 represents the total number of events
that causally precede e in the distributed computation.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 33 / 67

 Distributed Computing: Principles, Algorithms, and
Systems

Time

If a conflict is discovered, the offending processes are rolled back to

the time just before the conflict and executed forward along the
revised path.
Detection of conflicts and rollbacks are transparent to users.
The implementation of Virtual Time using Time Warp mechanism
makes the following optimistic assumption: synchronization
conflicts and thus rollbacks generally occurs rarely.
next, we discuss in detail Virtual Time and how Time Warp
mechanism is used to implement it.

A. Kshemkalyani and M. Singhal (Distributed Logical CUP 34 / 67