• An important issue in distributed system is how processes cooperate and

synchronize with one another

• Cooperation is partially supported by naming, which allows to share
resources
• Synchronization requirements
o Access to shared resources
o Agreement on the ordering of events

Clock Synchronization

• Problem: When each machine has its own clock, an event that occurred
after another event may be assigned an earlier time

Physical Clock
Quartz Clocks

• Computers have a circuit to keep track of time using a vibrating quartz

crystal
• Quartz clocks are found in most wristwatches, in computers, mobile
phones, microwave ovens, and many other everyday objects
• Piezoelectric effect: mechanical force ⇔ electric field
• Battery sends electricity to quartz crystal and it oscillates 32,768
times (1 second)
• However, quartz crystals at different computers often run at slightly
different speeds
o Manufacturing imperfection
o Frequency varies with the temperature
• The rate by which a clock runs fast or slow is called drift
o Drift measured in parts per million (ppm)
o 1 ppm = 1 microsecond/second = 86 ms/day = 32 s/year
o Most computer clocks correct within ≈ 50 ppm
o Clock skew between different machines

Atomic Clocks
• When greater accuracy is required, atomic clocks are used
• These clocks are based on quantum mechanical properties of certain
atoms, such as Caesium or Rubidium
• Atomic clocks combine a quartz crystal oscillator with an ensemble of
atoms to achieve greater stability
• The time unit of one second in the International System of Units (SI)
is defined using caesium-133 atom (International Atomic Time (TAI): 1
day is 24 × 60 × 60 × 9,192,631,770 periods of cesium-133’s resonant
frequency)
• Atomic clocks deviate only 1 second in up to 100 million years
• Atomic clocks are too expensive and bulky to build into every computer
and phone, so quartz clocks are used
Universal Time (UT1):
• Universal Time (UT) is a time standard that reflects the average speed
of the Earth's rotation
• It is not measured by clocks but by looking at the stars

Issues:
• The speed of rotation of the planet is not constant

We have two different definitions of time

• One based on quantum mechanics (atomic clock), the other based on
astronomy (universal time) and
• Those two definitions don’t match up precisely
• The solution is Coordinated Universal Time (UTC), which is based on
atomic time, but includes corrections to account for variations in the
Earth’s rotation
• The difference between UTC and TAI is that UTC includes leap seconds,
which are added as needed to keep UTC roughly in sync with the rotation
of the Earth
• Due to leap seconds, it is not true that an hour always as 3600
seconds, and a day always has 86,400 seconds
• Earth has been slowing down a bit over the past decades, so UTC is
currently running 37 seconds behind TAI

Time zone and Daylight Savings Time

• In everyday life we use our local time zone, which is specified as an
offset to UTC

In computing:
• A timestamp is a representation of a particular point in time
• Two representations of timestamps are commonly used: Unix time and ISO
8601

Unix time
• For Unix time, zero corresponds to the arbitrarily chosen date of 1
January 1970, known as the epoch
• Ignore leap seconds

Disadvantage
• This complicates software that needs to work with dates and times
• To be correct, software that works with timestamps needs to know about
leap seconds
• For example, if you want to calculate how many seconds elapsed between
two timestamps, you need to know how many leap seconds were inserted
between those two dates
• Besides, OS and distributed systems often need timings with sub-second
accuracy

Pragmatic solution:
• “smear” (spread out) the leap second over the course of a day
ISO 8601:
year, month, day, hour, minute, second, and time zone offset relative to UTC
example: 2020-11-09T09:50:17+00:00

Network Time Protocol (NTP)

• Since these clocks drift, they need adjustment from time to time, which
is most commonly done using the Network Time Protocol (NTP)
• All mainstream operating systems have NTP clients built in
• Many operating system vendors run NTP servers, configure OS to use them
by default

Estimating time over a network

• We can estimate that the server’s clock will have moved on to t3 plus
the one-way network latency
• We then subtract the client’s current time t4 from the estimated server
time to obtain the estimated skew between the two clocks

Correcting clock skew

• Once the client has estimated the clock skew θ, it needs to apply that
correction to its clock
• If |θ| < 125 ms, slew the clock:
slightly speed it up or slow it down by up to 500 ppm (brings clocks in
sync within ≈ 5 minutes)
• If 125 ms ≤ |θ| < 1,000 s, step the clock:
suddenly reset client clock to estimated server timestamp
• If |θ| ≥ 1,000 s, panic and do nothing
(leave the problem for a human operator to resolve)
Systems that rely on clock sync need to monitor clock skew
Hierarchy of clock servers arranged into strata:
o Stratum 0: atomic clock or GPS receiver
o Stratum 1: synced directly with stratum 0 device
o Stratum 2: servers that sync with stratum 1, etc.
• May contact multiple servers, discard outliers, average rest
• Makes multiple requests to the same server, use statistics to reduce
random error due to variations in network latency
• Reduces clock skew to a few milliseconds in good network conditions but
can be much worse!

Time-of-day clock:
• Time since a fixed date (e.g., 1 January 1970 epoch)
• May suddenly move forwards or backwards (NTP stepping), subject to leap
second adjustments
• Timestamps can be compared across nodes (if synced)
• Java: System.currentTimeMillis()
• Linux: clock_gettime(CLOCK_REALTIME)

Monotonic clock:
• Time since arbitrary point (e.g., when machine booted up)
• Always moves forwards at near-constant rate
• Good for measuring elapsed time on a single node
• Java: System.nanoTime()
• Linux: clock_gettime(CLOCK_MONOTONIC)

The Berkeley Algorithm

• Steps
o The time daemon tells all machine its time
o Other machines answers how far ahead or behind
o The time daemon computes the average and tell other how to adjust
Causality and happens-before
Ordering of messages

• m1 = “A says: Hello everyone!”

• m2 = “B says: Hello A”
• C sees m2 first, m1 second, even though logically m1 happened before m2
• A monotonic clock won’t work since its timestamps are not comparable
across nodes
• Physical timestamps inconsistent with causality

Let’s assume we assign timestamps with each message

Problem:

• Even with synced clocks, if B receives message at t2 from A at t1, t2 <

t1 is possible; even after synced with NTP
• Because clock skew can be larger than the latency
• Timestamp order is inconsistent with expected order

The happens-before relation

• An event is something happening at one node (sending or receiving a

message, or a local execution step)
• We say event a happens before event b (written a → b) iff:

1. a and b occurred at the same node, and a occurred before b in

that node’s local execution order; or
2. Event a is the sending of some message m, and event b is the
receipt of that same message m (assuming sent messages are
unique); or
3. There exists an event c where a→c and c→b (transitivity)

The happens-before relation is a partial order

• The ordering is only partial because events can be concurrent in which

case it is not known which event happened first
• It is possible that neither a→b nor b→a. In that case, a and b are
concurrent (written a ∥ b)
Happens-before relation example

• a→b, c→d, and e→f due to node execution order

• b→c and d→f due to messages m1 and m2
• a→c, a→d, a→f, b→d, b→f, and c→f due to transitivity
• a∥e, b∥e, c∥e, and d∥e (concurrent)

Causality

• Taken from physics (relativity)

• When a → b, then a might have caused b
• When a ∥ b, we know that a cannot have caused b
• Happens-before relation encodes potential causality

Logical vs. physical clocks

• Physical clock: count number of seconds elapsed

• Logical clock: count number of events occurred
• Physical timestamps: useful for many things but may be inconsistent
with causality
• Logical clocks: designed to capture causal dependencies

(e1 → e2) =⇒ (T(e1) < T(e2)) //T stands for timestamp

• We will look at two types of logical clocks:

o Lamport clocks
o Vector clocks

Lamport clock in words

• Each node maintains a counter t, incremented on every local event e

• Let L(e) be the value of t after that increment
• Attach current t to messages sent over network
• Recipient moves its clock forward to timestamp in the message (if
greater than local counter), then increments
Lamport clock example

• Used to create partial causal ordering of events

• Properties of this scheme:

o If a→b, then L(a)<L(b)known as clock consistency condition

o If event A comes before event B, then event A’s logical clock
comes before event B’s
o However, L(a)<L(b) does not imply a→b, because may be a||b
o Possible that L(a)=L(b) for a≠b
• Therefore, if event A’s logical clock comes before event B’s logical
clock, then A may have happened before or at the same time as B, but
did not happen after B
• Observation: the third event on node A and the first event on node B
both have a timestamp of 3
• If we need a unique timestamp for every event, each timestamp can be
extended with the name or identifier of the node on which that event
occurred

Vector clocks

• Given two arbitrary Lamport timestamps L(a) and L(b) with L(a) < L(b)
we can’t tell whether a → b or a||b
• Vector clock can do that

Strong consistency condition:

• If one event’s clock comes before another’s, then that event comes
before the other, i.e., it is a two-way condition
• Assume n nodes in the system, N = ⟨N1, N2, ..., Nn⟩
• Vector timestamp of event a is V(a)=⟨t1, t2, ..., tn⟩
• ti is number of events observed by node Ni
• Each node has a current vector timestamp T
• On event at node Ni, increment vector element T[i]
• Attach current vector timestamp to each message
• Recipient merges message vector into its local vector
Vector clocks example

• Assuming the vector of nodes is N = ⟨A, B, C⟩:

• The vector timestamp of an event e represents a set of events, e and

its causal dependencies: {e} ∪ {a | a → e}
• For example, ⟨2, 2, 0⟩ represents the first two events from A, the first
two events from B, and no events from C

Vector clocks ordering

• Define the following order on vector timestamps (in a system with n

nodes):
o T = T′ iff T[i]=T′[i] for all i∈ {1, ..., n}
o T ≤ T′ iff T[i]≤T′[i] for all i∈ {1, ..., n}
o T < T′ iff T ≤T′ and T ≠ T′
o T ∥ T′ iff T ≰ T′ and T′≰ T

Properties of this order:

• (V(a)<V(b)) ⇐⇒(a→b)
• (V(a)=V(b)) ⇐⇒(a=b)
• (V(a)∥ V(b)) ⇐⇒(a∥b)

• We say that one vector is less than or equal to another vector if every
element of the first vector is less than or equal to the corresponding
element of the second vector
• One vector is strictly less than another vector if they are less than
or equal, and if they differ in at least one element
• However, two vectors are incomparable if one vector has a greater value
in one element, and the other has a greater value in a different
element

o For example, T = <2, 2, 0> and T’ = <0, 0, 1> are incomparable

because T[1] > T’[1] but T[3] < T’[3]