Distributed Systems

International Institute of Information Technology

Hyderabad, India

Pictures not cited explicitly are taken

from the course book or wikepedia.
 Data Structures Week 1

Vector Time

Time is represented by n-dimensional non-negative integer
vectors.
 Each process pi maintains a vector vti [1..n], where vti[i] is the
local logical clock of pi and describes the logical time progress
at process pi.
 Vti[j] represents process pi ’s latest knowledge of process pj
local time.
 If vti [j]=x, then process pi knows that local time at process pj
has progressed till x.
 The entire vector vti constitutes pi ’s view of the global logical
time and is used to timestamp events.

The timestamp of an event is the value of the vector clock of
its process when the event is executed. 2
 Data Structures Week 1

Vector Time – Update Rules

Rules for Process pi to update its clock:
 Rule 1: Before executing an event, process pi
updates its local logical time as follows:
vti [i] := vti[i] + d where (d > 0)

Rule2: Each message m is piggybacked with the
vector clock vt of the sender process at sending
time. On the receipt of such a message (m,vt),
process pi executes the following sequence of
actions:
1. Update its global logical time as follows:
2. 1 ≤ k ≤ n : vti[k] := max(vti[k], vt[k])
3
3. Execute Rule1.
 Data Structures Week 1

Vector Time – Example

P2’s second event:
max([2,0,0], [0,1,0])=[2,1,0]
Then Rule1: [2,2,0]

4
 Data Structures Week 1

• For process i, for d=1, are the events internal to the

process ordered as a sequence1,2,3,4,....

That is for process i: will Vi[i] always be increasing

as 1,2,3,4,5,…and so on or can there be values
skipped?
 Data Structures Week 1

Vector Time – Example


Initially, a vector clock is [0, 0, 0, ...., 0]. d = 1.

Fill the timestamps for the other events.

6
 Data Structures Week 1

Vector Time – Example


Using vector clocks, two timestamps vh and vk
are compared as follows.

vh == vk iff for all indices i, vh[i] == vk[i]

vh <= vk iff for all indices i, vh[i]  vk[i]

vh < vk iff vh  vk and there exists an index i
where vh[i] < vk[i].

vh || vk iff not(vh < vk) and not(vk < vh)

7
 Data Structures Week 1

Properties of Vector Time


Event Counting: Use d = 1 always.

Then the i th component of vector clock at
process pi,vti[i], denotes the number of events that
have occurred at pi until that instant.

So, if an event e has timestamp vh, vh[j] denotes
the number of events executed by process pj that
causally precede e.
How many events causally precede e in the
distributed computation ?

8
 Data Structures Week 1

Properties of Vector Time


Event Counting: Use d = 1 always.

Then the i th component of vector clock at
process pi,vti[i], denotes the number of events that
have occurred at pi until that instant.

So, if an event e has timestamp vh, vh[j] denotes
the number of events executed by process pj that
causally precede e. Clearly, (j vh[j] ) − 1
represents the total number of events that
causally precede e in the distributed computation.

9
 Data Structures Week 1

Properties of Vector Time

Are Vector clocks strongly consistent ?

10
 Data Structures Week 1

Properties of Vector Time


Strong Consistency: Vector clocks are strongly
consistent.


What are the limitations of the vector clock?

11
 Data Structures Week 1

Limitations of Vector Time


Large message sizes owing to the vector being
piggybacked on each message.

The message overhead grows linearly with the
number of processors in the system and when
there are thousands of processors in the system,
the message size becomes huge even if there are
only a few events occurring in few processors.

Few techniques exist to reduce the overhead.


What can we do to reduce the load?

12
 Data Structures Week 1

Vector Time – Example


Initially, a vector clock is [0, 0, 0, ...., 0]. d = 1.

Fill the timestamps for the other events.

13
 Data Structures Week 1

Optimizations
 Consider a message from Pi to Pj.
 The message includes the vector time at Pi.

Of the n components of the vector time, only few
would have changed since the last time Pi sent a
message to Pj.

Need to include only those tuples.
 (i1, t1), (i2, t2), ...

Update rule changes slightly so that:
 Pj updates the component k of its vector time to
max{vtj[k], tk} where tk is the time reported by Pi with
Are we missing
14 something?
the tuple (k, tk)
 Data Structures Week 1

Optimizations

Message sizes decrease in general, but local
storage may increase.
 Need to know the vector time at Pi when the last
message was sent to Pj, for each j.

That is, each process stores the last msg sent to

every other process.

O(n2) storage per process.

A clever optimization can bring this storage to
O(n) – Singhal kshemkalyanis technique

15
 Data Structures Week 1
Singhal- Kshemkalyani Differential
Technique
• Process pi maintains the following two additional vectors:

◮LSi[1..n] (‘Last Sent’) : Lsi[j] indicates the value of vti[i] when

process pi last sent a message to process pj.

◮LUi[1..n] (‘Last Update’):LUi[j] indicates the value of vti[i] when

process pi last updated the entry vti[j].

LUi[i] =vti[i] ?

When do you update LS and LU ?

Can we put LS and LU together to know what is the information

that needs to be piggybacked on the message ?
 Data Structures Week 1
Singhal- Kshemkalyani Differential
Technique
• Process pi maintains the following two additional vectors:

◮LSi[1..n] (‘Last Sent’) : Lsi[j] indicates the value of vti[i] when

process pi last sent a message to process pj.

◮LUi[1..n] (‘Last Update’):LUi[j] indicates the value of vti[i] when

process pi last updated the entry vti[j].

Clearly,LUi[i] =vti[i] at all times

• Lui[j] needs to be updated only when the receipt of a message
causes pi to update entry vti[j].
• LSi[j] needs to be updated only when pi sends a message to pj
Data Structures Week 1
 Data Structures Week 1
Singhal- Kshemkalyani Differential
Technique
• Since the last communication from pi to pj , only those elements
of vector clock vti[k] have changed for which LSi[j] < Lui[k] holds.
Hence, only these elements need to be sent in a message from
pi to pj.
• When pi sends a message to pj , it sends only a set of tuples
{ (x,vti[x]) | LSi[j] < LUi[x] } as the vector timestamp to pj, instead
of sending a vector of n entries in a message.

Home Work !!
Think about whether vector clocks can detect out of order msgs.

Will Singhal-Kshemkalyanis technique work if the messages

are not in FIFO order ?
 Data Structures Week 1

Homework
• Suppose every process can only broadcast messages. Now, I want
to design an algorithm that ensures causal ordering. That is, when
two messages arrive at say Pi; they should be delivered in the order
they have been sent.

Consider these broadcast msgs that reach out of order where msg b
reaches before a.
First P1a
• Next P1 c
• Then, c P3

Design an algorithm, that will deliver msgs only in order.

Hint: use vector clocks. Assume that on receiving a msg, we don’t
increment event number of local process. Event number is
incremented only when broadcast made. Hence event number is
indicative of number of broadcasts sent out.
 Data Structures Week 1

Teasers-1
• Suppose an updation event at site D at time t was
propagated(broadcasted) to all sites

• Nodes need this information to determine when it is safe to

discard/stabilize updates.

• Can I decipher looking at B’s vector sent to me if B has received

the msg from D?
 Data Structures Week 1

Teasers-1
• Can I decipher looking at B’s vector sent to me if B has received
the msg from D?

No. Suppose B received a msg from C before it sent a msg to

me then

B updates as
VB[D]= max(VB[D], VC[D]).
Hence we do not know if VB[D] is update from C or an update B
already had from D.
 Data Structures Week 1

Causal Message Ordering

• Causual message ordering: the message which is sent first should
be received first. If send (M1)–>send(M2) then for all processes which
receive the messages M1 and M2 should receive M1 before M2
OR
If two msgs causually ordered btwn them are sent (even if by two diff
processes) to a process P; then process P should receve the msg also
by the same causal order.

Consider:
• A sends msg to B
• B sends to C
• A sends to C
At C: C receives two msgs. One from A and other from B. However, the
msg sent by B to C is causually ordered after the msg send by A to C.
Hence, msg sent by B to C should be received after msg sent by A to C
 Data Structures Week 1

Teasers-2
• Suppose I want to know whether an updation event at site D at time t
which was propagated to all sites has reached everyone assuming
causal order and no lost messages:

Consider the below:

B sends msg to me. Can I know if B received the msg from D?
 Data Structures Week 1

Teasers-2
• Let us assume that B got D’s news through C.
Now, that means there were two msgs with recipient as B. One Is the msg
from D and another is a msg from C due to which VB[D]=VC[D].

Now by causual order if two msgs are intended for the same destination;
then the one sent first should reach first.
Can the msg from C to B with D’s update be sent before D’s msg to B?
No as D had broadcast all its msgs. So C can be sending the update on D to
B only after D sent a msg to B.

Hence D’s msg to B will reach first and the update could not have come
through anyone else
 Data Structures Week 1

Teasers-3

Once again consider causal order:

I want to know if D’s msg has reached everyone.
Instead of receiving a msg from everyone if I receive a msg from B such that
VB[x] >= t for all x then can I conclude that all have received the broadcast from
D?