Distributed Systems

International Institute of Information Technology

Hyderabad, India

Pictures not cited explicitly are taken

from the course book or wikepedia.
Course Policies
Textbook(s)
– Distributed Computing: Principles, Algorithms, and
Systems: Kshemkalyani and Singhal.
– Distributed Algorithms: Gerald Tel, Nancy Lynch

Grading policy – Almost Final!

• Slip Tests(~7): 15
• Project:20
• Assignments(3) : 20: 2 coding and one theory
• 1 Quiz + Midsem + Endsem: (10+12)+23=45
Syllabus (Tentative)
 Module 1
 Introduction
 Time and Synchronization
 Mutual Exclusion
 Deadlock Detection

 Module 2
 Distributed graph algorithms
 Consensus, Agreement, Locking
 Deadlock Handling
 Distributed File systems
Syllabus (Tentative)
 Module 4
 Limitations of distributed computing
 Self-Stabilization
 CAP Theorem
 Block Chain and BitCoins (Guest lecture)
TAs
Ramaguru Guru
Kannav Mehta
Akshit Garg
Snehal Kumar
Sahiti Reddy
Prince Varshney
Nikunj Nawal
Lokesh V
Motivation
A group of computers working together as to appear as a
single computer to the end-user.– Tannenbaum
Eg: Airline reservation, Online Banking

Required Features:
Lot of computers
Perform Concurrently
Fail independently
Dont share a global clock

Why Distributed Systems?

Best hardware may be insufficient to scale vertically
(upgrade hardware of a system) hence the need to scale
horizontally. Other benefits: Fault tolerance and low
latency.
 Data Structures Week 1

Motivation
• Horizontal Scaling to fit data, shift data to manage load – scalability
There is a limitation to how much you can upgrade a machine. Every machine has its threshold for
RAM, storage, and processing power.

• Inherently distributed. Ex: Banking system across cities

• Data Sharing. Ex Libraries, certain files, Complete datasets

• Enhanced Reliability

• Increased Performance/cost ratio (parallelistion)

• Fault Tolerance
Data Structures Week 1
 Data Structures Week 1

Challenges with a Distributed System

• Unreliable communication – messages may drop, could be intercepted, could be
delayed, could be out of order.

• Lack of global knowledge – a participating node has information of only his local
memory and local computations. (no shared memory)

• Lack of synchronisation (different local clocks)-how to implement sequence in

which we want things to happen; how to order steps across participating nodes
where each node has a different clock

• Concurrency control – Data distributed across nodes. Moreover each has its own
clock. How do you implement Mutual exclusion now or critical sections now?

• Failure and recovery : Nodes may fail, communication channels may fail – how do
you still make sure the system functions. When a node recovers how do you make
the updates on it.

• Deadlocks, Termination Detection, File systems are other concerns

 Data Structures Week 1

Can we solve these -

The village without a mirror – consensus

War Preparation with messengers

Consensu
Consensus: All non-faulty processes must agree on a common value.
If initial value of all non-faulty processes is v, then the agreed upon
value must be v

Consensus with traitors included - the general too may be

malicious (Byzantine Agreement Problem)
 Data Structures Week 1

Time in Distributed Systems

 Data Structures Week 1

Time in Distributed Systems

• Recall the quartz oscillation based clock ticks used

in digital systems.
• These clocks drift.
• With such a drift, it is not possible to arrange
events according to a causal order.
 Data Structures Week 1

Why is time important

• to determining the order in which events occur
• helps maintain consistency in replicated databases,
• helps design correct deadlock detection algorithms to
avoid phantom and undetected deadlocks.
• Tracking of dependent events
• In debugging, it helps construct a consistent state
• for resuming reexecution; in failure recovery, it helps
build a checkpoint;
• in replicated databases,
• it aids in the detection of file inconsistencies in case of
a network partitioning
Motivation
– To ensure a correct and consistent view of a distributed
database.
– Sensor networks and sensing properties of the physical
world.
– Applications using time-outs to deduce certain
actions/events.
Physical Time and Synchronization
• In centralized systems, there is only single clock. A
process gets the time by simply issuing a system
call to the kernel.
– The time returned by the kernel is the “time”.
• In distributed systems, there is no global clock or
common memory. Each processor has its own
internal clock and its own notion of time.
• These clocks can easily drift seconds per day,
accumulating significant errors over time.
• This clearly poses serious problems to applications
that depend on a synchronized notion of time.
Cristian’s (Time Server) Algorithm
• Uses a time server to synchronize clocks
– Time server keeps the reference time (say UTC)
– A client asks the time server for time, the server responds with its
current time, and the client uses the received value T to set its clock
• But network round-trip time introduces errors…
– Let RTT = response-received-time – request-sent-time (measurable at
client),
– If we know (a) min = minimum client-server one-way transmission time and
(b) that the server timestamped the message at the last possible instant
before sending it back
– Then, the actual time could be between [T+min,T+RTT— min]
where RTT-min is the max time it took for the response to return which
happens when request took min time to go
Berkeley UNIX algorithm
• One master machine and set of slave nodes
• Periodically, this daemon polls and asks all the
machines for their time
• The machines respond.
• The daemon computes an average time and then
broadcasts this average time.

The algorithm assumes that each machine node in

the network either doesn’t have an accurate time
source or doesn’t possess an UTC server.
Decentralized Averaging Algorithm
• Each machine has a daemon without UTC
• Periodically, at fixed agreed-upon times, each
machine broadcasts its local time.
• Each of them calculates the average time by
averaging all the received local times.
Network Time Protocol (NTP)
• Most widely used physical clock synchronization protocol
on the Internet
• 10-20 million NTP servers and clients in the Internet
• Claimed Accuracy (Varies)
– milliseconds on WANs, submilliseconds on LANs
• Hierarchical tree of time servers.
– The primary server at the root synchronizes with the UTC.
– Secondary servers - backup to primary server.
– Lowest
• synchronization subnet with clients.
 Data Structures Week 1

• However even with NTP, physical time cannot be

perfectly synchronised. Moreover different nodes
might have different NTP servers they are talking to
as well.

But, do we really need to know the physical time or

do we only need to know the sequence of events
and that too among dependent events?
Modeling Distributed Systems

Can think of a distributed system as a collection
of processors, and

A communication network.

The processors

May fail

Share no global memory

We also assume that the network

Has delays that are finite but unpredictable.

May not respect message ordering

Can lose messages, duplicate messages

May have links that become unavailable/may fail.
A Distributed Program
distributed program is
 A set of n processes, P1, P2, …, Pn, typically each
running on a different processor.
 A set of channels Cij that connects Processor i to
Processor j.
 The state of a process Pi includes
 The local memory of Pi

Also depends on messages sent, and received
The state of a channel Cij includes

The messages in transit on this channel.
Types of events

We will assume that processor sending a message
does not wait for its delivery.


Any distributed program has three types of events

Local actions

Message Send

Message Receive


Can then view a distributed program as a
sequential execution of the above events.
 Data Structures Week 1

Logical Time
• Logical time is a notion we introduce to order
events. It provides a mechanism to define the
causal order in which events occur at different
processes. The ordering is based on the following:
• Two events occurring at the same process happen
in the order in which they are observed by the
process.
• If a message is sent from one process to another,
the sending of the message happened before the
receiving of the message.
• If e occurred before e' and e' occurred before e"
then e occurred before e".
Causality between events
 Think of events happening at a process Pi.

One can order these events sequentially,

In other words, place a linear order on these events.
 Let hi = {e1i, e2i, …}, be the events at process Pi.
 A linear order Hi is a binary relation →i on the events
hi such that eki →i eji if and only if the event eki occurs
before the event eji at Process Pi.
 The dependencies captured by →i are often called
as causal dependencies among the events hi at
P i.
Causality for msgs

What about events generated due to messages?
 Consider a binary relation →msg across messages
exchanged.
 Clearly, for any message m, send(m) →msg
recv(m).
 These two relations →i and →msg allow us to view
the execution of a distributed program
Causality Definition

One can now view the execution of a distributed
program as a collection of events.
 Consider the set of events H = Ui hi, where hi is
the events that occurred at process Pi.

Define a binary relation → expressing causality
among pairs of events that possibly occur at
different processes. → defined as:

i = j and x < y, or

exi → eyj iff { exi →msg eyj, or

There exists ezk in H s.t. exi → ezk and
ez k → e y j
Logical vs. Physical Concurrency

Two events are logically concurrent if and only if
the events do not causally affect each other. In
other words, ei || ej ↔ Not(ei → ej) and Not(ej →
ei).

Note that for logical concurrency of two events,
the events may not occur at the same time.

On the other hand, we define that two events are
physically concurrent iff the events occur at the
same physical time.
A Distributed Program in Execution
• Examples – Find concurrent events, Find some
non-trivial (across Pis) precedence among events.
A Distributed Program in Execution

• .

• Concurrent: e1
3
and e3
1
? e42 and e13 ?

• Causal: e 3
3
→ e5
1
, e 2
1
→ e3
2
, e 4
3
→ e5
1 ?
 Data Structures Week 1

Logical Time
• Armed with our notation of events and precedences amongst
events, we now study logical time.
• We now see how logical time can be maintained in a distributed
system.
• Three ways to implement logical time -
– scalar time,
– vector time, and
– matrix time
scalar time,
vector time, and
matrix time
 Data Structures Week 1

Logical Time


The logical clock C is a function that maps an event e in a
distributed system to an element in the time domain T, denoted as
C(e) and called the timestamp of e, and is defined as follows:
C:H→T
such that for two events ei and ej , ei → ej  C(ei) < C(ej).

Elements of T form a partially ordered or totally ordered set over a

relation < ?

32
 Data Structures Week 1

Logical Time

Consistent: When T and C satisfy the condition: for two events
ei and ej , ei → ej  C(ei) < C(ej).
This property is called the clock consistency condition.


Strongly Consistent: When T and C satisfy the condition:
for two events ei and ej , ei → ej  C(ei) < C(ej).
then the system of clocks is said to be strongly consistent.

33
 Data Structures Week 1

Logical Time

Each processor needs some data structure to
represent logical time.
 A local logical clock, denoted by lci, that helps process
pi measure its own progress.
 A logical global clock, denoted by gci, that is a
representation of process pi’s local view of the logical
global time.
 Allows pi to assign consistent time stamps to its local
events.
 Typically, lci is a part of gci.

Each processor needs a protocol to update the
data structures to ensure the consistency condition.
34
 Data Structures Week 1

Logical Time

Each processor needs a protocol to update the
data structures to ensure the consistency
condition.

Rule 1: Specify how the local logical clock is updated
by a process when it executes an event.

Rule 2: Specify how a process updates its logical
global clock to update its view of the global time and
global progress

all logical clocks systems implement Rule1 and
Rule2

35
 Data Structures Week 1

Scalar Time

Proposed by Lamport in 1978

Lamport won the 2013 Turing award for “contributions
to the theory and practice of distributed and
concurrent systems, notably the invention of concepts
such as causality and logical clocks, .....”.


Time domain is the set of non-negative integers.
 The logical local clock of a process pi and its local
view of the global time are combined into one
integer variable Ci.

36
 Data Structures Week 1

Scalar Time
 Data structure: The logical local clock of a process pi
and its local view of the global time are squashed
into one integer variable Ci.

Rule1: Before executing an event (send, receive, or
internal), process pi executes the following:
Ci := Ci + d (d > 0)

Rule 2: Each message piggybacks the clock value of its
sender at sending time. When a process p i receives a
message with timestamp Cmsg , it executes the following
actions:
1. Ci := max(Ci, Cmsg )
2. Execute Rule1. 37
3. Deliver the message.
 Data Structures Week 1

Scalar Time

Example. Use d = 1 and assign time stamps for
the events.

38
 Data Structures Week 1

Scalar Time

Example

39
 Data Structures Week 1

Scalar Time Properties


Monotonicity
 For two events ei and ej , ei → ej  C(ei) < C(ej).


Total Ordering: Can use the logical time given by the scalar
clocks to induce a total order .

Note that the timestamps alone do not induce a total order.

Two events at different processors can have an identical
timestamp.
 But the tuple (t, i) for each event with (t1,i1) < (t2, i2) if either t1
< t2 or ( (t1 == t2) and i1<i2) is a total order. In (t,i), t denotes
timestamp and i the identiy of the process.

This total order is consistent with the relation →.
Note: According to the total order above: For events e1 and
e2, e1 < e2  Either e1 → e2 or e1 || e2.
40
 Data Structures Week 1

Scalar Time: Properties

• Total Ordering: Tie breaking mechanism.Requests
are timestamped and served according to the total
order based on these timestamp

Consider the tuple (t, i) for each event where
t denotes timestamp and i the identiy of the process.
 (t ,i ) < (t , i ) if either t < t or ( (t == t ) and i <i ) is
1 1 2 2 1 2 1 2 1 2
a total order.
This total order is consistent with the relation →.
Note: According to the total order above: For events
e1 and e2, e1 < e2  Either e1 → e2 or e1 || e2.
 Data Structures Week 1

Scalar Time

Event Counting:

Strong Consistency:

Sriram Murthy

42
 Data Structures Week 1

Scalar Time

Event Counting: Set the increment d to 1 always.

If some event e has a timestamp t, then e is
dependent on t – 1 other events to occur.

This can be called as the height of event e.
Sriram Murthy

No Strong Consistency: Note that scalar time does
not provide strong consistency. [Strong consistency
requires that ei → ej  C(ei) < C(ej).]
Example suffices. Refer to the timeline again.

43
 Data Structures Week 1

Scalar Time

 No Strong Consistency: Consider third event of p1

and p2.

44
 Data Structures Week 1

Scalar Time: Limitation


One of the limitations of scalar time is the lack of
strong consistency.

Strong consistency not achieved as scalar time
uses a single time that represents the logical local
clock and the logical global clock.

This means that the causality of events across
processors is lost. That is, by looking at the times we
can not make out which two events are casually
related and which two are not.


Vector time solves this problem but with big data
structures.

45