CST402 DISTRIBUTED COMPUTING

MODULE: 3

Part 1: Distributed mutual exclusion algorithms

Introduction
➢ Mutual exclusion is a fundamental problem in distributed computing systems.
➢ Mutual exclusion in a distributed system states that only one process is allowed to
execute the critical section (CS) at any given time.
➢ Mutual exclusion is a concurrency control property which is introduced to prevent
race conditions.
➢ Message passing is the sole means for implementing distributed mutual exclusion.
➢ The decision as to which process is allowed access to the CS next is arrived at by
message passing.
➢ The design of distributed mutual exclusion algorithms is complex because, these
algorithms have to deal with unpredictable message delays and incomplete
knowledge of the system state.

Basic Approaches for mutual exclusion

➢ There are three basic approaches for implementing distributed mutual exclusion:
1. Token-based approach.
2. Non-token-based approach.
3. Quorum-based approach
1. Token-Based Approach
➢ Unique token (PRIVILEGE message) shared among sites.
➢ A site can enter its Critical Section (CS) only if it possesses the token and holds it
until CS execution is over.
➢ Mutual exclusion ensured by the uniqueness of the token.
➢ Example: Token Ring algorithm, where a token circulates among the sites in a logical
ring fashion.
2. Non-Token-Based Approach
➢ A site enters the critical section (CS) when an assertion, defined on its local variables,
becomes true.
3. Quorum-based Approach
➢ Each site requests permission to execute the CS from a subset of sites, called a
quorum.

1
 Token Based Algorithm Non-Token Based Algorithm
Uses token to enter into Critical No token is required to enter into
section critical section
Uses sequence number to execute thecritical
Uses timestamps to enter the criticalsection
section
Every request has a sequence number to Every request has a timestamp toenter
enter the critical section the critical section
Higher sequence number will be Higher the timestamp, lower the
having the lowest priority priority
Example: Suzuki-kasami’s broadcast
Example: Ricart-agrawal’s algorithm
algorithm

3.1 System model

➢ The system consists of N sites, S1, S2,….., SN .
➢ Assume that a single process is running on each site.
➢ The process at site Si is denoted by pi.
➢ All these processes communicate asynchronously over an underlying communication
network.
➢ A process wishing to enter the CS, requests all other or a subset of processes by sending
REQUEST messages, and waits for appropriate replies before entering the CS.
➢ A site can be in one of the following three states:
➢ Requesting the CS
➢ Executing the CS
➢ Neither requesting nor executing the CS (i.e., idle).
➢ In the “requesting the CS” state, the site is blocked and cannot make further requests
for the CS.
➢ In the “idle” state, the site is executing outside the CS.
➢ In the token-based algorithms, a site can also be in a state where a site holding the token
is executing outside the CS.
➢ Such state is referred to as the idle token state.
➢ At any instant, a site may have several pending requests for CS.
➢ A site queues up these requests and serves them one at a time.
3.2 Requirements of mutual exclusion algorithms
➢ A mutual exclusion algorithm should satisfy the following properties:

1. Safety property:
❖ Only one process can execute the critical section.
❖ This is an essential property of a mutual exclusion algorithm.
2. Liveness property:
❖ This property states the absence of deadlock and starvation.
❖ Two or more sites should not endlessly wait for messages that will never
arrive.

2
 ❖ Every requesting site should get an opportunity to execute the CS in finite
time.

3. Fairness:
❖ Each process gets a fair chance to execute the CS.
❖ In mutual exclusion algorithms, the fairness property generally means that the
CS execution requests are executed in order of their arrival in the system.

The safety property is absolutely necessary and the liveness & Fairness properties are
considered important in mutual exclusion algorithms.

Performance metrics

➢ The performance of mutual exclusion algorithms is generally measured by the

following four metrics:
1. Message complexity: This is the number of messages that are required per CS
execution by a site.
2. Synchronization delay: After a site leaves the CS, it is the time required and
before the next site enters the CS.
3. Response time: This is the time interval a request waits for its CS execution
to be over after its request messages have been sent out.
4. System throughput: This is the rate at which the system executes requests for
the CS.
➢ If SD is the synchronization delay and E is the average critical section execution time,
then the throughput is given by the following equation:

System throughput = 1
SD+E

3
Low and high load performance
➢ Under low load conditions, there is seldom more than one request for the critical
section present in the system simultaneously.
➢ Under heavy load conditions, there is always a pending request for critical section at
a site.

1. Non token based mutual exclusion algorithms

3.3 Lamport’s algorithm
➢ Lamport’s algorithm is a non-token based mutual exclusion algorithm.
➢ Lamport developed a distributed mutual exclusion algorithm as an illustration of his
clock synchronization scheme.
➢ The algorithm is fair in the sense.
➢ A request for CS is executed in the order of their timestamps and time is determined
by logical clocks.
➢ When a site processes a request for the CS, it updates its local clock and assigns a
timestamp.
➢ The algorithm executes CS requests in the increasing order of timestamps.
➢ Every Site Si keeps a queue called request_queuei .
➢ When a site removes a request from its request queue, its own request may come at
the top of the queue, enabling it to enter the CS.

4
 Lamport’s algorithm

Requesting the critical section:

➢ When a site Si wants to enter the CS,
❖ it broadcasts a REQUEST(tsi, i) message to all other sites and
❖ places the request on request_queuei.
❖ (tsi, i) denotes the timestamp of the request.
➢ When a Site Sj receives the REQUEST(tsi, i) message from site Si,
❖ it places site Si’s request on request_queuej and
❖ returns a timestamped REPLY message to Si.
Executing the critical section:
➢ Site Si enters the CS when the following two conditions hold:
L1: Si has received a message with timestamp larger than (tsi, i) from all other sites.
L2: Si’s request is at the top of request_queuei.
Releasing the critical section:
➢ Site Si, upon exiting the CS,
❖ removes its request from the top of its request queue and
❖ broadcasts a timestamped RELEASE message to all other sites.
➢ When a Site Sj receives a RELEASE message from Site Si, it removes Si’s request from
its request queue.

Example:
➢ Sites S1 and S2 are making requests for the CS and send out REQUEST messages to
other sites.
➢ The timestamps of the requests are (1,1) and (1,2), respectively.

➢ Both the sites S1 and S2 have received REPLY messages from all other sites.
➢ Site S1 has its request at the top of its request_queue but
➢ Site S2 does not have its request at the top of its request_queue.
➢ Site S1 enters the CS.

5
 ➢ S1 exits and sends RELEASE messages to all other sites.

➢ Site S2 has received REPLY from all other sites and also received a RELEASE message
from site S1.
➢ Site S2 updates its request_queue and its request is now at the top of its request_queue.
➢ It enters the CS next.

Performance
➢ For each CS execution, Lamport’s algorithm requires:
❖ N −1 REQUEST messages
❖ N −1 REPLY messages
❖ N −1 RELEASE messages
➢ Thus, Lamport’s algorithm requires 3N − 1 messages per CS invocation.

6
 ➢ The synchronization delay in the algorithm is T.
Advantages
➢ Simplicity
➢ Fairness
➢ Scalability
➢ Compatibility
Disadvantages
➢ Message Overhead
➢ Delayed Execution
➢ Limited Performance
➢ No Fault Tolerance

3.4 Ricart–Agrawala algorithm

➢ Ricart–Agrawala algorithm is a non-token based mutual exclusion algorithm.
➢ Here assumes the communication channels are FIFO.
➢ The algorithm uses two types of messages:
❖ REQUEST
❖ REPLY
➢ A process sends a REQUEST message to all other processes.
➢ A process sends a REPLY message to the requesting process.
➢ Processes use Lamport-style logical clocks.
➢ Timestamps are used to decide the priority of requests.

Ricart–Agrawala algorithm

Requesting the critical section:

(a) When a site Si wants to enter the CS, it broadcasts a timestamped REQUEST message
to all other sites.
(b) When site Sj receives a REQUEST message from site Si,
➢ it sends a REPLY message to site Si if site Sj is neither requesting nor executing
the CS, or
➢ if the Site Sj is requesting and Si’s request’s timestamp is smaller than site Sj’s
own request’s timestamp, the reply is deferred and Sj sets RDi[j] = 1.
Executing the critical section:
(c) Site Si enters the CS after it has received a REPLY message from every site it sent a
REQUEST message to.
Releasing the critical section:
(d) When site Si exits the CS,
➢ it sends all the deferred REPLY messages:
∀j if RDi[j] = 1,
➢ then sends a REPLY message to Sj and sets RDi[j] = 0.

7
Example
➢ Sites S1 and S2 are each making requests for the CS and sending out REQUEST
messages to other sites.
➢ The timestamps of the requests are (1,1) and (1,2), respectively.

➢ S1 has received REPLY messages from all other sites and enters the CS.

➢ S1 exits the CS and sends a REPLY message to site S2.

8
 ➢ Site S2 has received REPLY from all other sites and enters the CS next.

Performance
➢ For each CS execution, the Ricart– Agrawala algorithm requires:
❖ N − 1 REQUEST messages and
❖ N − 1 REPLY messages.
➢ Thus, it requires 2N −1 messages per CS execution.
➢ The synchronization delay in the algorithm is T.
Advantages
➢ Low Message complexity
➢ Scalability
➢ Non Blocking
Drawback
➢ Unreliable Approach

2. Quorum based mutual exclusion algorithms

3.5 Quorum based mutual exclusion algorithms

➢ A site does not request permission from all other sites, but only from a subset of the
sites.
➢ In quorum-based mutual exclusion algorithm, the request set of sites are chosen such
that:
∀i ∀j : 1 ≤ j, j ≤ N :: Ri ∩ Rj =∅
➢ In quorum-based mutual exclusion algorithm, a site can send out only one REPLY
message at any time.
➢ This algorithm is based on the notion of “Coteries” and “Quorums”
➢ A coterie C is defined as a set of sets, where each set g ∈C is called a quorum.

Quorum properties in a coterie

1. Intersection property:
❖ For every quorum g, h∈C, g ∩h = ∅.
❖ For example, sets {1,2,3}, {2,5,7} and {5,7,9} cannot be quorums in a coterie.

9
2. Minimality property:
❖ There should be no quorums g, h in coterie C such that g ⊇ h.
❖ For example, sets {1,2,3} and {1,3} cannot be quorums in a coterie.
➢ Coteries and quorums can be used to develop algorithms to ensure mutual
exclusion in a distributed environment.
➢ Due to the Intersection property, quorum “A” contains at least one site that is
common to the quorum of every other site.
➢ These common sites send permission to only one site at any time.
➢ Thus, mutual exclusion is guaranteed.
➢ Minimality property ensures efficiency rather than correctness.

3.6 Maekawa’s algorithm

➢ Maekawa’s algorithm was the first quorum-based mutual exclusion algorithm.

➢ Three types of messages:
1. REQUEST
2. REPLY
3. RELEASE
➢ A site sends a REQUEST message to all other site in its request set or quorum to
get their permission to enter critical section.
➢ A site sends a REPLY message to requesting site to give its permission to enter the
critical section.
➢ A site sends a RELEASE message to all other site in its request set or quorum upon
exiting the critical section.

Maekawa’s Algorithm
Requesting the critical section:
(a) A Site Si requests access to the CS by sending REQUEST(i) messages to all sites in its
request set Ri.
(b) When a Site Sj receives the REQUEST(i) message, it sends a REPLY(j) message to Si.
Otherwise, it queues up the REQUEST(i) for later consideration.
Executing the critical section:
(c) Site Si executes the CS only after it has received a REPLY message from every site in
Ri.
Releasing the critical section:
(d) After the execution of the CS is over, Site Si sends a RELEASE(i) message to every
site in Ri.
(e) When a Site Sj receives a RELEASE(i) message from Site Si, it sends a REPLY
message to the next site waiting in the queue and deletes that entry from the queue.

10
Performance
➢ For each CS execution, this algorithm requires:
❖ √N request messages
❖ √N reply messages
❖ √N release messages
➢ Requires invocation of 3√N messages per critical section execution.
➢ Synchronization delay is equal to twice the message propagation delay time.
Advantages
➢ Low message complexity
➢ Fairness
➢ Scalability
Disadvantages
➢ High overhead
➢ Limited Flexibility
➢ Delayed Execution

3. Token-based mutual exclusion algorithms

3.7. Token-based mutual exclusion algorithms
➢ In token-based algorithms, a unique token is shared among the sites.
➢ A site is allowed to enter its CS if it possesses the token.
➢ A site holding the token can enter its CS repeatedly until it sends the token to some
other site.
➢ Token-based algorithms use sequence numbers instead of timestamps.
➢ Every request for the token contains a sequence number.
➢ A site increments its sequence number counter every time it makes a request for the
token.

3.8 Suzuki–Kasami broadcast Algorithm

➢ Suzuki–Kasami algorithm is a token- based algorithm, for achieving mutual exclusion
in distributed systems.
➢ This is a modification of Ricart– Agrawala algorithm.
➢ In token-based algorithms, a site is allowed to enter its critical section if it possesses
the unique token.
➢ Token based algorithm uses sequence number to order the requests.
➢ Each request for critical section contains a sequence number.
➢ This sequence number is used to distinguish old and current requests.
Data structure and Notations
1. An array of integers RN[1…N]:
➢ A Site Si keeps RNi[1…N]
2. An array of integer LN[1…N]:
➢ This array is used by the token.
3. A queue Q:

11
 ➢ This data structure is used by the token to keep record of ID of sites waiting for
the token.

Suzuki–Kasami broadcast Algorithm

Requesting the critical section:

(a) If requesting Site Si does not have the token,
❖ then it increments its sequence number, RNi[i], and
❖ sends a REQUEST (i, sn) message to all other sites. (“sn” is the updated value
of RNi[i].)
(b) When a Site Sj receives this message,
❖ it sets RNj[i] to max (RNj[i],sn).
❖ If Sj has the idle token, then it sends the token to Si if RNj[i] = LN[i] + 1.
Executing the critical section:
(c) Site Si executes the CS after it has received the token.
Releasing the critical section:
❖ Having finished the execution of the CS, Site Si takes the following actions:
(d) It sets LN[i] element of the token array equal to RNi[i].
(e) For every Site Sj whose i.d. is not in the token queue, it appends its
i.d. to the token queue if RNi[j] = LN[j] + 1.
(f) If the token queue is nonempty after the above update, Si deletes the top site
i.d. from the token queue and sends the token to the site indicated by the i.d.

Example:
❖ There are 4 sites in the distributed environment S1, S2, S3 and S4.
❖ S3 is the initial state of the system.
❖ Currently, S2 holds the token.
❖ Since the number of sites is 4, the Request array & Token array of size is 4, i.e.
R1[0,0,0,0], R2[0,0,0,0], R3[0,0,0,0], R4[0,0,0,0] & T [0,0,0,0].

❖ S1 wants to enter the Critical section & broadcasts token request REQUEST (1,1).
❖ On starting the broadcast, S1 updates its Request Array with 1.

12
 REQUEST(1,1)
❖ On receiving S1’s request, the conditions have to be checked.
1. Check for outdated request
R2[1] =0 & n=1
n > R2[1]
i.e, (1>0)
So, not an outdated request.

2. Determining outstanding request

R2[1] =T+1
i.e, 0+1=1
❖ Now the token is granted to S1.

R3[1,0,0,0]

13
❖ After S1 completing the Critical Section, the token Array is updated and the queue is
emptied.

❖ Next, S3 & S4 are simultaneously requesting the Critical Section.

Q= [S3, S4]

❖ Now the top of Queue is S3, the 2 conditions to be checked:

1. Check for outdated request
R1[3]=0 and n=1
n > R1[3]
So not an outdated request
2. Determining outstanding request
R1[3] = T+1
0+1=1

14
 ❖ Now the token is issued to S3.

❖ After S3 completing the Critical Section, it sets T3=R3 and queue top is deleted.

❖ Only S4’s request is pending.

❖ The same procedure is followed by S4.

Performance
➢ Synchronization delay is 0.
➢ No message is needed if the site holds the idle token at the time of its request.
➢ In case site does not holds the idle token,
❖ maximum synchronization delay = maximum message transmission time
15
 ❖ a maximum of N message is required per critical section invocation.
➢ This N messages involves:
❖ (N – 1) request messages
❖ 1 reply message
Drawback

➢ Non-symmetric Algorithm

Part 2: Deadlock detection in distributed systems

3.9 Deadlock detection in distributed systems
➢ A deadlock can be defined as a condition where, a set of processes request resources
that are held by other processes in the set.
➢ Deadlocks can be dealt with using any one of the following three strategies:
❖ Deadlock prevention
❖ Deadlock avoidance
❖ Deadlock detection
3.10 System model

➢ A distributed system consists of a set of processes that are connected by a

communication network.
➢ The communication delay is finite but unpredictable.
➢ A distributed program is composed of a set of n asynchronous processes P1, P2,….,
Pi,….., Pn.
➢ Assume each process is running on a different processor.
➢ The processors do not share a common global memory.
➢ Communication solely by passing messages over the communication network.
➢ There is no physical global clock in the system.
➢ The communication medium may deliver messages:
➢ out of order
➢ messages may be lost
➢ garbled
➢ duplicated
➢ The system can be modelled as a directed graph:
❖ Vertices processes
❖ Edges unidirectional communication channels
➢ We make the following assumptions:
❖ The systems have only reusable resources.
❖ Processes are allowed to make only exclusive access to resources.
❖ There is only one copy of each resource.
❖ A process can be in two states:
➢ Running (Active State)
➢ Blocked

16
Wait-For Graph (WFG)
➢ In distributed systems, the state of the system can be modelled by directed graph, called
a wait-for graph (WFG).
➢ In a WFG,
❖ Nodes are processes and there is a directed edge from node P1 to node P2, if
P1 is blocked and waiting for P2 to release some resource.
➢ A system is deadlocked:
❖ if and only if there exists a directed cycle or knot in the WFG.

Fig. Example of a WFG

➢ Figure shows a WFG, where process P11 of site 1 has an edge to process P21 of site
1 and an edge to process P32 of site 2.
➢ Process P32 of site 2 is waiting for a resource that is currently held byprocess P33
of site 3.
➢ At the same time process P21 at site 1 is waiting on process P24 at site 4to release a
resource, and so on.
➢ If P33 starts waiting on process P24, then processes in the WFG areinvolved in
a deadlock depending upon the request model.

3.11 Deadlock handling strategies

➢ There are three strategies for handling deadlocks:
❖ Deadlock prevention
❖ Deadlock avoidance
❖ Deadlock detection
➢ Handling of deadlocks becomes highly complicated in distributed systems.
1. Deadlock prevention
➢ This is typically accomplished either by having a process obtain all the necessary
resources at once before starting execution, or by interrupting a process that currently
holds the required resource.
➢ This approach is highly inefficient and impractical in distributed systems.

17
2. Deadlock avoidance
➢ A resource is granted to a process if the resulting global system state is safe.
➢ Global state includes all processes and resources of the distributed system.
➢ Due to several problems, deadlock avoidance is impractical in distributed systems.
3. Deadlock detection
➢ Requires an examination of the status of process– resource interactions for the
presence of cyclic wait.
➢ Deadlock detection in distributed systems seems to be the best approach to handle
deadlocks in distributed systems.

3.12 Issues in deadlock detection

➢ Addressing two basic issues:
1. Detection of existing deadlocks
2. Resolution of detected deadlocks
1. Detection of deadlocks
➢ Detection of deadlocks involves addressing two issues:
1. Maintenance of the WFG
2. Searching of the WFG for the presence of cycles (or knots)
➢ A cycle or knot may involve several sites.
➢ The search for cycles greatly depends upon how the WFG of the system is represented
across the system.
➢ Depending upon the way WFG information is maintained and the search for cycles is
carried out:
❖ There are centralized, distributed, and hierarchical algorithms for deadlock
detection in distributed systems.
➢ A deadlock detection algorithm must satisfy the following two conditions:
1.Progress (no undetected deadlocks)
❖ The algorithm must detect all existing deadlocks in a finite time.
❖ Once a deadlock has occurred, the deadlock detection activity should
continuously progress until the deadlock is detected.

2. Safety (no false deadlocks)

❖ The algorithm should not report deadlocks that do not exist, called phantom
or false deadlocks.

2. Resolution of a detected deadlock

➢ Deadlock resolution involves breaking existing wait-for dependencies between the
processes to resolve the deadlock.
➢ It involves rolling back one or more deadlocked processes and assigning their
resources to blocked processes so that they can resume execution.
➢ When a wait-for dependency is broken, the corresponding information should be
immediately cleaned from the system.

18
 ➢ If this information is not cleaned in a timely manner, it may result in detection of
phantom deadlocks.

3.13 Models of deadlock

➢ The various models of deadlock are:

1. Single Resource Model
2. AND Model
3. OR Model

4. p Model (p out of q model)

q

5. Unrestricted model

1. Single Resource Model

➢ A process can have at most one outstanding request for only one unit of a resource.
➢ The maximum out-degree of a node in a WFG for the single resource model can
be 1, the presence of a cycle in the WFG shall indicate that there is a deadlock.
2. AND Model
➢ In the AND model, a passive process becomes active only after a message from each
process in its dependent set has arrived.
➢ In the AND model, a process can request more than one resource simultaneously and
the request is satisfied only after all the requested resources are granted to the process.
➢ The out degree of a node in the WFG for AND model can be more than 1.
➢ The presence of a cycle in the WFG indicates a deadlock in the AND model.
➢ Each node of the WFG in such a model is called an AND node.
➢ In the AND model, if a cycle is detected in the WFG, it implies a deadlock but not vice
versa.

Deadlock

Deadlock

19
 ➢ Consider process P44, it is not a part of any cycle but is still deadlocked as it is
dependent on P24, which is deadlocked.

3. OR Model
➢ A process can make a request for numerous resources simultaneously and the request
is satisfied if any one of the requested resources is granted.
➢ Presence of a cycle in the WFG of an OR model does not imply a deadlock in the OR
model.
➢ In the OR model, the presence of a knot indicates a deadlock.
➢ With every blocked process, there is an associated set of processes called dependent
set.
➢ A process shall move from an idle to an active state on receiving a grant message from
any of the processes in its dependent set.

Not deadlocked

Requesting Resources

Granting Resources

➢ The process P11 is not deadlocked because once process P33 releases its resources, P32
shall become active as one of its requests is satisfied.
➢ After P32 finishes execution and releases its resources, process P11 can continue with
its processing.

4. The AND-OR model

➢ A generalization of OR model and AND model is the AND-OR model.

➢ In the AND-OR model, a request may specify any combination of AND and OR in the

20
 resource request.
➢ For example, in the AND-OR model, a request for multiple resources can be of the form
x and (y or z).
➢ To detect the presence of deadlocks in such a model, there is no familiar construct of
graph theory using WFG.
➢ A deadlock in the AND-OR model can be detected by repeated application of the
test for OR-model deadlock.
➢ However, this is a very inefficient strategy.

P
5. q Model (p out of q model)

➢ This is a variation of AND-OR model.

➢ This allows a request to obtain any k available resources from a pool of n resources.
➢ Every request in this model can be expressed in the AND-OR model and vice-versa.

p
6. Unrestricted model
➢ No assumptions are made regarding the underlying structure of resource requests.
➢ In this model, only one assumption that the deadlock is stable is made and hence it
is the most general model.

**************************************************

21