Distributed Systems

Coordination
(Mutual exclusion & Election)

Thoai Nam
High Performance Computing Lab (HPC Lab)
Faculty of Computer Science and Engineering
HCMC University of Technology

HPC Lab - CSE - HCMUT 1

Mutual exclusion

HPC Lab - CSE - HCMUT 2

Mutual exclusion
• Problem
o A number of processes in a distributed system want exclusive access to some
resource
• Goal (avoiding)
o Starvation: not every process gets a chance to access the resource
o Deadlocks: several processes are waiting for each other to proceed
• Basic solutions
o Permission-based: A process wanting to enter its critical section, or access a resource,
needs permission from other processes
o Token-based: A token is passed between processes. The one who has the token may
proceed in its critical section, or pass it on when not interested.

HPC Lab - CSE - HCMUT 3

Critical section
• Critical section
o Considering a system of N processes pi (i = 1, 2,..., N), that do not share variables.
The processes access common resources, but they do so in a critical section
• The application-level protocol for executing a critical section is as follows:
o enter(): enter critical section – block if necessary
o resourceAccesses(): access shared resources in critical section
o exit(): leave critical section – other processes may now enter
• Mutual exclusion requirements

ME1: (safety) At most one process may execute in the critical section (CS) at a time
ME2: (liveness) Requests to enter and exit the critical section eventually succeed
ME3: (Optional, stronger) Requests to enter granted according to causality order

HPC Lab - CSE - HCMUT 4

Permission-based, centralized (1)
• Simply use of a coordinator
• One process is elected as the coordinator, which only lets one process at a
time to access the resource:
a) Process P1 asks the coordinator for permission to access a shared resource. Permission is
granted.
b) Process P2 then asks permission to access the same resource. The coordinator does not reply.
c) When P1 releases the resource, it tells the coordinator, which then replies to P2.

HPC Lab - CSE - HCMUT 5

Permission-based, centralized (2)
• Pros
o Fairness
o No starvation
o Simplicity, only three messages (request, grant, release).
• Cons
o Coordinator is a single point of failure and performance bottleneck
o Distinguishing crashed coordinator from permission denied.

HPC Lab - CSE - HCMUT 6

A distributed algorithm
• Return a response to a request only when:
o The receiving process has no interest in the shared resource; or
o The receiving process is waiting for the resource, but has lower priority
(known through comparison of timestamps).
• In all other cases, reply is deferred, implying some more local
administration.

HPC Lab - CSE - HCMUT 7

Multicast
• FIFO ordering: If a correct process issues multicast(g,m) and then
multicast(g,m’), then every correct process that delivers m’ will have already
delivered m
• Causal ordering: If multicast(g,m) → multicast(g,m’) then any correct
process that delivers m’ will have already delivered m
o Note that → counts multicast messages delivered to the application, rather than all
network messages
• Total ordering: If a correct process delivers message m before m’, then any
other correct process that delivers m’ will have already delivered m.

HPC Lab - CSE - HCMUT 8

A distributed algorithm: Ricart & Agrawala (1981)
On initialization
state := RELEASED;
To enter the section
state := WANTED;
Multicast request to all processes; request processing deferred here
T := request’s timestamp;
Wait until (number of replies received = (N – 1));
state := HELD;
On receipt of a request <Ti , pi > at pj (i ≠ j)
if (state = HELD or (state = WANTED and (T, pj) < (Ti, pi)))
then
queue request from pi without replying;
else
reply immediately to pi;
end if
To exit the critical section
state := RELEASED;
reply to any queued requests;
HPC Lab - CSE - HCMUT 9
Ricart & Agrawala (1981) (1)
• Based on total ordering of all events in the system
• A process wanting to access a resource builds a message containing the name of the
resource, its process number and the current (logical) time
• The process sends the message to all other processes
• A process receiving the request has three alternatives:
1. If the receiver is not accessing the resource and does not want to access it, it sends back an
OK message to the sender
2. If the receiver already has access to the resource, it simply does not reply. Instead, it queues
the request
3. If the receiver wants to access the resource as well but has not yet done so, it compares the
timestamp of the incoming message with the one contained in the message that it has sent
everyone. The lowest one wins. If the incoming message has a lower timestamp, the receiver
sends back an OK message. If its own message has a lower timestamp, the receiver queues
the incoming request and sends nothing
• The process waits until everyone has given permission.

HPC Lab - CSE - HCMUT 10

Ricart & Agrawala (1981): an example

① Two processes want to access a shared resource at the same moment

② P0 has the lowest timestamp, so it wins
③ When process P0 is done, it sends an OK also, so P2 can now go ahead.

HPC Lab - CSE - HCMUT 11

(1) (2)

(4) (3)

HPC Lab - CSE - HCMUT 12

(5) (6) (7)

(11) (10) (9) (8)

HPC Lab - CSE - HCMUT 13

Ricart & Agrawala (1981)

• Pros
o Mutual exclusion is guaranteed without deadlock or starvation
• Cons
o N points of failure (crashed process interpreted as denial of access)
o Requires more communication
o Low efficiency, as all processes are involved in all decisions (n bottlenecks).

HPC Lab - CSE - HCMUT 14

Observation
V1
V2 P0 P3

• In order for a process to enter a critical section, it is

not necessary for all of its peers to grant it access
• Processes need only obtain permission to enter P1 P2 V0
from subsets of their peers, as long as the subsets
used by any two processes overlap. V3

HPC Lab - CSE - HCMUT 15

Maekawa’s algorithm (1985)

• A voting set Vi with each process pi ( i = 1, 2,..., N ), where Vi ⊆ {p1, p2, ..., pN}
• The sets Vi are chosen so that, for all i, j = 1, 2,..., N:
o pi ∈ V i
o Vi ⋂ Vj ≠ ∅, there is at least one common member of any two voting sets
o |Vi| = K - to be fair, each process has a voting set of the same size
o Each process pi is contained in M of the voting sets Vi.
• Optimal solution
o K ~ !!, M = K

HPC Lab - CSE - HCMUT 16

 !!

HPC Lab - CSE - HCMUT 17

Mutual exclusion: Token ring algorithm

• Organize processes in a logical ring, a token is introduced, and the idea

is to let the token be passed between processes. The one that holds the
token is allowed to enter the critical region (if it wants to)
• An overlay network constructed as a logical ring with a circulating token.

HPC Lab - CSE - HCMUT 18

Token ring algorithm

• Pros
o No starvation
o Relatively easy to recover
• Cons
o Token can disappear (be loss)
o Long delay between successive appearances of the token.

HPC Lab - CSE - HCMUT 19

Leader election

HPC Lab - CSE - HCMUT 20

Online payment (1)

Customer Online Store Service Payment

Order
Charge

Success
Order dispatched

HPC Lab - CSE - HCMUT 21

Online payment (2) Leader
Server 1 Server 2 Server 3

Online Store Service Payment Service Payment Service Payment

Charge

Success

Multiple copies

HPC Lab - CSE - HCMUT 22

Other applications
• Master/Slave
o Hadoop (Map-reduce jobs)
• Primary/Secondary
o DNS
• Primary/Replica
o Drupal, Django, Amazon Relation Database Service
• Controller/Agent
o Jenkins.

HPC Lab - CSE - HCMUT 23

Leader election
• Sometimes a single process in the system needs to have special powers, e.g., grant
access of a shared resource and assign work to others
• Selection of a leader given a candidate of processes/nodes
• When selecting a leader, two properties need to be guaranteed:
o Safety: There is at most one leader at any given time
o Liveness: The election eventually is completed
• The leader election process can be extended to provide consensus.

HPC Lab - CSE - HCMUT 24

Election algorithms
• Principle: an algorithm requires that some process acts as a coordinator
o The question is how to select this special process dynamically.
• In many systems the coordinator is chosen by hand (e.g. file servers)
o This leads to centralized solutions → single point of failure.
• Questions:
o If a coordinator is chosen dynamically, to what extent can we speak about a
centralized or distributed solution?
o Is a fully distributed solution, i.e. one without a coordinator, always more robust than
any centralized/coordinated solution?

HPC Lab - CSE - HCMUT 25

Basic assumptions
• All processes have unique id’s
• All processes know id’s of all processes in the system (but not if they are up
or down)
• Election means identifying the most suitable process based on different
factors, e.g., highest id.

HPC Lab - CSE - HCMUT 26

Election by bullying (1)

Principle (Garcia-Molina 1982)

Consider N processes {P0.....PN-1} and let id(Pk) = k. When a process Pk notices that the coordinator is
no longer responding to requests, it initiates an election:
1. Pk sends an ELECTION message to all processes with higher identifiers: Pk+1, Pk+2,..., PN-1
2. If no one responds, Pk wins the election and becomes coordinator
3. If one of the higher-ups answers, it takes over and Pk ’s job is done.

HPC Lab - CSE - HCMUT 27

Election by bullying (2)

1. (a) Process 4 first notices that coordinator has crashed and sends ELECTION to processes with
higher numbers 5,6, and 7
2. (b)-(d) Election proceeds, converging into process 6 winning
3. (e) By sending COORDINATOR message process 6 announces it is ready to take over
4. If process 7 is restarted, it will send COORDINATOR message to others and bully them into
submission.
HPC Lab - CSE - HCMUT 28
 election
C
election

Election by bullying (2) Stage 1

p
answer
p p p
1 2 3 4
answer
Principle (Garcia-Molina 1982) election
C
Consider N processes {P0.....PN-1} and let id(Pk) = k. When a process Pk election election
notices that the coordinator is no longer responding to requests, it Stage 2
initiates an election: answer
p1 p p p
1. Pk sends an ELECTION message to all processes with higher 2 3 4
identifiers: Pk+1, Pk+2,..., PN-1
2. If no one responds, Pk wins the election and becomes
timeout
coordinator
3. If one of the higher-ups answers, it takes over and Pk ’s job is Stage 3
done.
p p p p
1 2 3 4

Eventually.....
coordinator
C
Stage 4
p p p p
1 2 3 4

The election of coordinator p2, after the failure of p4 and then p3

HPC Lab - CSE - HCMUT 29
Election in a ring
• Process priority is obtained by organizing processes into a (logical) ring
• Process with the highest priority should be elected as coordinator
o Any process can start an election by sending an election message to its successor. If a
successor is down, the message is passed on to the next successor
o If a message is passed on, the sender adds itself to the list. When it gets back to the
initiator, everyone had a chance to make its presence known
o The initiator sends a coordinator message around the ring containing a list of all living
processes. The one with the highest priority is elected as coordinator.

HPC Lab - CSE - HCMUT 30

Election in a ring: an example

• The initiators are P6 and P3.

HPC Lab - CSE - HCMUT 31

Election in wireless networks
(1) (2) (3)

(5) (4)

HPC Lab - CSE - HCMUT 32