UNIT IV

4 Consensus and Recovery

Syllabus

Failure-Free System (Synchronous

:
ConsensusandAgreement Algorithns Problem Definition - Overview of Results -Agreementin a
and Asynchronous) -Agreementin Synchronous Systems with
Failures; Check-pointing and Rollback Recovery :Introduction - Backgroundand Definitions -
Issues in Failure Recovery- Checkpoint-based Recovery- Coordinated Checkpointing Algorithm -
Algorithm forAsynchronousCheckpointing and Recovery.

Contents
4.1 Consensus and Agreement Algorithms: Problem Definition
4.2 Byzantine Agreement Problem
4.3 Overview of Results
4.4 Solution to Byzantine Agreement Problem
4.5 Agreement in a Failure-Free System (Synchronousand Asynchronous)
4.6 Agreement in Synchronous Systems
with Failures May-22, Marks 13

4.7 Introduction of Check-pointing

and Rollback Recovery Dec.-22, Marks 13

4.8 Background and Definitions

4.9 Consistent Set of Checkpoints

4.10 Issues in Failure Recovery May-22, Dec.-22, Marks 13

4.11 Checkpoint-basedRecovery
4.12 Coordinated Checkpointing Algorithm

4.13 Algorithm for AsynchronousCheckpointing and Recovery

4.14 Two Marks Questions with Answers

(4-1)
Distributed Computing 4-2 Consensus and Recovery

4.1 Consensus and Agreement Algorithms :Problem Definition

• Processes/Sites in distributed systems often compete as well as cooperate to

achieve a common goal. MutualTrust/agreement is very much required.

• In distributed data bases, there may be a situation where data managers have to
decide "Whether to commit or Abort the Transaction". When there is no failure,
reaching an agreement is easy.

• However, in case of failures, processes must exchange their values with other
processes and relay the values received from others several times to isolate the
effect of faulty processor.

Agreement protocols helps to reach an agreement in presence of failures.

• Examples : Agreeing whether to commit or to abort a transaction in a distributed

database management system. Agreeing on a common clock value in a distributed

system.

In the absence of failures or faulty processors, values (that is to be decided) can be

exchanged. A vote can be taken and decision/agreement can be made based on :
majority, minimum vote, mean, etc.

• Presence of failure : Processors can fail or misbehave intentionally. Several rounds

of message exchanges might be needed beforeagreement can be reached.

4.2 Byzantine Agreement Problem

The Problem : "Several divisions of the Byzantine army are camped outside an
enemy city, each
commanded by its own
division general. After observing the
enemy, they must decide upon a common plan of action. Some of the generals
may be traitors, trying to prevent the loyal generals from reaching agreement."
Three or more generals are agree to attack or to retreat.
Once the commander is

issues the order, lieutenants to the commander are to decide to attack or retreat.
• But the one or more of the generalsmay be treacherous, i.e. faulty.

• If the commander is treacherous,he proposes attacking to one general and

retreating to another.

. If a lieutenant is treacherous, he tells one of his peers that the commander told
him toattack and another that they are to retreat.

• Source processor broadcasts its values to others. Solution must meet following
objectives :
Agreement : All non-faulty processors agree on the same value.

TECHNICALPUBLICATIONS an up-thrust for knowledge

Distributed Computing 4-3 Consensusand Recovery

Validity : If source is nonfaulty, then the common agreed value must be the value
supplied by the source processor.

• "If source is faulty then all non - faulty processors can.agree on any common
value"."Value agreed upon by faulty processorsis irrelevant"

• Fig 4.2.1 shows Byzantine agreement.

General1 General 1

Attack Attack Attack Retreat

General 2 General 3 General 2 General3

Retreat Retreat

Fig. 4.2.1

• No solution for three processes can handle a single traitor. In a system with m
faulty processes agreement can be achieved only if there are 2m+1 (more than 2/3)
functioning correctly.

4.2.1 Consensus Problem

i. Every processor broadcasts its initial value to all other processors.

ii. Initial values may be different for differernt processors.

iii. Every processor has its own initial value.

iv. All non faulty processors must agree on a single common value.
• If initial value of non-faulty processors is different then all non - faulty processors

can agree on any comnmon value.

is irrelevant
• Value agreed upon by faulty processors

1. Agreement : Allnon-faulty processors agree on the same single value.

2. Validity : If initial value of every processor is

non-faulty v,then the common
agreed value by all non-faulty processors must be v.

"If initial value of non-faulty processors is different then all non -

faulty
processors can agree on any common value. Value agreed upon by faulty

processors is irrelevant".

4.2.2 Interactive Consistency Problem

1. Every processor has its own initial value.

ii, All non faulty processors must agree on a set of common values.

TECHNICAL PUBLICATIONS- an up-thrust for krnowledge

Distibuted Computing 4-4 Consensus and Recovery

• In all the previous mentioned problems, all non faulty processors must reach an
agreement.

• In Byzantine and consensus problems, agreement is on a single value.

• In interactive consistency problem, agreement is on a set of common values.

• In Byzantine agreement problem, only one processor initializes the value where as
in other two cases, every processorhas its own initial value.

4.3 Overview of Results

Consensus is not solvable in asynchronous system even if one process can fail by
crashing

Sr. No. Failure mode Synchronous system Asynchronous system

1 No failure
Agreement attainable Agreement attainable
2 Crash failure
Agreement attainable Agreement not attainable
[f<n process]
3.
Byzantine failure Agreement attainable Agreement not attainable
[f (n- 1) /3] byzantine
process

4.4 Solution to Byzantine Agreement Problem

• First defined and solved by Lamport.
• An arbitrary source processor broadcasts its initial value to all others.
• If the source processor is faulty, other non-faulty processor can agree on any
Common value.

• Faulty processors' values and agreements do not matter.

• If faulty processors are in majority, then non-faulty processors cannot reach an

agreement.
Number of faulty processors, m, cannot exceed: trunc[(n-1)/3].
This bound can be relaxed for systems using authenticated messages.
•
:
Solutionmust meet following objectives
:
i. Agreement Allnon-faulty processors agree on the same
value.
ii. Validity : If source is nonfaulty, then the common agreed value must be the value
supplied by the source processor.

TECHNICAL PUBLICATIONS - an up-thrust for knowledge

Distributed Computing
4-5 Consensus and Recovery

4.4.1 Impossible Scenario

• Consider a system with 3processors : p0, pl, p2.

• Twopossibilities :
Case I:pO (source) is not faulty. p2 is faulty. p1 should agree upon 1 as the
value. Not
. Case 2: p0is
possible.

faulty. pl may agree on 1 and p2 on 0

Fig. 4.4.1 processor p0 non-faulty and processor pO faulty

p0 p2

7
p1 p2 p1 0 po

Fig. 4.4.1

4.4.2| Lamport-Shostak-Pease Algorithm

This algorithm also known as Oral Message Algorithm OM(m) where m is the
number of faulty processors

• 'n'= Number of processors and n >= 3m+1

Algorithm is recursively defined as follows:

Algorithm OM(0)
1. Source processorsends its values to every processor

2. Each processor uses the value it receives from source. [If no value is received
default value 0 is used]
Algorithm OM(m), m>0
1. Source x broadcasts value to all processes

2. Let vi = value received by process i from source (0 if no value received).

Process i acts as a new source and initiates OM(m - 1), sending vi to

remaining (n -2) processes

3. For each i, j, ? j,let vj =value received
i by process i from process j in step 2
using O(m-1). Process i uses the value majority(v1, v2, ..., vn - 1)
• Time complexity =m+1 rounds
• Message complexity = On")
• You can reduce message complexity to polynomial by increasing time

TECHNICAL PUBLICATIONS - an up-thrust for knowledge

Distnbuted Computing 4-6 Consensusand Recovery

Lamport's Algorithm :Example 1

• System processors:p0, pl,p2, p3. p0 source,p2

:
with 4 is is faulty.

Assumption possible values are only 1 and 0.

• Step 1:p0initiates the initial value to be 1. (Algorithm OM(1), m =1).

• Step 2: OM(0). pl sends 1 to (p2, p3). p3 sends 1 to (pl,p2)

• p2 (the faulty one) sends 1 to pl and 0 to p3.

Step 3 : Majority function at pl and p3 is 1, which is the desired result. (Not

bothered about p2, the faulty one).

p0 p0

p1 p3 p1

(a) Processor p0 executes (b) Processor p1, p2, p3 executes

the algorithm OM(1) the algorithm OM(0)

Fig. 4.4.2

Example 2:
System with 4 processors : p0, p1, p2, p3. p0 is source, and is faulty.

Assumption :
Possible values are only 1 and 0.

Step 1 : p0 initiates the initial value to be 1 for pl and p3. For p2, it sends a

0(Algorithm OM(1),m =1).

Step 2 : OM(0). p1 sends 1 to (p2, p3}. p3 sends 1 to (p1,p2

• p2 sends 0 to pl and p3.

Step 3: Majority function at pl, p2, p3 is still the same (1), which is the desired
result.

p0

Fig. 4.4.3

TECHNIOCAL PUBLICATIONSo - an up-thrust for knowledge

Distributed Computing 4-7 Consensusand Recovery

A.5 Agreement in a Failure-Free System (Synchronous

and Asynchronous)

• In a failure-free system,consensus can be reached by collecting information from

the different processes, arriving at a "decision," and distributing this decision in

the system.

A distributed mechanism would have each process broadcast its values to others,

and each process computes the same functionon the values received.

• Synchronous system : This is implemented a in constant number of rounds.

Common knowledge of the decision value can be obtained using an additional
round.

Asynchronous
of message hops.
system : Consensus can similarly be reached in a constant number

4.6 Agreement in Synchronous Systems with Failures AU: May-22

Algorithm for consensus with up to f fail-stop processes in a system of n processes

are as follows
int f:// globalconstant : maximum number of crash failures tolerated

integer: x+ local value;

Process P;executes the consensus algorithm for up to f crash failures
for round from 1 to f +1 do
if the curent value of x has not been broadcast then
broadcast(x);

- y, value (if any)received from process j in this round;

xmin (x, yj)
output x as the consensus value
The agreement condition is satisfied because in the f +1 rounds, there must be at

least one round in which no process failed.

The validity condition is satisfied because processes do not send fictitious values
in this failure model

• The terminationcondition is seen to be satisfied.

Complexity :There are f + 1 rounds, where f<n. The number of messages is at

most O(n) in each round, and each message has one integer. Hence the total

number of messages is O (f+1)-n).

TECHNICAL PUBLICATIONS - an up-thrust for knowledge

Distnbuted Computing 4 -8 Consensusand Recovery

University Question

1. List the agreement statements that should be follotoed in synchronous systems with failure.

AU: May-22, Marks 13

4.7 Introduction of Check-pointing and Rollback Recovery

AU : Dec.-22

Checkpointing and rollback recovery are well-known techniques that allprocesses

to make progress in spite of failures. The failures under consideration are transient
problems such as hardware errors and transaction aborts.

When failure occurs, the process rolls back to its most recent checkpoint, assume
the state saved in that checkpointand resumes execution.

• Rollback recovery treats a distributed system application as a collection of

processes that communicate over a network. The saved state is called a checkpoint
and the procedure of restarting from a previously checkpointed state is called
rollback recovery.

• In distributed system, rollback recovery is complicated because message induce

inter-process dependencies during failure free operation.

• Checkpoint is a designated place in a program at which normal process is

interrupted specifically to preserve the status information necessary to allow
resumption of processing at a later time.

• Coordinated checkpointing: here processes coordinate their checkpoints to form a

system wide consistent state.

University Question

1. Illustrate briefly the two kinds of checkpoints for checkpoint algorithm.

AU: Dec.-22, Marks 13

4.8 Background and Definitions

4.8.1 System Model

We consider the distributed system with fixed number of processes

(P,,P,..., P).
They are communicating with each other by using messages.
The processes do not
share a common memory or a common clock.

. Fig. 4.8.1 shows system consisting

. Rollback recovery protocol
communication.
generally
of three processes

make assumptions about

and their interactions.

the reliability of the

inter-process

TECHNICAL PUBLICATIONS an up-thrust for knowledge

 Computing
4-9 Consensusand Recovery
Dstnbuted

Input message Output message

M,
Process
M3
Process,
Mo
Process,
M Ms

Fig.4.8.1 Three processes and their interactions

• The computation is asynchronous, i.e., each process progresses at its own speed
and messages are exchanged through reliable channels, whose transmission delays
are finite but arbitrary.

• The messages generated by the underlying distributed application are referred to

as computation messages.

• The messages generated by processes to advance checkpoints are referred to as

system messages.

4.8.2 Local Checkpoint

•Local checkpoint is a snapshot of the state of the process at a given instance and
the event of recording the state of a process is called local checkpointing)

• Each checkpoint taken by a process is assigned a unique sequence number. The in

(i > 0) checkpoint of process P, is assigned a sequence number and is denoted i

by Cp,i"
• We also assume that each process P, takes an initial checkpoint Cp,0 immediately
before execution begins and ends with a virtual checkpoint that represents the last
state attained before termination.

The ¡th checkpoint interval of process P, denotes all the computation performed

between its ith and (i+1) n checkpoint, including the ih checkpoint but not the

(i+1)th checkpoint.

4.9 Consistent Set of Checkpoints

• Fig. 4.9.1 shows consistent and inconsistent set.

• Checkpointing in distributed systems requires that all processes (sites) that interact
with one another establish periodic checkpoints.

• All the sites save their local states : local checkpoints. All the local checkpoints,
one from each site,collectively form a global checkpoint.

TECHNICALPUBLICATIONS - an up-thrust for knowledge

 Consensus and Recovery
Distributed Computing 4-10

Po
Po
m.
m4
P
P
m2
P2
P2

(b) Inconsistent state

(a) Consistent state

Fig. 4.9.1

caused by orphan messages, which in turn are caused by

• The domino effect is

rollbacks.

Strongly consistent set of checkpoints

snapshot in a system is also called the recovery line.
The most recent distributed

• Fig. 4.9.2 shows recovery line and checkpoint.

Initialstate Recovery line Checkpoint

P1

Failure

P2
Time
Messagesent Inconsistent cut
from P2 to P1

Fig. 4.9.2 Recovery line and checkpoint

. Establish a set of local checkpoints(one for each process in the set) such that no

no orphan messages) during the interval

information flow takes place (i.e.,

spanned by the checkpoints.

A strongly consistent set of checkpoints (recovery line) corresponds to a strong!y

consistent global state.

. There is one recovery point for each process in the set during the interval spanned
is no information flow between any pair in the
of processes
by checkpoints;there
and any process outside the set.
set and process in the set

.A Consistent set of checkpoints corresponds to a consistent global state.

TECHNICALPUBLICATIONS- an up-thrust for knowedge

 4-11 Consensus and Recovery
Computing
Asbuted

. Fig.
4.93shows consistent set of checkpoint.

Z
Time

Fig. 4.9.3 Consistent set of checkpoint

. Set (x1 y1 2}is a strongly consistent set of checkpoints.

checkpoints(need to handle lost messages).

• Set lr2 y2. Z2l is a consistent set of

an whose would be undone due to the rollback

• No local checkpoint includes effect

of another process.

checkpoints
.
Consistent set of

Similar to the consistent global state.

checkpoint (state) should also be recorded as

• Each message that is received in
a

sent in another checkpoint (state).

Suppose that Y fails after receiving message m. If Y restarts from checkpoint,

due
message 'm' is lost to rollback.

Checkpoint notation
• Each node maintains:
from node is
counter with which each message that
1. Monotonically increasing
labelled.

message from and the first message to all other nodes.

2. Records of the last

last_label_ received,M

m.1 (amessagem and its label I)

first_label_sent [X]

Fig. 4.9.4

TECHNICAL PUBLICATIONS an up-thrust for knowledge

Distributed
Computing 4- 12 Consensusand Recovery

Note : "sl" denotes a "smallest label" that is < any other label and "l" denotes
"largest label" that is > any other label.

Simple method for taking consistent set of

checkpoint
• Every process takes a checkpoint after sending every message.
• The set of the most recent checkpoints is always consistent.

4.9.1 Synchronous Checkpointing and Recovery

Livelock problem during recovery is avoided by taking a consistent set of

checkpoints.

• Algorithm is said to be synchronous whern the processes involved coordinate their

local checkpointing actions such that the set of all recent checkpoints in the system
guaranteed to be consistent.

4.9.1.1| Checkpointing
Algorithm
• Make some
V simplifying assumptions
Processes communicate by exchanging
messages through channels.
2 Channels are FIFO, end-to-end
protocols cope with message loss due to
rollback recovery.

3. Communication failures do not partition

•A single process invokes the algorithm.
the network.
The checkpoint and the rollback
algorithms are not invoked recovery
concurrently.

Two types of checkpoints

1. Tentative : A temporary checkpoint that

is made a permanent checkpoint on
successful terminationof the checkpoint the
algorithm
2. Permanent : A local checkpointat a process.

Phase One
Initiating process P, takes a
tentative checkpointand requests that all the
take processes
tentative checkpoints.

Each process informs P, whether it succeeded in taking a tentative checkpoint.

A P. learns that all processes
have taken tentative checkpoints,P, decides that all
tentative checkpoints should be made
permanent.
.Otherwise, P decides that all tentative checkpointsshould be
discarded.

TECHNICAL PUBLICATIONS
an up-thrust for knowledge
 Computing 4- 13 Consensus and Recovery
Dstibutead

Two
Phase
its decision to all processes.
propagates

On receiving the message from P, all processes act accordingly.

Between tentative checkpoint and commit/abort of checkpoint process must hold

back messages.

,Does this guarantee we have a strongly consistent state ? Can you construct an
example that shows we can still have lost messages

synchronous Checkpointing : Properties

• Al or none of the processes take permanent checkpoints.

There is no record of a message being received but not sent.

optimization of the Checkpoint Algorithm

• A minimal number of processes take checkpoints.

messages after it has taken its last

• All processes from which P, has received

checkpoint take a checkpoint to record the sending of those messages.

• Fig. 4.9.5 shows the checkpoints taken unnecessarily.

Tentative
Time
checkpoint X2
X
Messages
m
take acheckpoint

Fig. 4.9.5 Checkpoints taken unnecessarily

1. Process X decides to initiate checkpoint algorithm after receiving message 'm'.

2. Ittake a tentative checkpoints X and sends take tentative checkpoint messages to

processes Y and Z,causing Y and Z to take checkpointsy, and respectively. z

3. Now {Xy, y, Z)forms a consisternt set of checkpoints.

4. {X, y,z}also forms a consistent set of checkpoints.

TECHNICALPUBLICATIONS - an up-thust for knowledge

Distributed Computing 4- 14 Consensusand Recovery

5. Y takes tentative received by X from Y

checkpoint only if the last message was

sent after Y sent the first message after the last checkpoint (last_recv(x, y)>.
first_send(y, x)).

• When a process takes a checkpoint,it will ask all other processes that sent

messages to the process to take checkpoints.

Synchronous Checkpointing Disadvantages

1. Additionalmessages must be exchanged to coordinate checkpointing.

2. Synchronization delays are introduced during normal operations.

3. No computational messages can be sent while the checkpointing algorithm is in

progress.

4. II failure rarely occurs between successive checkpoints, then the checkpoint

algorithm places-an unnecessary extra load on the system,which can significantly

affect performance.

4.9.2 The Rollback Recovery Algorithm

• Restore the system state to a consistent state after a failure with assumptions:
single initiator, checkpoint and rollback recovery algorithms are not invoked

concurrently.

Phase One :

Process P checks whether all processes are willing to restart from their previous

checkpoints.

• A process may reply "no" if it is already participating in a checkpointing or

recovering process initiated by some other process.

• If all processes are willing to restart from their previous checkpoints, P, decides

that they should restart.

• Otherwise, P; decides that all the processes continue with their normal activities.

Phase Two :
P, propagates its decision to all processes.

. On receiving P{s decision, the processes act accordingly.

Optimization
6A minimum number of processes roll back.

. y will restart from its permanent checkpoint only if X is rolling back to a state

where the sending of one or more messages from X to Y is being undone.

TECHNICAL PUBLICATIONS.-an up-thrust for knowledge

 4- 15 Consensus and Recovery
Computing
Distributed

shows the unnecessary rollback.

.Fig. 4.9.6

Time
X
Failure
m

Fig. 4.9.6 Unnecessary rollback

4.9.3 Message Types

• In-transit message :Messages that have been sent but not yet received

done but "receive" is undone due to

• Lost messages : Messages whose "send" is

rollback

not recorded because the

Delayed messages : Messages whose "receive" is

the message arrived after rollback

receiving process was either down or
with "receive" recorded but message "send" not
Orphan messages: Messages
back to a consistent global state
recorded - do not arise if processes roll

Duplicate messages : Arise due to message logging and replaying during process

recovery.

AU : May-22, Dec.-22
4.10 lssues in Failure Recovery

Recovery refers to restoring a system to its normal operational state. Once a failure
has occurred, it is essential that the process where the failure happened can
recover to a correct state.

Fundamental to fault tolerance is the recovery from an error.

Resources are allocated to executing a computer. For example : a

processes in
process has memory allocated to it and a process may have locked shared
resources, such as files and memory.

• Following are some solution on process recovery :

1. Reclaim resources allocated to process

2. Undo modification made to databases and

3. Restart the process

4. Or restart process from point of failure and resume execution

TECHNICAL PUBLICATIONS - an up-thrust for knowledge

Distributed Computing 4- 16
Consensus and |Recovery

process wiL
•In distributed process recovery, undo effect of interactions oft tailed

other cooperating processes.

4.10.1 Basic Concept

System is combination of hardware and software components. These components

provide a specified service.

Failure of a system occurs when the system does not perform its service in the
manner specified.

An erroneous state of the system is a state which could lead to a system failure by
a sequence of valid state transitions.

•A system is said to "fail" when it cannot meet its promises. A failure is brought
about by the existence of "errors" in the system.
• A system is said to have a failure if the service it delivers to the user deviates
from compliance with the system specification for a specified period of time.
• Fig. 4.10.1 shows concept of fault and recovery.

Fault

Causes

Erroneous state error

Recovery

Leads to

Failure
Valid state

Fig. 4.10.1 Concept of recovery

System failure : System does not

meet requirements, ie.
services as specified. does not perform its

Erroneous systenm state :

State which could

. of valid state transitions.

:
Error the part of the system
lead

state which differs

to

from
a system

its
failure

intended value.
by a sequence

Faüt: Anomalous physical

condition, e.g.
damage, external disturbances, design errors,
manufacturing problems,

TECHNICAL PUBLICATIONS
an up-thrust for
knowledge
 4-17 Consensus and Recovery
Computing
Astrbuted

Questions
University

1 Discuss the issues in failure recovery with an example. AU:May-22, Marks 13

2 llustrate the different types of failures m distributed systems and explain how to prevent

AU:Dec.-22, Marks 13
them.

A11 Checkpoint-based Recovery

.The basic idea behind checkpoint-recoveris the saving and

restoration of system

code
periodically or before critical
state. By saving the current state of the system
of lost
sections, it provides the baseline information needed for the restoration

state in the event of a system failure.

of checkpoint-recovery can be high, by using techniques like

While the cost
system to have as small a critical state as
memory exclusion, and by designing a

to be useful in even cost

may minimize the cost of checkpointing enough
possible
sensitive embedded applications.

of the entire system is saved to

checkpointed, the state
When a system is

non-volatile storage.
system state and stores the
mechanism takes a snapshot of the
The checkpointing
on some non-volatile storage medium.
data
amount of state required to be
the
Clearly,the cost of a checkpointwill vary with
mechanism being used to save
to the storage
saved and the bandwidth available

the state.
internal state of the system can be restored,
• In the event of a system failure, the
at which its state was last saved.
and it can continue service from the point
this involves restarting the failed task or system, and providing some
Typically
that there is state to be recovered.
parameter indicating
and the bandwidth to the
• Depending on the task complexity, the amount of state,

this process could take from a fraction of a second to many

storage device
seconds.

against the transient fault model. Typically

This technique provides protection
continue processing in an identical manner
uDon state restoration the system will

as it did previously.

This will tolerate any transient fault, however if the fault was caused by a design
then the system will continue to fail and recover endlessly. In some cases,
error,

this may be the most important type of fault to guard against, but not in every
case.

TECHNICAL PUBLICATIONs an up-thrust for knowledge

 Consensus and
Distrnbuted Computing
4-18 Recovery

1.Uncoordinated Checkpointing
when to take checkpoints
• Each process has autonomy in deciding

havantages :The lower runtime overhead

during normal execution

Disadvantages :
a. Domino effect during a recovery
to find .
slow because processes need
to iterate
D, Recovery from a failure is

consistent set of checkpoints.

C. checkpoints and periodically invoke a

Each process maintains multiple

garbage collection algorithm

d. Not suitable for application with frequent output commits

dependencies among their checkpoints caused by

• The processes record the

message exchange during failure-free operation.

• In order to determine a consistent global checkpoint during recovery,the processes

record the dependencies among their checkpoints caused by message exchange

during failure free operation.

Direct dependency tracking technique

Assume each process P, starts its execution with an initial checkpoint C;0

• Ij, is a checkpoint interval and it is an interval between Ci,x-1 and Ci,x

When P receives a message m during I,y it records the dependency from Ij,, to

Iy which is later saved onto stable storage when P takes C;,y:

• When failure occurs, the recovering process initiátes rollback by broad casting a

dependency request message to colect all the dependency information maintained

by each process.

2. Coordinated Checkpointing

Coordinated checkpointing simplifies failure recovery and eliminates domino

effects in case of failures by preserving a consistent global checkpoint on stable

storage.

However, the approach suffers from high overhead associated with the

checkpointing process.

Two approaches are used to reduce the overhead: first is to minimizethe number
of synchronization messages and the number of checkpoints, the other is to make
the checkpointing process nonblocking.

TECHNICAL PUBLICATIONS an up-thrust for knowledge

 4-19 Consensus and Recovery
Computing
Dstributed

Blocking
Checkpointing
remains
After a process takes a local checkpoint,to prevent orphan messages, it

blocked until the entire checkpointing activity s complete,

, Fig. 4.11.1 shows blocking checkpointing.

X
X

Fig. 4.11.1 Blocking checkpointing

engages a protocol to coordinate with other

• Whern a process takes a checkpoint, it

processes to take checkpoint

takes a checkpoint; broadcasts a message to all processes.
a) Coordinator

halts execution; takes tentative checkpoint.

b) Process receives this message and

) Coordinator receives acknowledgement from all processes; broadcasts commit

message to end protocol.

commit message, removes old permanent checkpoint and

d) Process receives

makes tentative checkpoint permanent.

e) Processes resume execution.

• Disadvantages :The computation is blocked during the checkpointing.

Non-blocking Checkpointing
need not stop their execution while taking checkpoints.
The processes

Key issue with coordinated checkpointing: Being able to prevent a process from
receiving application messages that could make the checkpoint inconsistent.

Problem can be avoided by preceding the first post-checkpointmessage on each

channel by a checkpoint request, forcingeach process to take a checkpoint upon

receiving the first checkpoint-request message.

• A fundamental problem in coordinated checkpointingis to prevent a process from

receiving application messages that could make the checkpoint inconsistent.

TECHNICALPUBLICATIONS - an up-thrust for knowledge

 Consensus and Recovery
4- 20
Distributed Computing

• Fig. 4.11.2 shows non-blocking checkpoint. Initiator

Checkpoint request
Initiator
Initiator
request
Checkpoint
Checkpoint request
Po Cox m
Po Cox
Po Cox
PA
C1x
P1
P
C1x (c)

(b)
(a)
checkpoint
Fig. 4.11.2 Non-blocking

algorithm for coordinated

is the nonblocking
algorithm
• The Chandy-Lamport
checkpointing.

Example of Coordinated Checkpointing

"m" after receiving a

The process Po sent message

a) Checkpoint inconsistency:
Assume mnessage "m" reaches
the checkpoint coordinator.
checkpoint request from an inconsistent
This situation results in
checkpoint request.
process P, before the
message "m" from P, while
C1.x shows the receipt of
checkpoint since checkpoint
cheçkpoint Co.x does not show m
being sent from Po

b) Solution with FIFO channels :

If channels are FIF0, this problem can be avoided

the post-checkpoint message on each channel by a checkpoint

by preceding first

to take a checkpoint before

receiving the first
request, forcing each process

post-checkpoint message.

3. Communication-induced Checkpointing
The Communication-Induced Checkpointing (CIC) protocols are popular, because
they help in bounding rollback propagation during failure recovery, by ensuring

that each checkpoint taken is part of a consistent global checkpoint of the

distributed computation, while at the same time allowing each process to take

checkpoints independently.

Wavoids domino effect, while allowing processes to take some of their checkpoints
independently.
. Communication-induced checkpointing forces each process to take
based on information
checkpoints
piggybacked on the application messages it receives
from
other processes.
Checkpoints are taken such a system-wide consistent
that
state always exists on
stable storage, thereby avoiding the domino effect.

TECHNICAL PUBLICATIONS -an

up-thrust for knowledge
 4-21 Consensus and Recovery
Computing
nstibuted

and related
Communication-induced checkpointing piggybacks protocol

on each application message.

information

of each application message uses the piggybacked information to

• The receiver
has to take a forced checkpoint to advance the global recovery line.
determine if it

checkpoint must be taken before the application may process the

The forced

of the message. In contrast with coordinated checkpointing, no special

contents

coordination messages are exchanged.

are model-based
.Two types of communication-induced checkpointing

checkpointing and index-based checkpointing.

on preventing patterns of communications and
Model-based checkpointing relies

states among the existing checkpoints.

checkpoints that could result in inconsistent
could be forming
A model is set up to detect the possibility that such patterns

within the system, according to some heuristic.

works by assigning
Index-based communication-induced checkpointing

checkpoints, such that the checkpoints having

monotonicallyincreasing indexes to
the same index at different processes
form a consistent state

protocols do not take useless checkpoints.

Communication-Induced Checkpointing

Coordinated
4.11:1 Difference between Uncoordinated,
and Communication Induced Check Pointing

Coordinated check Communication

Parameters Uncoordinated,
induced check
check pointing pointing
pointing

One Many
Number of check Many
point

Possible No No
Domino effect

Possible No No
Orphan process

Unbounded Last global checkpoint Possible several

Rollback extent
checkpoints

Global coordination required Global coordination

Output commit Not possible
required

PUBLICATIONS - an up-thrust for knowledge

TECHNICAL
 Consensus and
Recovery
4-22
DistributedComputing

Algorithm
4.12 Coordinated Checkpointing to live-lock.
or
effect
to domino
• Uncoordinated checkpointing
may lead
coordination
: the system-wide
• Two basic approaches tocheckpoint a process to
initiate
has
which
1. The Koo-Toueg algorithm,

checkpointing process. checkpoints can help

in time; Staggering
checkpoints
which staggers
2. An algorithm
the disk system.
heavy loading of
avoid near-simultaneous

Koo-Toueg Algorithm : recovery technique

checkpointing and
1987 proposed a
coordinated Livelork
in effect and
Koo-Toueg and avoids domino
a consistent set of checkpointing
that takes

problems during the recovery,

qj was
P_3. This will record that
to establish a checkpoint at
Suppose P wants
orphaned, Q must checkpoint as well.

received from Q,to prevent q1 from being

Fig. 4.12.1 shows Koo-Toueg algorithm.

P_3
P_1 P_2

41

Fig. 4.12.1 Koo-Toueg algorithm

• Thus, establishing a checkpointat P_3 by P forces Q to take a checkpoint to record

that g 1 was sent

• An algorithm for such coordinated checkpointing has two types of checkpoints -

tentative and permanent
.P first records its current state in a tentative checkpoint,then sends a message to

all other processes from whom it has received a message since taking its last

checkpoint
Call the set of such processes II

. The message
received from
tells each process
it before
in II (e.g., Q), the last message, m gp, that P has
the tentative checkpoint was taken.

.Ifm_qp was not recorded in a checkpoint by Q : to prevent m_gp from beng

orphaned, Q is asked to take a tentative checkpoint to record sending m_qp.

TECHNICAL PUBLICATIONS® an up-thrust for knowledge

 4-23 Consensus and Recovery
Computing
Duted
in II, that need to, confirm taking a checkpoint as requested,then
,If all processes
checkpoints can be converted to permanernt.
all tentative
and all members
1f some members of I1, are unable to checkpoint as requested, P
thetentative checkpoints,and none are made permanent.
of II abandon

set off a chain reaction of checkpoints.

.This may
checkpoints among processes in
.Fach member of Ilcan potentially spawn a set of

set.
its corresponding

Asynchronous Checkpointing and Recovery

A13 Algorithm for

. Here
recovery.
we discuss, Juang-Venkatesan algorithm for asynchronous
checkpointing and

messages in FIFO
channels are reliable, delivery
Assumptions : communication
transmission delay is arbitrary but finite.
order, infinite buffers, message
checkpointing. During the
that is based on asynchronous
• They gave an algorithm
to which the system can
a consistent set of checkpoints
recovery, we need to find
be restored.
of both the number of
each process keeps track
• In this recovery algorithm
received from other processes.
messages it has send to and This
in this recovery.
by processes are also involved
iterations of rollback
Several
of Orphan messages.
algorithm avoids the existence
other processes to find if any
it is necessary for all
Whenever a process rollbacks,
back process has become an orphan message.
message send by the rolled
processor
if the number of messages received by
Orphan messages are discovered, P} to
P} is greater than number of messages sent by process
Pi from process
state of processes, then
one or more message
process Pi, according to the current

at process Pj are orphan messages.

of messages received are
• Then process must rollback to a state where number
Pj

equal to the number of messages sent by the process.

• Two type of log storage are maintained:

time to access but lost if processor crash. Move to stable log
a. Volatile log : short

periodically.
remained if crashed
b. Stable log : longer timne to access but

executing an event, the triplet is recorded

• Asynchronous checkpointing : After
processes. Local checkpoint consists of set
without any synchronizationwith other
in volatile log, then moved to stable log.
of records, first are stored

PUBLICATIONS - an up-thrust for knowledge

TECHNICAL
 Consensusand Recovery
4- 24
Distnbuted Computing

Recovery algorithm : of computation

from the beginning
• Number of messages received by p, from p,,

to checkpoint of computation to

from the beginning

• Number of messages sent by p; to Pj'

checkpoint
checkpoints. Doing
that
find a set of consistent
the set of checkpoints,
• ldea : From
received.
based on the number of messages sent and

Answers
4.14 Two Marks Questions with
AU :Dec.-22

Q.1 State the use of Rollback recovery.

after a failure.
a consistent state
Ans. • Restore the system back to
:
process during the
by saving the state of a
periodically
• Achieve fault tolerance
failure-free execution.
of processes that
application as a collection
a distributed system
Treats
communicate over a network. AU :Dec-22
Q.2 What is consensus in distributed system ?
the correct processes
must agree on a
has an initial value and all
Ans.: Each process

single value.
AU: May-22

Q.3 Write the purpose of using checkpoints.

fault tolerance toapplications.
most typically used to provide
Ans. : Checkpointing is but also for program
are useful.not only for availability,
Checkpointing techniques
migration, and load balancing.
debugging, process
distributed system ?
What do you mean by agreement problem
in
Q.4
AU : May-22

Ans. :In the agreement problem, to achieve overall system

has the
reliability

initial value.
in the presence of

a number of faulty processes and single process

What is the difference between agreement and consensus problem ?

Q.5

Ans. :The difference between the agreement problem and the consensus problem
value, whereas in
is

the
that, in the agreement problem, a single process has the initial
have an initial value.
consensus problem, all processes

Q.6 Define recovery.

state. Once a
Ans. : Recovery refers to restoring a system to its normal operational
failure has occurred, it is essential that the process where the failure happened can
recover to a correct state. Fundamental to fault tolerance is the recovery from an error.

TECHNICAL PUBLICATIONS - an up-thrust for knowledge

 4- 25 Consensus and Recovery
Computing
nstibuted

List classification
of failures.

Q.7

Ans.
: Failures
in

failure
a computer system

2. System
can be classified

failure
as follows :
1. Process

4. Communication medium failure

a Secondary storage failure

Define domino effect.

Q8
Ans. :The process of a cascaded rollback may lead to what is called the domino effect.

What is orphan process ?

Q.9
another process, but
Ans. :An orphan process is a process

with the crashed

that survives

process after
the crash
its
of

recovery.
whose state is inconsistent

Q.10 Explain two types of checkpoints.

that is made a permanent checkpoint on

Ans. : 1. Tentative : A temporary checkpoint

of the checkpoint algorithm.

the successful termination

2. Permanent : A local checkpoint at a process.

Q.11 List drawback of synchronous check pointing.

coordinate check pointing.

to
Ans. : 1. Additional messages must be exchanged
delays are introduced during normal operations.
2. Synchronization
the check pointing algorithm is
3. No computational messages can be sent while
in progress.
successive checkpoints, then the checkpoint
failure rarely occurs
between can
4. If
extra load on the system, which
algorithm places an unnecessary

significantly affect performance.

versions are helpful in recovery

?
Q.12 How shadow
objects in a file

Ans. :Shadow version uses a map

associates the identifiers
The map
to locate versions
of the server's objects
with
of the server's
the
called a version store. by each
store. The versions written
of their current versions in the version
positions status
versions. The transaction
of the previous- committed
transaction are 'shadows' a new
a transaction commits,
are stored separately. When
entries and intentions lists the shadow
map and entering the positions of
map is made by copying the old the old map.
process, the new map replaces
versions. To complete the commit

failure. What are different approaches to fault-tolerance ?

Q.13 Define fault and

Ans. :Fault: Anomalous

disturbances
physical condition, e.g. design errors, manufacturing

problems, damage, external

service in the manner
a system occurs when the system does not perform its
Failure of

specified.

PUBLICATIONs - an up-thrust for knowledge

TECHNICAL
 Consensus and Recoven
DistributedComputing 4-26

Q.14 to hold for execution.

List the requirements of consensus algorithm
tor execution are
algorithm to hold
consensus
ns. he requirements of

Termination, Agreement and Integrity.

Q.15 agreement protocols ?

What are the performance aspects of
Ans. : Following metrics are used :

1. Time:No of rounds needed to reach an agreement.

messages exchanged to reach an agreement.

2. Message traffic: Number of

that needs to stored at processors

overhead: Amount of information
3. Storage
during execution of the protocol.

Q.16 What are the application of agreement algorithm ?

Ans. : Applicationsof agreement algorithms

Fault-tolerant clock synchronization.

physical clocks to synchronized.

• Distributed systems require

Physical clocks have drift problem.

protocols may help to reach a common clock value.

Agreement

Synchronizing distributed clocks :

• At any time, values of clocks of all non-faulty processes must be approximately

equal.

small bound on amount by which of a non-faulty process is

• There is a the clock

changed during re-synchronization.

Q.17 State Byzantineagreement problem.

Ans. :• In the Byzantine agreement problem, n processors communicate with each

other in order to reach an agreement on a binary value b. There are bad

processors

that may collaborate with each other in order to prevent

an admissible agreement. Each
to a certain extent the
processor has an initialbinary value. The agreement must reflect

majority among the initial value.

Q.18 What is local checkpoints ?
Ans. :A process may take a local check point anytime during the
to form
execution.

a global
The local

consistent
checkpoints of different processes are not coordinated

checkpoint.

Q.19 What is forced checkpoints ?

Ans. : To guard against the domino effect, a communication induced checkpoint

protocolpiggybacks protocol-speci?c information to application messages that processes

and occasionally is forced to take

exchange. Each processexamines the information a

to the protocol.
checkpoint according

TECHNICAL PUBLICATIONS an up-thrust for knowledge

 Computing 4-27
Distnbuteo
Consensus and Recovery

useless checkpoints.
Explain
Q20

Ans.
: A useless

state.
checkpoint
Useless
of a process
one that will never be part of a
is

checkpoints are not desirable becausethey do

global

Consistent not contribute to

of the system from failures, but they consume
the recovery resources and cause

performance
overhead

021 What is checkpoint intervals ?

Ans. : A checkpoint
checkpoints in the
interval

execution of a
is the

process.
sequence of events between two consecutive

0.22 Define orphan messages.

Ans. : Messages with receive recorded but message send not recorded are called the

orphan messages.

Q.23 Write down the goals to achieve an optimal assignment. AU :Dec.-16

Ans. : Goal to achieve an optimal assignment is finding minimum weight cutest. The
a

weight of a cutset is the sum of the weights of the edges in the cutset. This sums up
the execution and communication costs for that assignment.

Q.24 Define consistent cut. AU : May-17

}is there are no events e; and

Ans. : A cut C=C1,C, Cay ... consistent if for all sites

e, such that (e; --> e; )and (e; --> c;) and (e; -/-> c;)

Q.25 What is the basic idea behind task assignment approach ? AU:May-17

Ans. : Basic idea :

been up into pieces called tasks.
a. A process has already split

and the speed of each CPU

b. The amount of computation required by each task

are known.
c. The cost of processing each task on every node is known.
d. The IPC costs between every pair of tasks is known.
are known.
e. Precedencerelationships among the taks

f. Reassignment of tasks is not possible.

AU: May-18
Q.26 Mention some motivations for replication.

Ans. The motivationsfor replication include :

at clients and servers is by now
Performance enhancement : The caching of data
familiar as a means of performance
enhancement.

Increased availability : Users require services to be highly available.

: Highly available data is not necessarily strictly correct data. It may be

Fault tolerance

out of date.

TECHNICAL PUBLICATIONS- an up-thrust for knowledge