Distributed Systems

Consistency and
Replication
Chapter 7

1
 Reasons for Replication

• Data are replicated to increase the reliability

of a system.
• Replication for performance:
– Scaling in numbers
– Scaling in geographical area
• Caveat:
– Gain in performance
– Cost of increased bandwidth for maintaining
replication
2
 More on Replication
• Replicas allows remote sites to continue working in
the event of local failures.
• It is also possible to protect against data corruption.
• Replicas allow data to reside close to where
it is used.
• Even a large number of replicated “local” systems
can improve performance: think of clusters.
• This directly supports the distributed systems goal
of enhanced scalability.

3
 Replication and Scalability
• Replication is a widely-used scalability technique: think of
Web clients and Web proxies.
• When systems scale, the first problems to surface are those
associated with performance – as the systems get bigger
(e.g., more users), they get often slower.
• Replicating the data and moving it closer to where it is
needed helps to solve this scalability problem.
• A problem remains: how to efficiently synchronize all of
the replicas created to solve the scalability issue?
• Dilemma: adding replicas improves scalability, but incurs
the (oftentimes considerable) overhead of keeping the
replicas up-to-date!!!
• As we shall see, the solution often results in a relaxation of
4 any consistency constraints.
 Replication and Consistency
• But if there are many replicas of the same thing,
how do we keep all of them up-to-date? How
do we keep the replicas consistent?
• Consistency can be achieved in a number of
ways. We will study a number of consistency
models, as well as protocols for implementing
the models.
• So, what’s the catch?
– It is not easy to keep all those replicas consistent.
5
 Data-centric Consistency Models

The general organization of a logical data store,

physically distributed and replicated across
6 multiple processes
 Continuous Consistency (1)

An example of keeping track of consistency

7 deviations [adapted from (Yu and Vahdat, 2002)]
 Continuous Consistency (2)

Choosing the appropriate granularity for a Conit.

8 (a) Two updates lead to update propagation.
 Continuous Consistency (3)

Choosing the appropriate granularity for a Conit.

9 (b) No update propagation is needed (yet).
 What is a Consistency Model?
• A “consistency model” is a CONTRACT
between a DS data-store and its processes.
• If the processes agree to the rules, the
data-store will perform properly and as
advertised.
• We start with Strict Consistency, which is
defined as:
– Any read on a data item ‘x’ returns a value
corresponding to the result of the most recent write
on ‘x’ (regardless of where the write occurred).
10
 Consistency Model Diagram Notation
• Wi(x)a – a write by process ‘i’ to item ‘x’ with
a value of ‘a’. That is, ‘x’ is set to ‘a’.
• (Note: The process is often shown as ‘Pi’).
• Ri(x)b – a read by process ‘i’ from item ‘x’
producing the value ‘b’. That is, reading ‘x’
returns ‘b’.
• Time moves from left to right in all diagrams.

11
 Strict Consistency

• Behavior of two processes, operating on same data item:

a) A strictly consistent data-store.
b) A data-store that is not strictly consistent.
• With Strict Consistency, all writes are instantaneously
visible to all processes and absolute global time order is
maintained throughout the DS. This is the consistency
model “Holy Grail” – not at all easy in the real world,
and all but impossible within a DS. So, other, less strict
12 (or “weaker”) models have been developed …
 Sequential Consistency (1)

• A weaker consistency model, which represents

a relaxation of the rules.
• It is also much easier (possible) to implement.
• Definition of “Sequential Consistency”:
– The result of any execution is the same as if the
(read and write) operations by all processes on the
data-store were executed in the same sequential
order and the operations of each individual process
appear in this sequence in the order specified by its
program.
13
 Sequential Consistency (2)

(a) A sequentially consistent data store.

14 (b) A data store that is not sequentially consistent.
 Sequential Consistency (3)

15 Three concurrently-executing processes

 Sequential Consistency (4)

Four valid execution sequences for these

processes. The vertical axis is time.
16
 Problem with Sequential Consistency

• With this consistency model, adjusting the

protocol to favor reads over writes (or vice-
versa) can have a devastating impact on
performance (refer to the textbook for the gory
details).
• For this reason, other weaker consistency
models have been proposed and developed.
• Again, a relaxation of the rules allows for these
weaker models to make sense.
17
 Causal Consistency

• This model distinguishes between events that

are “causally related” and those that are not.
• If event B is caused or influenced by an earlier
event A, then causal consistency requires that
every other process see event A, then event B.
• Operations that are not causally related are
said to be concurrent.

18
 Causal Consistency (1)

• For a data store to be considered causally

consistent, it is necessary that the store
obeys the following conditions:
– Writes that are potentially causally related
• must be seen by all processes in the same
order.
– Concurrent writes
• may be seen in a different order on different
machines.

19
 Causal Consistency (2)

(a) A violation of a causally-consistent store

20
 Causal Consistency (3)

(b) A correct sequence of events in

21 a causally-consistent store
 Causal Consistency (4)

This sequence is allowed with a

causally-consistent store, but not with a
sequentially consistent store
22
 Introducing Weak Consistency
• Not all applications need to see all writes, let
alone seeing them in the same order.
• This leads to “Weak Consistency” (which is
primarily designed to work with distributed
critical sections).
• This model introduces the notion of a
“synchronization variable”, which is used to
update all copies of the data-store.

23
 Weak Consistency Properties
• The three properties of Weak Consistency:
1. Accesses to synchronization variables associated
with a data-store are sequentially consistent.
2. No operation on a synchronization variable is
allowed to be performed until all previous writes
have been completed everywhere.
3. No read or write operation on data items are allowed
to be performed until all previous operations to
synchronization variables have been performed.

24
 Weak Consistency: What It Means
• So …
• By doing a sync., a process can force the just written
value out to all the other replicas.
• Also, by doing a sync., a process can be sure it’s
getting the most recently written value before it reads.
• In essence, the weak consistency models enforce
consistency on a group of operations, as opposed to
individual reads and writes (as is the case with strict,
sequential, causal and FIFO consistency).

25
 Grouping Operations
• Necessary criteria for correct synchronization:
• An acquire access of a synchronization variable, not allowed
to perform until all updates to guarded shared data have been
performed with respect to that process.
• Before exclusive mode access to synchronization variable by
process is allowed to perform with respect to that process, no
other process may hold synchronization variable, not even in
nonexclusive mode.
• After exclusive mode access to synchronization variable has
been performed, any other process’ next nonexclusive mode
access to that synchronization variable may not be performed
until it has performed with respect to that variable’s owner.
26
 Entry Consistency
• At an acquire, all remote changes to guarded data
must be brought up to date.
• Before a write to a data item, a process must ensure
that no other process is trying to write at same time.

• Locks associate with individual data items, as opposed

to the entire data-store. Note: P2’s read on ‘y’ returns
27 NIL as no locks have been requested.
 Client-Centric Consistency Models
• The previously studied consistency models concern
themselves with maintaining a consistent (globally
accessible) data-store in the presence of concurrent
read/write operations
• Another class of distributed data-store is that which is
characterized by the lack of simultaneous updates.
Here, the emphasis is more on maintaining a
consistent view of things for the individual client
process that is currently operating on the data-store.

28
 More Client-Centric Consistency
• How fast should updates (writes) be made
available to read-only processes?
– Think of most database systems: mainly read.
– Think of the DNS: write-write conflicts do no
occur, only read-write conflicts.
– Think of WWW: as with DNS, except that heavy
use of client-side caching is present: even the
return of stale pages is acceptable to most users.
• These systems all exhibit a high degree of
acceptable inconsistency … with the replicas
gradually becoming consistent over time.
29
 Toward Eventual Consistency
• The only requirement is that all replicas will
eventually be the same.
• All updates must be guaranteed to propagate to
all replicas … eventually!
• This works well if every client always updates
the same replica.
• Things are a little difficult if the clients are
mobile.

30
 Eventual Consistency

The principle of a mobile user accessing

31 different replicas of a distributed database
 Monotonic Reads (1)

• A data store is said to provide

monotonic-read consistency if the
following condition holds:
– If a process reads the value of a data item x
 any successive read operation on x by
that process will always return that same
value or a more recent value.

32
 Monotonic Reads (2)

The read operations performed by a single process

P at two different local copies of the same
data store.
33 (a) A monotonic-read consistent data store.
 Monotonic Reads (3)

The read operations performed by a single process

P at two different local copies of the same
data store.
(b) A data store that does not provide
34 monotonic reads.
 Monotonic Writes (1)

• In a monotonic-write consistent
store, the following condition holds:
– A write operation by a process on a
data item x
 is completed before any successive write
operation on x by the same process.

35
 Monotonic Writes (2)

The write operations performed by a single

process P at two different local copies of the
same data store.
36
(a) A monotonic-write consistent data store.
 Monotonic Writes (3)

The write operations performed by a single

process P at two different local copies of the
same data store. (b) A data store that does not
37 provide monotonic-write consistency.
 Read Your Writes (1)

• A data store is said to provide

read-your-writes consistency, if the
following condition holds:
– The effect of a write operation by a
process on data item x
 will always be seen by a successive read
operation on x by the same process.

38
 Read Your Writes (2)

(a) A data store that provides

39 read-your-writes consistency
 Read Your Writes (3)

40
(b) A data store that does not
 Writes Follow Reads (1)

• A data store is said to provide

writes-follow-reads consistency,
if the following holds:
– A write operation by a process
 on a data item x following a previous
read operation on x by the same process
is guaranteed to take place on the same
or a more recent value of x that was read.
41
 Writes Follow Reads (2)

42
(a) A writes-follow-reads consistent data store
 Writes Follow Reads (3)

(b) A data store that does not provide

43 writes-follow-reads consistency
 Replica-Server Placement

44 Choosing a proper cell size for server placement

 Content Replication and Placement
• Regardless of which consistency model is
chosen, we need to decide where, when and by
whom copies of the data-store are to be placed.

45
 Replica Placement Types
• There are three types of replica:
1. Permanent replicas: tend to be small in number,
organized as COWs (Clusters of Workstations) or
mirrored systems.
2. Server-initiated replicas: used to enhance
performance at the initiation of the owner of the
data-store. Typically used by web hosting
companies to geographically locate replicas close
to where they are needed most. (Often referred to
as “push caches”).
3. Client-initiated replicas: created as a result of client
requests – think of browser caches. Works well
assuming, of course, that the cached data does not
46 go stale too soon.
 Server-Initiated Replicas

47 Counting access requests from different clients

 Content Distribution
• When a client initiates an update to a distributed
data-store, what gets propagated?
• There are three possibilities:
1. Propagate notification of the update to the other
replicas – this is an “invalidation protocol” which
indicates that the replica’s data is no longer up-to-
date. Can work well when there’s many writes.
2. Transfer the data from one replica to another –
works well when there’s many reads.
3. Propagate the update to the other replicas – this is
“active replication”, and shifts the workload to each
48 of the replicas upon an “initial write”.
 Push vs. Pull Protocols
• Another design issue relates to whether or not the
updates are pushed or pulled?
1. Push-based/Server-based Approach: sent
“automatically” by server, the client does not request
the update. This approach is useful when a high
degree of consistency is needed. Often used between
permanent and server-initiated replicas.
2. Pull-based/Client-based Approach: used by client
caches (e.g., browsers), updates are requested by the
client from the server. No request, no update!
49
 Pull versus Push Protocols

A comparison between push-based and pull-based

protocols in the case of multiple-client, single-
server systems
50
 Hybrid Schemes

• “Leases” – a promise from a server to push

updates to a client for a period of time.
• Once the lease expires, the client reverts to a
pull-based approach (until another lease is
issued).
• Three kinds of leases:
– Age-based Leases
– Renewal-Frequency based Leases
– State-based Leases
51
 Epidemic Protocols
• This is an interesting class of protocols that can be
used to implement Eventual Consistency (note: these
protocols are used in Bayou).
• The main concern is the propagation of updates to all
the replicas in as few a number of messages as
possible.
• Of course, here we are spreading updates, not
diseases!
• With this “update propagation model”, the idea is to
“infect” as many replicas as quickly as possible.
52
 Epidemic Protocols: Terminology

• Infective replica: a server that holds an update

that can be spread to other replicas.
• Susceptible replica: a yet to be updated server.
• Removed replica: an updated server that will
not (or cannot) spread the update to any other
replicas.
• The trick is to get all susceptible servers to
either infective or removed states as quickly as
possible without leaving any replicas out.
53
 The Anti-Entropy Protocol

• Entropy: “a measure of the degradation or

disorganization of the universe”.
• Server P picks Q at random and exchanges
updates, using one of three approaches:
1. P only pushes its own updates to Q
2. P only pulls in new updates from Q
3. P and Q send updates to each other
• Sooner or later, all the servers in the system
will be infected (updated). Works well.
54
 The Gossiping Protocol
• This variant is referred to as “gossiping” or
“rumour spreading”, and works as follows:
1. P has just been updated for item ‘x’.
2. It immediately pushes the update of ‘x’ to Q.
3. If Q already knows about ‘x’, P becomes
disinterested in spreading any more updates
(rumours) and is removed.
4. Otherwise P gossips to another server, as does Q.
• This approach is good, but can be shown not
to guarantee the propagation of all updates to
55 all servers. Oh dear.
 The Best of Both Worlds
• A mix of anti-entropy and gossiping is regarded as the
best approach to rapidly infecting systems with
updates.
• However, what about removing data?
– Updates are easy, deletion is much, much harder!
– Under certain circumstances, after a deletion, an “old”
reference to the deleted item may appear at some replica and
cause the deleted item to be reactivated!
– One solution is to issue “Death Certificates” for data items –
these are a special type of update.
– Only problem remaining is the eventual removal of “old”
56 death certificates (with which timeouts can help).
 Consistency Protocols

• This is a specific implementation of a

consistency model.
• The most widely implemented models
are:
1. Sequential Consistency.
2. Weak Consistency (with sync variables).
3. Atomic Transactions.

57
 Primary-Based Protocols

• Each data item is associated with a “primary”

replica.
• The primary is responsible for coordinating
writes to the data item.
• There are two types of Primary-Based
Protocol:
1. Remote-Write
2. Local-Write

58
 Remote-Write Protocols

59 The principle of a primary-backup protocol

 The Bad and Good of Primary-Backup
• Bad: Performance!
• All of those writes can take a long time (especially
when a “blocking write protocol” is used).
• Using a non-blocking write protocol to handle the
updates can lead to fault tolerant problems (which is
our next topic).
• Good: The benefit of this scheme is, as the primary is
in control, all writes can be sent to each backup replica
IN THE SAME ORDER, making it easy to implement
sequential consistency.
60
 Local-Write Protocols
• In this protocol, a single copy of the data item
is still maintained.
• Upon a write, the data item gets transferred to
the replica that is writing.
• That is, the status of primary for a data item is
transferable.
• This is also called a “fully migrating approach”.

61
 Local-Write Protocols Example

Primary-based local-write protocol in which a single copy is

62 migrated between processes (prior to the read/write).
 Local-Write Issues
• The big question to be answered by any process
about to read from or write to the data item is:
– “Where is the data item right now?”
• It is possible to use some of the dynamic
naming technologies studied earlier in this
course, but scaling quickly becomes an issue.
• Processes can spend more time actually
locating a data item than using it!

63
 Local-Write Protocols

Primary-backup protocol in which the primary migrates to the

process wanting to perform an update, then updates the
64 backups. Consequently, reads are much more efficient.
 Replicated-Write Protocols
• With these protocols, writes can be
carried out at any replica.
• Another name might be: “Distributed-
Write Protocols”
• There are two types:
1. Active Replication
2. Majority Voting (Quorums)

65
 Active Replication
• A special process carries out the update
operations at each replica.
• Lamport’s timestamps can be used to achieve
total ordering, but this does not scale well
within distributed systems.
• An alternative/variation is to use a sequencer,
which is a process that assigns a unique ID# to
each update, which is then propagated to all
replicas.
• This can lead to another problem: replicated
66 invocations.
 Replicated Invocations: The Problem

The problem of replicated invocations – ‘B’ is a replicated object

(which itself calls ‘C’). When ‘A’ calls ‘B’, how do we ensure
67 ‘C’ isn’t invoked three times?
 Replicated Invocations: Solutions

a) Using a coordinator for ‘B’, which is responsible for forwarding an

invocation request from the replicated object to ‘C’.
b) Returning results from ‘C’ using the same idea: a coordinator is
responsible for returning the result to all ‘B’s. Note the single result
68 returned to ‘A’.
 Quorum-Based Protocols
• Clients must request and acquire permissions from
multiple replicas before either reading/writing a
replicated data item.
• Consider this example:
– A file is replicated within a distributed file system.
– To update a file, a process must get approval from a
majority of the replicas to perform a write. The replicas
need to agree to also perform the write.
– After the update, the file has a new version # associated
with it (and it is set at all the updated replicas).
– To read, a process contacts a majority of the replicas and
asks for the version # of the files. If the version # is the
same, then the file must be the most recent version, and
the read can proceed.
69
 Quorum Protocols: Generalisation

NR + N W > N

NW > N/2

70
 Quorum-Based Protocols

Three examples of the voting algorithm:

(a) A correct choice of read and write set.
(b) A choice that may lead to write-write
conflicts. (c) A correct choice, known as
71 ROWA (read one, write all).