GlobData: A Platform for Supporting Multiple Consistency Modes

F. Muñoz, L. Irún, P. Galdámez, H. Decker, J. Bernabéu, J. Bataller, C. Bañuls

Instituto Tecnológico de Informática, Universidad Politécnica de Valencia, Spain
ffmunyoz,lirun,pgaldam,hendrik,josep,bataller,[email protected]
16th July 2002

ABSTRACT In this paper, we propose protocols which meet the con-

GlobData is a platform that provides an object-oriented sistency and availability requirements posed by GlobData.
view of wide-area-networked relational databases with Due to space limitations, we focus on their essential char-
replicated data for ensuring high availability. We dis- acteristics. More details are elaborated in [12]. Besides
cuss the embedding of protocols in GlobData, for main- an optimistic support of lazy replication, the most notable
taining the consistency of replications. The protocols are feature of the protocols is their ability to alternate online
able to alternate between three different modes of consis- between three different consistency modes. Transactions
tency. Modes can be changed on-line and per session, i.e., are allowed to proceed locally, being checked for consis-
GlobData supports different and changeable consistency tency violations only at commit time. When a consistency
modes in simultaneous sessions. violation is detected, the transaction is rolled back.
Section 2 introduces basic concepts and characteristics
KEY WORDS of the GlobData consistency policy. Section 3 outlines the
Distributed Databases, Consistency, Avaliability algorithm underlying each protocol. Section 4 describes
more details of one of the protocols. In section 5, we
conclude.
1 Introduction
In networked applications, data are often replicated for
enhancing their availability. On the other hand, main-
2 Common Concepts
taining the consistency of replications frequently needs Within GlobData, the database nodes communicate via
to lock data, thus conflicting with the requirement of local consistency agents. They can be viewed as media-
high availability. Traditional approaches for replicat- tors for every data access and each transaction performed
ing databases are based fast LANs [9, 10, 8, 5], using by the local sessions. Because of lazy replication, not all
network-intensive protocols. However, in applications nodes necessarily have the latest version of each data ob-
based on WANs (e.g., the internet), the network is a lim- ject. Thus, for each data object, each database node as-
ited resource, and the ensuing problems must be appropri- sumes one one of the following roles:
ately dealt with [11, 3].
GlobData [7] strives to solve the aforementioned prob-  Owner node: Initially, that role is assumed by the
lems, focusing on applications in which efficient recovery node where the data object has been created. It han-
from failures, availability and high volumes of data pro- dles access confirmation requests for the object; i.e.,
cessing are key requirements. It does so by defining a it allows or denies such requests when asked by the
specific software architecture for networked data replica- initiator of a session.
tions, together with an API and a choice of flexible con-
sistency modes for data access.  Synchronous nodes: These nodes maintain up-to-
 date replicas. For each object, the synchronous transplanted to one or several of the remaining alive nodes
nodes and their number are supposed to be precon- (if any) with replicas of the owned objects.
figured.
 Deferred nodes: They usually do not store up-to-date 3 An Outline of the Protocols
replicas, although they may do so, sometimes (e.g.,
when one of them has initiated the session which has Each of our protocols proceeds as follows: The consis-
caused the latest change of the object). tency manager of each node maintains several metadata
Our approach replicates data lazily. To each object, an tables, containing information about the versions of lo-
owner node is assigned, for controlling accesses to that cally stored objects.
object. However, control is leveraged only when a session When a session wants to read an object, the local con-
wants to commit. Then, the accessed objects’ owners are sistency manager checks its version. If outdated, it sends a
requested to check if the session has used the latest com- message to the owner, requesting the recent updates. The
mitted versions. If so, the owners validate the accesses, session is blocked until these updates have been applied.
and the session can commit. If any of the read objects is Only then, the read access can be executed. As opposed
outdated, the session is flagged for abortion. We call the to that, write accesses are executed without any check.
requests for these checks ”access confirmation requests”, For each access, the sets of accessed objects are entered
since the owners are asked to check only those objects that into the meta-data tables, for building the read- and write-
have been used in the given session. set (cf. sections 3.3, 4). When a session wants to com-
To cope with failures of nodes and partitioning faults, a mit, the local consistency manager retrieves the read- and
notification service is needed. It assigns static identifiers write-sets and sends an access confirmation request to the
to each database node in the network. The protocols need accessed objects’ owners including OID, version number
these identifiers for recording the object ownership and and access mode (read or write) of each object.
also each node’s role. When an owner receives one of the above messages, it
Since we use access confirmation request management, compares the object’s version number with its latest ver-
each session identifier (SID) must include the identifier of sion. If equal, the owner grants access and sets the SID to
the node that has initiated the session. Thus, the protocols be the access grant’s tenant. The grant is exclusive if the
know which granted access requests did belong to a faulty access mode was ”write”, and shared if it was ”read”. The
node, which is necessary to manage a node’s failure. grant persists until the session has propagated its updates
For each object, an object identifier (OID) identifies its to the synchronous nodes with replicas of the accessed
initial owner. When a node fails, ownership is transferred objects. Once that is done, the session initiator updates
to another node. Clearly, this change of ownership needs the version number of the write-accessed objects and then
to be noticed by all alive nodes. Besides OIDs, also a releases the access grants.
version number is associated to each object, for the pur- If a session holds an access grant, other sessions re-
pose of node recovery. A recovering database then only questing the same grant in a conflicting mode receive a
needs to send to one of its neighbors the last known ver- service denial. Then, the consistency manager which re-
sion number of each of its objects. The receiving node quested that grant aborts the corresponding session.
replies with all changes to be applied, for re-establishing
an up-to-date state. An object version number includes
3.1 Different Consistency Modes
the SID of the session that has written its last value. The
purpose of that is explained in sections 3.3 and 4. Different sessions may use different consistency modes,
When some network links break and only some mutu- which however may change for each transaction. In in-
ally isolated subgroups of nodes remain connected, access creasing order of restrictiveness, the three modes are:
to an object is granted only if the subgroup containing that
object contains more than 50% of alive nodes with up-  Plain: It only allows isolated read requests and guar-
todate replicas. The ownership roles of a faulty node are antees that all accessed objects respect a session-

2
 causal commit order. Thus, the accessed objects may each set of synchronous nodes. Propagation by ses-
not be up-to-date. sion entails that plain mode consistency is always
satisfied (which otherwise is not guaranteed).
 Checkout: This is a permissive variation of the trans-
action mode (see below). Isolation is not guaranteed. Depending on the consistency mode, the consistency
Thus, if several sessions read the same object, only manager takes the following steps when a session reads
one of them is allowed to promote its mode to write or commits.
mode. This may break serializability, thus blocking
the promoting transaction in a standard transactional  On read accesses:
setting, or entailing abortion of the transaction. For plain mode, the object version in the local
database need not be the latest one. But causal com-
 Transaction: The usual transaction properties of
mit consistency must be guaranteed. So, the have
atomicity, consistency, isolation and durability must
to be applied to a database only when all sessions
be provided.
preceding the commiting one have already been ap-
plied. So, plain mode is no problem for read accesses
In practice, when two sessions with different consis-
(though some care must be taken when commits us-
tency modes conflict, priority is usually given to the more
ing other modes were made).
restrictive mode.
For read access in checkout and transaction modes,
the local consistency manager has to ensure that the
3.2 Protocol Alternatives latest version exists, for avoiding a roll-back. If this
The basic protocol outline at the beginning of section 3 version is not present, it has to be requested from
admits various alternatives of implementation. They en- the object’s owner. database of the session initiator.
gender different kinds of consistency protocols. One of The owner then returns the requested version to the
them is described in the next section. The characteristics session initiator. If plain mode is granted, not only
that admit alternative choices are: the latest version is needed, but all versions between
the one in the initiator’s database and the latest one.
 Update multicast when a session commits: A pro- but also all other changes of all sessions that have
tocol may broadcast the session changes at commit caused these updates.
time, in which case all replicas are synchronous.  On commit time: There are no commits in plain con-
Thus, by properly observing the session commit or- sistency mode because only read access is allowed.
der, all consistency modes can be easily guaranteed.
In checkout mode, read accesses are treated as in
Replication is lazy if the updates are multicast only transaction mode, but the commit procedure differs.
to a preconfigured subset of replicas Since different If a session has read a version that is not the latest one
objects may reside in different sets of synchronous at commit time, it is not aborted. However, objects
nodes, special care has to be taken for guaranteeing in the write-set have to be checked at commit time:
all consistency modes for lazy replication. their version must be the latest one. If not, the ses-
sion is aborted. Thus, read objects may change while
 Per object or per session update propagation: For
the session is executing, but written objects must not
lazy replication, all updates must either be propa-
be overwritten by other concurrently committing ses-
gated per session to each node with a synchronous
sions.
replica of any updated object, or each update is prop-
agated per object. Propagation per object implies: In transaction mode, both read and written objects
if two objects have been modified but not reside in must be up to date at commit time. If a subsequent
the same set of synchronous nodes, then only the change has been made by a concurrent session, the
changes involving a single object are propagated to terminating session must be aborted.

3
3.3 Three Consistency Protocols node that has not received any update for a long period of
time, and some of the objects with a synchronous replica
We present three consistency protocols with different in that node have been modified in the same session. The
characteristics, as follows: sequence of steps needed to get a group of missing ses-
Full object broadcasting: This protocol implements sions is as follows:
immediate updates of all database replications, so it does
not use lazy replication. Thus, the write-set of a commit- 1. A request is sent to the owner, asking for the session
ted session is broadcast and applied to all database nodes that has made the latest change on that object, and
immediately. Of course, not all sessions are committed, also for all preceding sessions according to causal
since owners must grant the access confirmation to do commit order. Each node maintains a log of com-
so. These access permissions depend on the consistency mitted sessions until they have been applied to all
mode used. The protocol supports each of the three con- system nodes. Thereupon, the session is removed
sistency modes. Read accesses do not need any additional from the log.
action, i.e., they can be performed locally without further
ado. The request also carries the local (and out-of-date)
Simple object update: This protocol uses lazy repli- object version number for the requested object. So,
cation and object updates, instead of session updates. Al- the owner is able to build a graph of precedent ses-
though this protocol complies with all consistency modes, sions, by tracing the SIDs stored in the object ver-
it requires more effort in plain mode, since the way sions and scanning the logs.
updates are propagated in sessions using transaction or
2. The owner replies by returning the mentioned graph.
checkout mode does not entail the guarantees required for
Since the log maintains SIDs of object versions as
plain mode accesses.
well as read- and write-set of each session, the graph
Note that, at commit time, the updates are only prop-
can be built inductively. The root is the session hav-
agated to the preconfigured synchronous replicas of each
ing caused the last update. For any subsequent layer
modified object. Thus, the full effects of a session may
of the graph, only the read-sets of the current layer
possibly be not reflected at each node which has received
have to be inspected, and all sessions that appear in
an update message. That is, a node may have a syn-
the object versions of those read-set objects are in-
chronous replica of one of the objects involved, and a de-
cluded in the next layer. When a session does not
ferred replica of one of the other objects.
appear in the log, it can not be added to the graph,
When a read operation needs a more recent version than since all its changes have been applied in each node.
the one stored in the local database, only the latest version Note that the graph only maintains the SIDs of the
is requested (and obtained) from the owner. No other con- sessions, but neither read- nor write-sets.
tents need to be transferred.
Session set update: This protocol uses lazy replica- 3. The requester checks the received graph and scans it
tion and session updates. That is, when the updates are depth-first, starting at the leaves and removing from
transferred to other nodes, not only the object changes are the graph all sessions that have already been applied
transmitted to their synchronous replicas, but all session to the local database. When the scanning reaches a
updates (i.e., the session write-set) are transferred to each level where no session has been removed, it termi-
node which has a synchronous replica of at least one of nates. The resulting graph is returned to the node of
the changed objects. the object’s owner.
For supporting plain mode, an additional problem ap-
pears: before the effects of a session can be applied, all 4. The owner receives the graph and replies with the
sessions preceding it in causal order need to have already read- and write-sets of all sessions in the graph.
been applied to the same database. Sometimes, however, These data are stored in the requesting node upon
this may have not yet been done. For instance, this may reception, thus terminating the retrieval of precedent
happen when an object has a deferred replica in a given sessions.

4
4 More Protocol Details i.e., when each object in the write-set of S has a
lower version than the one in the local database.
This section gives additional details for the session set
update protocol. We assume TCP/IP reliability of mes-
T
Note that a session cannot be applied to the local
database when an object in its read-set has a higher
sage transport and the existence of a service which no-
version than the one in the local database (i.e., there
tifies alive nodes about failures and recoveries of others.
is a causally precedent session which is yet unknown
Each time a session commits, the whole set of session up-
dates is transferred to each node which has a synchronous
T
to the requesting node). For applying , the out-
ofdate object must first be updated.
replica for at least one of the updated objects. Thus, plain
mode reads can be locally completed without any further  When the session elimination process is completed,
message exchange. Each consistency manager proceeds the requesting node notifies the owner by requesting
as follows. the complete write-set (values and version numbers)
of the ”session” resulting from the compacted graph.
 Every node maintains a log containing each session
applied in its local database. A process for updating  The owner replies with a message containing the
all nodes is run asynchronously in each node. After write-sets of each session included in the request.
that is done for each node, the session can be elimi- That can be extracted from the meta-data tables,
nated from the logs. since these sessions have been locally applied by this
owner, which therefore knows about these write-sets.
 When a node detects an out-of-date object No (say)
in a read request, it locates the owner node of No Note that “plain” consistency mode is easily imple-
and sends a request message to it in order to update mented. It ensures that each update preserves causal con-
its object copy. This request contains the identifier sistency. In order to provide this functionality, a log of
and version number of No . every applied session for each node is needed. (Note that
this may imply a redundancy of logs.) A session is kept
 The owner receives the request and checks its meta- logged until an asynchronous process has checked that the
data for the set of causally dependent sessions, which session has been applied to each node.
are needed to update the requested object to the ver-
sion held in the owner’s local database. More pre-
cisely, the following is done. 5 Conclusion
The owner node checks the meta-data for the last ses-
T
sion that has modified the requested object o (say). Global data access is increasingly important to a grow-
In that session, other objects may have been read. ing number of networked applications. The lockings nec-
RT
Let ( o ) denote the read-set of o .T essary for maintaining replication consistency often ob-

o RT
struct the availability of data objects. In this paper, we
For each object i in ( o ), the node searches its log have proposed protocols to reconcile availability and con-
T o
for each session j which has i in its write-set (i.e., sistency, by a flexible choice among different consistency
T
each j which causally precedes o ). T modes of a protocol implementing lazy replication.
The node then constructs a graph of causal depen- The protocols are appropriate for applications which
T T T
dencies of o and each j . It starts from o and iter- predominantly work with “local” data, of which replica
ates through layers of causal dependencies. The iter- are needed to ensure high availability. The session set up-
ation ends when all the logged sessions with causal date protocol has been implemented within the GlobData
precedence have been included in the graph. project [7, 6], and the other two protocols are currently
being implemented. Performance measurements, as well
 The resulting graph is sent to the requesting node. as details about failure handling, role migration and node
The latter eliminates from the graph each session Srecovery, will be presented and discussed in a follow-up
that has already been applied to its local database, paper.

5
References [13] E. Rahm. Empirical performance evaluation of
concurrency and coherency control protocols for
[1] L. Alonso. Optimistic data object replication for database sharing systems. ACM TODS, 18(2):333–
mobile computing. Siemens ZT SE 2 (inter- 377, 1993.
nal manuscript), 1998. Extended abstract in 9th
IFIP/IEEE Workshop on Distributed Systems: Op- [14] O. Wolfson, S. Jajodia. Distributed algorithms for
erations and Management, 1998. dynamic replication of data. PODS’92, 149–163,
1992.
[2] P. Bernstein, V. Hadzilacos, N. Goodman. Con-
currency control and recovery in database systems.
Addison-Wesley, 1987.
[3] F. Ferrandina, T. Meyer, R. Zicari. Implementing
Lazy Database Updates for an Object Database Sys-
tem. 20th VLDB, 261–272, 1994.
[4] J. Gray, P. Helland, P. O’Neil, D. Shasha. The dan-
gers of replication and a solution. SIGMOD’96,
173–182, 1996.
[5] M. Herlihy. Apologizing versus asking permis-
sion: Optimistic concurrency control for abstract
data types. ACM TODS, 15(1):96–124, 1990.
[6] Instituto Tecnológico de Informática, GlobData web
site: http://globdata.iti.es, 2002.
[7] ITI, UPNA, FFCUL. COPLA programming inter-
face, GlobData Deliverable 04 (WP 01). IST-1999-
20997, 2001.
[8] S. Jajodia, D. Mutchler. Dynamic voting algo-
rithms for maintaining the consistency of a repli-
cated database. ACM TODS, 15(2):230–280, 1990.
[9] B. Kemme, G. Alonso. A suite of database replica-
tion protocols based on group communication prim-
itives. ICDCS’98, 156–163, 1998.
[10] N. Krishnakumar, A. J. Bernstein. Bounded igno-
rance: A technique for increasing concurrency in
a replicated system. ACM TODS, 19(4):586–625,
1994.
[11] R. Ladin, B. Liskov, L. Shrira, S. Ghemawat. Pro-
viding high availability using lazy replication. ACM
TOCS, 10(4):360–391, 1992.
[12] F. Muñoz, L. Irún, P. Galdámez, J. Bernabéu, J.
Bataller, C. Bañuls. Consistency Protocols for Glob-
Data. ITI, UPV, Tech. Report, 2002.