International Symposium on Parallel and Distributed Processing with Applications

Distributed Transactions for Semantic Web

Workflows
Overcoming the CAP Limitations on Virtual Organizations

Daniel J. Buehrer and Chun-Yao Wang

Institute of Computer Science and Information Engr.
National Chung Cheng University
Chiayi, Taiwan 621
[email protected]; [email protected]

Abstract—Short term and long term transactions will become to implement consistent systems for security, load-balancing
very important for both personal and cooperative work involving and failover, and consistent upper-level workflows based on
workflows that use ontologies of the Semantic Web. We describe cooperation of many users, such as updates to hierarchical
a way to get around the CAP theorem (the impossibility of command and control systems based on performance
simultaneous consistency, availability, and the possibility of measurements. Moreover, cloud computing must synchronize
network partitioning) and the worst-case infinite delay for any its copies and switch-overs for load balancing and failovers.
distributed decision-making.
In Section 2 we describe the CAP Theorem and worst-case
Keywords-CAP theorem, transactions, cooperative engineering, infinite delay of distributed decisions. We then discuss the
virtual synchrony assumptions of Cadabia’s model in Section 3. Sections 4 and
5 describe the short-term and long-term transactions. Section 6
I. INTRODUCTION compares Cadabia’s solution to other solutions. Section 7
describes how distributed workflows are used to implement
In 2000, Eric Brewer [1] made a conjecture which was later Web-based vertical organizations. Finally, in Section 8 we
proven by Nancy Lynch [2] in 2002 as the “CAP theorem”. summarize and give some conclusions.
This theorem states that in a distributed environment it is
impossible to simultaneously have consistency, availability,
and the possibility of network partitioning (e.g. due to mobile II. CAP THEOREM AND INABILITY TO MAKE DISTRIBUTED
devices or wide-area network overloading). Another theorem DECISIONS
is that any distributed decision can always be delayed for an
arbitrarily long time by having processors fail at inappropriate A. CAP Theorem
times. Distributed computations involve at least two theoretical
limitations. The CAP theorem states that it is impossible to
In this paper we describe the methodology which is being
simultaneously have consistency, availability, and the
used in the Cadabia middleware system. We first define the
possibility of network partitioning. That is, the possibility of
implementation of short-term transactions [3] with fixed
network partitioning will, in general, cause copies of data to
participants. These short-term transactions are then used to
either become inconsistent or unavailable. Many real-time
implement long-term transactions which guarantee that all
systems, such as airplane seat assignments or online
current participants will be notified of any changes to the data
acquisitions may prefer to show users inconsistent data rather
which they have viewed since the last checkpoint.
than have the system become unavailable. It is quite easy to
The use of virtual synchrony [4] for short-term transactions see how copies of data will become out-of-date in the presence
provides a stable platform for performing distributed of network failures.
computations. In particular, the long-term transactions provide
As mentioned in the article on “Virtual Synchrony” on
allow for changing memberships where all current members
Wikipedia [4], there are basically three approaches to dealing
are guaranteed of being notified of changes to any data that
with the CAP Theorem:
they have read since the last transaction checkpoint. The
participants are also guaranteed to be notified of changes in 1. One-copy serializability. Usually transactional
participants of the long-term transaction. It is then quite easy replication implementations use transaction logs which

978-0-7695-4190-7/10 $26.00 © 2010 IEEE 465

DOI 10.1109/ISPA.2010.17
 may cause inconsistencies if the failure occurs as the We assume that Web servers will be on a cluster of servers
logs (i.e. updates) are being transmitted. which is always running. It would be very difficult to
2. Virtual Synchrony – Extremely high performance and maintain consistency of backup copies where users are
wide use in existing systems. Usually hidden from constantly turning their machines on/off. The cluster will be
end-users because it is difficult to implement correctly. responsible for most tasks of backup and failover. The use of
a cluster will help get rid of most of the problems of network
3. Paxos – state machine – seldom used except by partitioning, since local area networks are fairly stable. We
Google’s “Chubby” locking service. plan to use OpenSSI [9], Single System Image for Linux
Actually, an extended version of virtual synchrony was clusters, where any machine can be used to redistribute the
already developed in 1996 for the Transis Project [6,7]. Like load balance or do failover by making appropriate system
our short-term transactions [3], they used multicasting to pass calls. The transactions will guarantee that all users and all
the “current” states of each process group to implement an processes see a common view of the “current state”.
extended version of virtual synchrony. We discuss the Cadabia is based on the ideas of the Semantic Web. It
comparison in more detail in Section 6. stores all data, including all data typing information, in the
How does virtual synchrony help to solve the CAP form of triples (i.e. object-attribute-value or object-relation-
problems? The application-layer broadcast mechanism is used objectSet). Every binary relation has an inverse relation, and
to reach other members of the transaction as soon as they short-term transactions are used to maintain their consistency.
become reachable. There are no ack’s or event-based replies to Relations can refer to objects on other machines via a URL,
wait for. The only “blocked wait” occurs after all nodes have class name, and object name. “PartOf” relations inherit the
pre-committed and have ack’ed recently within a given time owners from the surrounding objects, and they are “cascade
period. Although it is possible that one of the servers will go deleted” when the surrounding independent objects of a given
down at this time and generate an abort due to timeout, it is class are deleted. When objects are deleted, all of the inverse
unlikely. pointers to that object are also deleted within a short-term
transaction. Otherwise, if any of the inverse relations are
The main idea is that the state of each participant of the
transaction has an associated clock or counter, so that messages unreachable, the transaction fails. That is, no inconsistencies
from different partitions can be easily merged to get the most are ever allowed to enter into the Cadabia data.
recent state of all participants. Cadabia is now aimed at vertical workflows. The vertical
work flows contain common definitions of messages, events,
B. Possibility of Delaying Distributed Decisions Forever error conditions, and message queues that are necessary to
describe a complete system and its users and documents.
The workflows take the place of traditional programming
There is another theorem about distributed systems. It involving subroutine calls. It is fairly well-recognized that
states the impossibility of reaching a consensus in a distributed distributed applications are not well-suited for traditional
system which has a faulty process [8]. If that faulty process subroutines calls, which require the caller to block and wait
fails at just the times that are necessary for making a decision, until the subroutine finishes. Rather, a workflow is a better
then the decision can be postponed indefinitely. There is representation of the possible flow of control, using events,
really no way around this other than to assume that it won’t messages, and error conditions that are defined in a shared
happen. That is, we assume that it will be possible for all ontology. The workflows are also more flexible, allowing
members of a transaction to be reachable for long enough dynamic modification based on perceived problems or feature
periods for them to agree to commit or abort the transaction. additions. Of course, such updates must be coordinated so
that all members of a transaction are using the same version of
III. CADABIA’S ENVIRONMENT the ontology. The OWL ontology, for instance, represents the
relationships between the various versions of the ontology,
and which versions are compatible.
Cadabia is based on the idea of a “personal” Semantic Moreover, Cadabia keeps track of which users are using
Web. The ontology of class, attribute, relation definitions are which ontologies, so that all of the users who have
shared. However, each user has his own data stored on either downloaded a particular ontology can be located by following
his own machine or his own server(s) in either SQL tables or the binary “include” relation or its inverse “includedBy”
XML files. relation for ontologies. Cadabia queries can be used to find a
Cadabia was originally based on peer-to-peer networks, but suitable set of similar ontologies for queries based on shared
with the advent of Web-based computation, we have shifted objects of other users. That is, the query “environment” can
the focus of the project to include the possibility of Web be set by limiting the number of inclusion levels, the number
servers. These Web servers are assumed to be always of servers, the number of objects returned by queries, etc.
available. However, we would prefer to keep the servers This environment is also part of the query kernel, and it is like
“local”, so that users will still be able to control which servers a global setting for the kinds and sizes of answers that are
will be allowed to store backup copies of their non-shared desired.
data.

466
 IV. SHORT-TERM TRANSACTIONS changes in membership will alternate with notifications of
The key point is that all of Cadabia’s typing constraints, changes to the data that have been queried so far.
security constraints, and cloud descriptions are stored in a
form which can be queried in the same way as user-defined That is, one purpose of the long-term transactions is to
objects. Therefore, once there is a means of “locking” eliminate “dirty reads”. The long-term transaction will
attributes and binary relations, this can be used by both end guarantee that all “current participants” of the transaction will
applications and by kernel-level implementations, such as have been notified of changes to any attributes or relations
security and cloud kernel layers. That is, the short-term since the last checkpoint. Those participants will thus know
transactions provide read/write locks for attributes and which attributes or relations may have become out-of-date.
relations. These transactions can be aborted in case of The applications can then choose whether or not to re-query,
possible deadlock or timeout. The short-term transactions or simply to “gray out” those parts of the data in their user
have a fixed set of members. interface.
The short-term transactions are sufficient to implement Typically, a Web app may involve a flow of documents,
long-term transactions, where the participants can change. events, error messages, and control operations among various
Any current participant of the long-term transaction will be participants of a virtual organization. For example, a school
guaranteed that he will be notified of any changes to either the Web app involves teachers, students, and an IT manager. It
participants of the transaction or to the attributes or relations may also involve, however, the invocation of apps in other
which have been viewed by the transaction since the last clouds. For example, we have an app which uses Google’s
checkpoint. GWT to translate student addresses into X,Y coordinates for
The implementation of short-term transactions has been mapping out minimal routes for the school’s bus drivers.
described elsewhere [3], so we will not go into detail here. Although the app runs on the local school’s cluster, it is able
Basically, application-layer multicasting is used to transmit a to interact with other apps on other clouds via standard http
vector of clock times and current states, where all participants messages.
use a common finite-state automaton to describe their current The main idea of Cadabia’s long-term transactions is that
states. Any application-layer topology may be used, such as a once changes of membership are controlled, it is fairly easy to
tree of bi-directional rings. A timer is also used to send out get consistent views of the data for making group decisions.
messages in case all other messages are lost due to process or For example, all participants know how many participants are
network failures. Each participant merges the messages by currently online, and how many votes are needed for a simple
taking the values associated with the maximum vector clock majority. If participants join or leave the transaction during
values. Therefore, network partitioning and different arrival the voting process, the majority requirement can be
times will not cause inconsistencies in the definition of automatically re-adjusted. The long-term transaction basically
“current” states, and a partial order of state changes is guarantees that the results will not be committed unless all
sufficient to allow the participants to agree. Once all of the members of the transaction have been notified of all changes
participants have all pre-committed a transaction, they will all in membership as well as all changes in votes.
eventually discover this condition, and commit the transaction. Basically, virtual synchrony has been used by the short-
If desired, before committing the transaction, an extra round of term transactions to prevent undetected changes in
messages can make sure that all participants are still membership. These notifications can be mixed with other
reachable, or else the transaction can be aborted. It is assumed messages in one of several ways:
that any such crashed process will eventually recover, and be 1. Old messages from dead processes may sometimes be
able to check on the final abort/commit decision for that received, but there is a way to ignore them based on
transaction. When it recovers, after finding out the transaction something like vector time counts.
outcome, it will first process its transaction log to update or 2. Old messages from dead processes will be thrown
restore the database before accepting any new messages. away if their vector counts skip; i.e. if there were
missed messages.
3. “Extended virtual synchrony” will send all messages
V. LONG-TERM TRANSACTIONS to current participants only after sending notifications
The short-term transactions of Cadabia are used to of membership changes to all current members.
read/write lock attributes or binary relations that are being 4. “Safe mode”: Messages will be completely serialized
viewed/updated by Cadabia queries. This is then used to into a FIFO queue, with all messages in the queue to
implement various communications mechanisms where be received by all active participants. Again, changes
memberships are allowed to change. For example, all users in membership have priority, with all new members
and processes that share the queues of a workflow diagram can being guaranteed to receive all messages, including the
be guaranteed to have a consistent view. notifications of change in membership. A message
which is not acked cannot be delivered. Therefore, a
The basic purpose of our long-term transactions is to
short-term transaction is used to guarantee that all acks
enable cooperative decision-making and cooperative editing of are received, or else a new notification of change of
Cadabia data. As members join and leave the transaction, the
membership is sent before sending the message.
other members of the transaction will be notified. The

467
 VI. COMPARISON TO TRANSIS avoid future financial woes similar to those that occurred in
In terms of the Transis Project [6,7], four kinds of 2009.
communication may be used to make sure that all members are
informed of all updates. Consider, for example, an application VIII. SUMMARY AND CONCLUSIONS
which allows many users to decide on what each of them will
take to a picnic. The four Transis modes below take increasing In this paper we have described the methodology which is
amounts of time, and they correspond to the four modes being used in the Cadabia middleware system. As mentioned
described in the previous section. above, there are many middleware systems that are also based
FIFO mode - failed process’s message may or may not be on virtual synchrony and application-layer multicast
sent mechanisms, like Cadabia. Cadabia’s rather unique feature is
Causal mode - maintains causal order between messages; that the short-term transactions are based on non-changing
won’t send messages of failed processes. memberships. This is sufficient to provide consistency when
Agreed mode - messages are delivered in the same order at there are changes to any binary relation. By using stateless
all processes. This order is consistent with causal order. RESTful HTTP transactions, programmers can be guaranteed
that all members of the process group (i.e. workflow members)
Safe mode - delivers messages only if all other group
will see the “current” state of the database, since they will be
members' machines have all received the message. notified of any changes to attributes or binary relations that
they have viewed since the last checkpoint. This includes the
The communication mode is specified in the Transis send “current” members of the workflow, the “current” workflow
command. Different communication modes can be used branch conditions, the “current” list of users who are
concurrently. authorized to do something, or the “current” list of users who
The main difference with Transis is that Cadabia are online for some Web app group, etc.
implements the four solutions using stateless, RESTful HTTP Cadabia will also guarantee that even if the current Web
transactions, which either read/write single or multiple objects server or its cluster loses power, the service and its state will be
in the database, with no intermediate state stored in the rolled over to another cluster backup provider. The cluster
messages themselves. The guarantee of notification of changes provides the roll-over capabilities locally, while an ack-based
in transaction participants allows the processes involved to re- wide-area network transaction checkpoint is used between
adjust to such changes, as when voting or determining clusters.
authorizations of users.
REFERENCES
VII. EFFECT ON VIRTUAL ORGANIZATIONS
[1] Eric Brewer, Towards robust distributed systems. (Invited Talk).
Once an ontology for a virtual organization has been Principles of Distributed Computing, Portland, Oregon, July 2000.
defined, a flow diagram can also be defined for the [2] Nancy Linch and Gilber Seth. “Brewer’s conjecture and the feasibility of
participants, their apps, and the messages, events, and errors consistent, available, partition-tolerant web services.” ACM SIGACT
sent by those apps to each other. The model of the News, v. 33 issue 2, 2002, p. 51-59.
organization then becomes the organization. This allows easy [3] Chun-Yao Wang, Daniel J. Buehrer, “A Ring-Based Decentralized
experimentation with “better” organizations. Should the Collaborative Non-Blocking Atomic Commit Protocol”, The 2008
IEEE/WIC/ACM International Conference on Web Intelligence (WI'08),
chairman of the board be more than one person? How many Held in conjunction with International Conference on Intelligent Agent
people should be on the board? What should be their Technology (IAT’08), December 9-12, 2008, Sydney, Australia.
membership requirements? What are the processes used in the [4] http://en.wikipedia.org/wiki/Virtual_synchrony last modified on 25
acquisitions department to prevent kick-backs? Should more January 2010 at 11:07.
than one secretary read incoming/outgoing correspondence? [5] K. P. Birman., Reliable Distributed Systems Technology, Web Service
What is the best login mechanism for preventing attacks? and Application, Springer-Verlag, 2005, ISBN:0-387-21509-3.
[6] Danny Dolev and Dahlia Molki, “The Transis Approach to High-
Web 3.0 may be defined in terms of individuals and Availability Cluster Communication”, CACM Special Issue on Group
organizations having more control over their own private data Communications, Vol. 39, April, 1996.
while being able to use shared ontologies. That is, individuals [7] Transis Project Home Page http://www.cs.huji.ac.il/labs/transis/
and individual organizations can improve on existing [8] Michael Fischer, J., Nancy A. Lynch, and Michael S. Paterson,
organizational structures. Second-order Science [14] is “Impossibility of Distributed Consensus with One Faulty Process”, 2nd
sometimes defined as scientific models which model the ACM Symposium on Principles of Database Systems, March 1983. Also
in Journal of ACM, vol. 32, no. 2, pp.374-382, 1985.
process of modeling things in the real world, such as
organizations or scientific theories. Science 2.0 extends [9] OpenSSI (Single System Image) Clusters for Linux
http://openssi.org/cgi-bin/view?page=openssi.html
traditional science by including the effects of the observers on
[10] Stefan B¨ottcher, Le Gruenwald, Sebastian Obermeier, “A Failure
the model. That is, the models of reality are modified in order Tolerating Atomic Commit Protocol for Mobile Environments”, In
to better match some evaluation criteria. This theory can help Proceedings of the The 8th International Conference on Mobile Data
us learn how to build better organizational structures. In Management (MDM 2007), Mannheim, Germany, 2007.
particular, these financial regulatory structures might help us to [11] S. Frølund and R. Guerraoui, “E-Transactions: End-to-End Reliability
for Three-Tier Architectures,” IEEE Trans. Software Eng., vol. 28, no.
4, pp.378-395, 2002.

468
[12] Paolo Romano, Francesco Quaglia, “Providing e-Transaction [14] S2ES 2010 International Symposium on Science 2 and the Expansion of
Guarantees in Asynchronous Systems with Inaccurate Failure Science, http://www.iiis2010.org/wmsci/website/default.asp?vc=37
Detection”, Fifth IEEE International Symposium on Network Computing
and Applications (NCA'06), pp.155-162, 2006.
[13] C.K. Yeo, B.S. Lee and M.H. Er, A survey of application level multicast
techniques, Computer Communications 27 (2004), pp. 1547–1568.

469