1

Untangling Blockchain: A Data Processing View

of Blockchain Systems
Tien Tuan Anh Dinh, Rui Liu, Meihui Zhang, Member, IEEE, Gang Chen, Member, IEEE,
Beng Chin Ooi, Fellow, IEEE, and Ji Wang

Abstract—Blockchain technologies are gaining massive momentum in the last few years. Blockchains are distributed ledgers that
enable parties who do not fully trust each other to maintain a set of global states. The parties agree on the existence, values and
histories of the states. As the technology landscape is expanding rapidly, it is both important and challenging to have a firm grasp of
what the core technologies have to offer, especially with respect to their data processing capabilities. In this paper, we first survey the
arXiv:1708.05665v1 [cs.DB] 17 Aug 2017

state of the art, focusing on private blockchains (in which parties are authenticated). We analyze both in-production and research
systems in four dimensions: distributed ledger, cryptography, consensus protocol and smart contract. We then present BLOCKBENCH,
a benchmarking framework for understanding performance of private blockchains against data processing workloads. We conduct a
comprehensive evaluation of three major blockchain systems based on BLOCKBENCH, namely Ethereum, Parity and Hyperledger
Fabric. The results demonstrate several trade-offs in the design space, as well as big performance gaps between blockchain and
database systems. Drawing from design principles of database systems, we discuss several research directions for bringing blockchain
performance closer to the realm of databases.

1 I NTRODUCTION
Blockchain technologies are taking the world by storm, development of new blockchain platforms designed for
largely due to the success of Bitcoin [1]. A blockchain, private settings where participants are authenticated. Block-
also called distributed ledger, is essentially an append-only chain systems in such environments are called private (or
data structure maintained by a set of nodes which do not permissioned), as opposed to the early systems operating in
fully trust each other. Nodes in the blockchain agree on an public environments (or permissionless) where anyone can
ordered set of blocks, each containing multiple transactions, join and leave. Applications such as security trading and
thus the blockchain can be viewed as a log of ordered trans- settlement [6], asset and finance management [7], [8], bank-
actions. In the database context, blockchain can be viewed ing and insurance [9] are being built and evaluated. These
as a solution to distributed transaction management: nodes applications are currently supported by enterprise-grade
keep replicas of the data and agree on an execution order database systems like Oracle and MySQL, but blockchain
of transactions. However, traditional databases assume a has the potential to disrupt this status quo because it incurs
trusted environment and employ well known concurrency lower infrastructure and human costs [9]. In particular,
control techniques [2], [3], [4] to order transactions. Block- blockchain’s immutability and transparency help reduce
chain’s key property is that it assumes nodes behave in arbi- human errors and the need for manual intervention due
trary (or Byzantine) manner. Being able to tolerate Byzantine to conflicting data. Blockchain can help streamline business
failure by design, blockchain offers stronger security than processes by removing duplicate efforts in data governance.
incumbent database systems. Goldman Sachs estimated 6 billion saving in current cap-
In the original design, Bitcoin’s blockchain stores coins ital market [9], and J.P. Morgan forecast that blockchains
as the system states. For this application, Bitcoin nodes will start to replace currently redundant infrastructure by
implement a simple replicated state machine model which 2020 [8].
moves coins from one address to another. Since then, block-
chain has grown beyond crypto-currencies to support user- Amid the growing commercial and academic interest, a
defined states and Turing complete state machine mod- large number of blockchain systems have sprung up, each
els. For example, Ethereum [5] enables any decentralized, claiming some unique capabilities. Both private and public
replicated applications known as smart contracts. More im- sector are clamoring to adopt blockchains, but they face
portantly, interest from the industry has started to drive overwhelming choices. While challenging, it is important
to have a firm grasp on what the technology can and
cannot do. A quest for understanding blockchain must
• Tien Tuan Anh Dinh, Rui Liu, Ji Wang, Beng Chin Ooi are with the
Department of Computer Science, National University of Singapore, ultimately answer the following questions:
Singapore.
{
E-mail: dinhtta, liur, wangji, ooibc @comp.nus.edu.sg
• }
Meihui Zhang is with Singapore University of Technology and Design, 1) What is a blockchain? Specifically, what are its
Singapore. unique properties that benefit current and future
E-mail: meihui [email protected] applications?
• Gang Chen is with Zhejiang University.
E-mail: [email protected] 2) How do current blockchains differ from each other,
both qualitatively in the design and quantitatively
in their performance?
3) What are the current challenges? And what do future blockchains look like?
 2
To answer these questions, in this paper, we start by
distinguishing two major classes of blockchain systems,
namely public and private blockchains. We then explain
four key technical concepts by which current systems can
be categorized: distributed ledger, cryptography,
consensus protocol and smart contract. Next, we describe
BLOCK- BENCH [10], our benchmarking framework for
quantita- tively evaluate and compare private blockchains. block t block t+1
Using BLOCKBENCH, we conduct comprehensive PrevBlockNonce PrevBlockNonce
TransactionTransaction
evaluation of three major blockchains: Ethereum [5], Parity roothashroothash

[11] and Hy- perledger [12]. The results show that current
blockchains’ performance is limited, far below what a state-
of-the-art database system can offer. Finally, we draw from
Transaction list Transaction list
our experi- ence in building large-scale database systems
several design principles that can improve future
Fig. 1: Blockchain data structure. Transactions are packed
blockchains.
into blocks which are linked to previous blocks.
In summary, our contributions are:
1) We provide an in-depth survey of blockchain sys-
tems. We discuss state of the art, and categorize semantics. The key difference is the failure model under con-
current systems along four dimensions: distributed sideration. Current transactional, distributed databases [13],
ledger, cryptography, consensus protocol and [14] employ classic concurrency control techniques such as
smart contract. two-phase commit to ensure ACID. They can achieve high
2) We describe our benchmarking framework, performance, because of the simple crash failure model. In
BLOCKBENCH, that is designed for understanding contrast, the original blockchain design considers a much
performance of private blockchains against data hostile environment in which nodes exhibit Byzantine be-
processing workloads. havior. Under this model the overhead of concurrency con-
3) We present a comprehensive evaluation of Eth- trol is much higher [15].
ereum, Parity and Hyperledger. The results show At a high level, a blockchain system can be categorized
the limitation of blockchains as data processing as either public or private. In the former, any node can join
platforms. They identify several performance bot- and leave the system, thus the blockchain is fully decentral-
tlenecks, and therefore can serve as a baseline for ized, resembling a peer-to-peer system [16]. In the latter,
future blockchain research and development. the blockchain enforces strict membership. More
In the next section, we provide an overview of block- specifically, there is an access control mechanism to
chain systems, separating them into public and private determine who can join the system. As the result, every
settings. Section 3 explains the four building blocks which node is authenticated and its identity is known to the other
are used in Section 4 to categorize existing blockchains. nodes.
Section 5 describes BLOCKBENCH, followed by the evalu-
ation of three blockchains in Section 6. Section 7 discusses 2.1 Public Blockchain
a number of lessons learned from the performance study,
and how to bring design principles from databases to Bitcoin [1] is the most well known example of public
improve blockchains. Section 8 concludes. blockchains. In Bitcoin the states are digital coins (crypto-
currencies), and a transaction moves coins from one set of
addresses to another. Each node broadcasts a set of
2 B LOCKCHAINS : PRIVATE VS. P UBLIC transac- tions it wants to perform. Special nodes called
A typical blockchain system consists of multiple nodes miners collect transactions into blocks, check for their
which do not fully trust each other. Some nodes exhibit validity, and start a consensus protocol to append the
Byzantine behavior, but the majority is honest. Together, blocks onto the block- chain. Bitcoin uses proof-of-work
the nodes maintain a set of shared, global states and (PoW) for consensus: only a miner which has successfully
perform transactions modifying the states. Blockchain is a solved a computationally hard puzzle (finding the right
special data structure which stores historical states and nonce for the block header) can append to the blockchain.
transac- tions. All nodes in the system agree on the PoW is tolerant of Byzantine failure, but it is probabilistic
transactions and their order. Figure 1 shows the blockchain in nature: it is possible that two blocks are appended at the
data structure, in which each block is linked to its same time, creating a fork in the blockchain. Bitcoin
predecessor via a cryptographic pointer, all the way back resolves this by only considering a block as confirmed after
to the first (genesis) block. Because of this, blockchain is it is followed by a number of blocks (typically six blocks).
often referred to as a distributed ledger. This probabilistic guarantee leads to security and
A transaction in a blockchain is the same as in performance issues: attacks have been demonstrated by an
traditional adversary controlling only 25% of the nodes [17], and
databases: a sequence of operations applied on some Bitcoin transaction throughput remains very low (7
states. As such, the blockchain transaction requires the transactions per second [18]).
same ACID Most public blockchain systems employ variants of PoW
for consensus. PoW works well in the public settings be-
cause it guards against Sybil attacks [16]. However, being
 3
non-deterministic and computationally expensive, it is un-
suitable for applications such as banking and finance which the other hand, a general-purpose blockchain may use low-
must handle large volumes of transactions in a level model such as table or key-value. Second, the system
deterministic manner. may have one or more ledgers which may be connected
to each other. A large enterprise, for example, may own
multiple ledgers, one for each of its departments: engi-
2.2 Private Blockchain neering, customer care, supply chain, payroll, etc.. Third,
Hyperledger [12] is among the most popular private ledger ownership may vary from completely open to public
blockchains. Since node identities are known in the pri- to strictly controlled by one party. Bitcoin, for example is
vate settings, most blockchains adopt one of the protocols completely open, and as a consequence requires expensive
from the vast literature on distributed consensus. Zab [19], consensus protocol to identify who can update the ledger.
Raft [20], Paxos [21], PBFT [15] are popular protocols that Parity [11], on the other hand, pre-determines a set of
are in active use today. Hyperledger directly uses PBFT1, owners who can write to the ledger simply by singing the
while others like Parity [11], Ripple [6] and ErisDB [22] blocks.
develop their own variants. PBFT is a three-phase protocol.
In the pre-prepare phase, a leader broadcast a value to be 3.2 Consensus
commit by other nodes. Next, in the prepare phase, the
nodes broadcast the values they are about to commit. The content of the ledger reflects historical and current
Finally, the commit phase confirms the committed value states maintained by the blockchain. Being replicated, up-
when more than two third of the nodes agree in the dates to the ledger must be agreed on by all parties. In
previous phase. PBFT is communication bound, but it other words, multiple parties must come to a consensus.
achieves both safety and liveness in partially synchronous Note that this is not the case in many real-world
networks. Besides deterministic consensus, another key applications such as fiat currency, in which one entity (e.g.
property of private blockchains is that they support smart the bank or the government) decides the updates.
contracts which can express highly complex transaction One key property of a blockchain system is that the
logics. These properties are particularly desirable in nodes do not trust each other, meaning that some may
business and financial systems. Indeed, private blockchains behave in Byzantine manners. The consensus protocol
evoke such interest from major banking and financial must therefore tolerate Byzantine failures. The research
institutions that some even claim that they have the litera- ture on distributed consensus is vast, and there are
potentials to disrupt current practices in data management many variants of previously proposed protocols being
[8], [9]. developed for blockchains [23]. They can be largely
classified along a spectrum. One extreme consists of
purely computation based protocols that use proof of
3 K EY C ONCEPTS computation to randomly select a node which single-
handedly decides the next op- eration. Bitcoin’s proof-of-
Categorizing blockchains as public or private is useful
work (PoW) is an example. The other extreme are purely
for identifying major characteristics of many blockchains.
communication based protocols in which nodes have equal
However, understanding their subtle differences warrants
votes and go through multi- ple rounds of communication
a finer taxonomy. This section introduces four
to reach consensus. These protocols, PBFT [15] being the
underpinning concepts, based on which a more detailed
prime example, are used in private settings because they
classification of the systems can be obtained.
assume authenticated nodes. In between these extremes
are hybrid protocols which aim to improve performance of
3.1 Distributed Ledger PoW and PBFT. For instance, Proof-of-Elapsed-Time
A ledger is a data structure that consists of an ordered list (PoET) eliminates expensive mining in PoW by leveraging
of transactions. For example, a ledger may record monetary trusted hardware such as Intel SGX. Another example is
transactions between multiple banks, or goods exchanged Proof-of-Authority (PoA) [24] which improves PBFT by
among known parties. In blockchains, the ledger is repli- pre-selecting a small set of trusted nodes that vote among
cated over all the nodes. Furthermore, transactions are themselves to reach consensus. Similarly, Stellar [25] and
grouped into blocks which are then chained together. Thus, Ripple [6] improve PBFT by executing consensus in smaller
the distributed ledger is essentially a replicated append-only networks.
data structure. A blockchain starts with some initial states,
and the ledger records entire history of update operations 3.3 Cryptography
made to the states.
A system supporting distributed ledgers can be char- Blockchain systems make heavy use of cryptographic tech-
acterized in three dimensions, as illustrated in Table 1. niques to ensure integrity of the ledgers. Integrity here
First, the application built on top of the ledger determines refers to the ability to detect tampering of the blockchain
the data model of what being stored in the ledger. For data. This property is vital in public settings where there
example, a crypto-currency application may adopt the is no pre-established trust. For example, public confidence
user- account model resembling traditional banking in crypto-currencies like Bitcoin, which determines values
systems. On of the currencies, is predicated upon the integrity of the
ledger; that is the ledger must be able to detect double
1. Hyperledger has two main releases: v0.6.0 and v1.0.0-rc1. The spending. Even in private blockchains, integrity is equally
former supports PBFT, but the latter adopts a no-Byzantine consensus essential because the authenticated nodes can still act
protocol based on Kafka.
maliciously.
 TABLE 1: Examples of distributed ledgers.
Data Model Number of ledgers Owner Example
Accounts One Administrator Traditional ledgers used in financial institutions.
Private ledger used within a financial institu-
Assets Many Group of users tion, or between small groups of financial orga-
nizations, e.g. global financial services.
Coins or accounts One Any user Crypto-currencies like Bitcoin or Ethereum.

There are at least two levels of integrity protection. contract Doubler

First, the global states are protected by a hash (Merkle) { struct Partitipant
tree whose root hash is stored in a block. Any state {
address etherAddress;
change results in a new root hash. The tree’s leaves contain uint amount;
the states, the internal nodes contain the hashes of their }
children. For instance, Hyperledger v0.6 uses a bucket hash Partitipant[] public participants;
tree, in which states are grouped (by hashing) into a pre- uint public balance = 0;
...
defined number of buckets. Ethereum, on the other hand, function enter() {
employs a Patricia- Merkle tree which resembles a trie and ...
whose leaves are key-value states. Second, the block balance+= msg.value;
history is protected, that is the blocks are immutable once ...
if (balance >
they are appended to the blockchain. The key technique is 2*participants[payoutIdx].amount) {
to link the blocks through a chain of cryptographic hash transactionAmount = ...
participants[payoutIdx].
pointers: the content of block number n + 1 contains the
etherAddress.send(transactionAmount);
hash of block number n. This way, any modification in block ...
n immediately invalidates all the subsequent blocks. By }
combining Merkle tree and hash pointers, blockchain offers }
...
a secure and efficient data model that tracks all historical }
changes made to the global states.
Blockchain’s security model assumes the availability Fig. 2: An example of Ethereum smart contract, written in
of public key cryptography. Identities, including user and Solidity, which implements a pyramid scheme.
transaction identities, are derived from public key certifi-
cates. Secure key management, therefore, is essential to any
blockchains. As in other security systems, losing private
200 opcodes from which users can write stack-based scripts.
keys means losing access. But in blockchain applications
For example, the following script verifies if 2 out of 3 valid
such as crypto-currencies, losing the keys has direct and
signatures are available.
irrevocable financial impact. We discuss in Section 4.2 dif-
ferent schemes for key and identity management. OP_2 <Pub1> <Pub2> <Pub3> OP_3 OP_CHECKMULTSIG
There exist many research systems that extend the orig-
inal blockchain design with novel and complex crypto- At the other extreme, Ethereum smart contracts can specify
graphic protocols. They aim to improve security and per- arbitrary computations, i.e. they are Turing complete code.
formance with esoteric techniques such as zero-knowledge Figure 2 shows a snippet of a real smart contract running
proofs, group signatures and trusted hardware. We discuss on Ethereum. It implements a pyramid scheme: users send
money to this contract which then pays interests to early
them in greater detail in Section 4.2.
participants. The contract has its own states, namely the list
of participants, and exports a function called enter. A user
3.4 Smart Contracts
invokes the contract by sending his money through a trans-
A smart contract refers to the computation executed when action. When executed, the contract can access the input
a transaction is performed. It can be regarded as a stored address (user account) via msg.sender and the transaction
procedure invoked upon a transaction. The inputs, outputs value via msg.amount. It updates the accumulated balance,
and states affected by the smart contract execution are computes the interest for each participants. Finally, payment
agreed on by every node. is made by invoking etherAddress.send.
All blockchains have built-in smart contracts that im-
In between the two extremes are smart contract sys-
plement their transaction logics. In crypto-currencies, for
tems that offer more expressiveness than Bitcoin’s op-
example, the built-in smart contract first verifies transaction
codes, but they reject Turing-completeness. Kadena [26]
inputs by checking their signatures. Next, it verifies that
and BigchainDB [27] support contracts with complex, but
the balance of the output addresses matches that of the
constrained semantics so that they can be formally checked
inputs. Finally, it applies changes to the states. In the rest
for safety.
of the paper we do not refer to such built-in logics as smart
contracts. Instead, we only consider smart contracts that can Another way to categorize smart contract systems is by
be defined by users. their runtime environments. Most systems execute smart
One way to characterize a smart contract system is by contracts in the same runtime as the rest of the blockchain
its language. At one extreme, Bitcoin provides fewer than stack. We refer to them as employing native runtimes. For
example, Kadena parses contracts written in its Haskell-
like
language and executes them directly as Haskell programs.
Ethereum, on the other hand, comes with its own virtual IOTA issue their own assets (tokens) and offer their ledgers
machine for executing Ethereum bytecodes. Hyperledger, as a medium of exchange or a platform for micro-payment.
opting for portability, employs Docker containers to IOTA, in particular, allows for zero-fee micropayment via
execute its contracts. its tokens, which makes the ledger useful for exchanges
among IoT devices. The ledgers in these systems adopt
account- based data models. One ledger exists per system
4 STATE OF THE ART and it is open; that is anyone can buy tokens and take part
In this section we compare current blockchains using the in the exchanges.
four concepts discussed in Section 3. We explain their
design in more detail and highlight their subtle differences. General applications
We also discuss research problems that are being tackled. Going beyond crypto-currency and asset management,
A list of blockchains and their properties are shown some ledgers support running general, user-defined com-
in Table 2. Major systems are included, but we stress that putations (or smart contracts). Ethereum and its
the list is not exhaustive, especially given the growing derivatives, namely Hydrachain, Quorum, Monax, Parity
commercial and academic interest in blockchains. Systems and Dfinity let users write arbitrary business logics
shown in italics are either no longer in active development, executed on top of the ledger. For example, Ethereum
or are still in initial phases of development. For examples, contracts range from simple crowdfunding campaigns to
Hydrachain [34] codebase was last updated about 8 complex investment funds like the DAO [43]. Dfinity has a
months ago at the time of writing, and IOTA’s current special type of contract — the governance contract — that
codebase consists of only a reference implementation 2. The enforces real-world regulations on Ethereum-like
table has no column for cryptography, since all systems blockchains. Hyperledger and its close cousin Sawtooth
(except for ZCash) employ standard techniques described Lake likewise support running Turing- complete code. They
in Section 3. Novel cryptographic protocols that are not yet offer key-value data model, with which the applications can
integrated are discussed in Section 4.2. create and update key-value tuples on the blockchain.
Kadena and Tezos restrict how powerful the applications
4.1 Distributed Ledger can be by devising their own languages which are not
Turing complete but can be formally verified. Tezos data
Recall that a system supporting distributed ledgers is model is account-based, whereas Kadena’s is based on
characterized by its target applications, by the number of
table. In particular, Kadena applications operate on key-
ledgers, and by the ledger ownership. In the following,
we group the systems listed in Table 2 by their target row structures with schemas, versions and column history.
applications.
4.2 Cryptography
Crypto-currency Identity management
The most successful adoption of blockchain technology is A user in a blockchain is uniquely identified by her public
crypto-currency. In the wake of Bitcoin’s success, multiple key certificate. In public settings, the user first generates
competing currencies appear. Most of these alternative cur- a key pair (the default option being ECDSA based on the
rencies (or alt-coins) such as Litecoin or Dodgecoin, adopt Secp256k1 elliptic curve), then derives the identity as the
similar data models to Bitcoin’s. Ethereum, departs from hash of the public key. This hash serves as a transaction
Bitcoin’s transaction-based model and instead implements address or an account number in crypto-currencies systems.
an account-based model. The nature of currency applica- To claim ownership of the transaction output or of the
tions requires that the ledger must be open and the system account, the user signs transactions with the corresponding
maintains only one ledger. private key. In private settings, there is an additional access
control layer. Hyperledger separates this layer from the
Digital assets
blockchain, in the form of a membership provider service
Crypto-currency is one instance of digital assets — pieces and a certificate authority service. The administrator can
of data with attached real-world values. Unlike crypto- implement arbitrary policies with these services to control
currencies which are created on and derive their values who gets access to the blockchain. Signed requests sent to
directly from the blockchains, digital assets are often Hyperledger are first checked against these services before
issued by real world entities and blockchains are merely a processed by the next (consensus) component. Multichain
medium to record their existence and exchanges. offers a simpler model with a fixed number of global per-
Multichain [32], BigchainDB and Corda offer ledgers for missions, while the remaining systems provide little detail
storing and tracking asset history. Like Bitcoin, their data of their protocols.
models are transaction- based which are centered around The problem of managing user keys is the same in
assets. These systems target private settings, in which private blockchains as in typical enterprise systems, thus
multiple organizations can spin up a network to trade existing solutions can be readily integrated. In public
assets among each other. The organizations are the ledger blockchains, however, the sheer scale and monetary
owners, and it is common to have more than one ledger impacts of losing private keys calls for more secure and
among them. Stellar, Ripple and more usable protocols. Bitcoin, in particular, embodies the
challenges, as Bitcoin users themselves are tasked with
2. As many other blockchain projects do, IOTA is raising fund for
development via token sale. managing large numbers of keys which are refreshed on
new transac- tions. Eskandari et al. [44] evaluated six
approaches for
 TABLE 2: Comparison of blockchain systems. Ones in italics are deemed inactive or at early phases of development.
Smart contract
Application Smart contract language Data model Consensus
execution
Hyperledger General
Dockers Golang, Java Key-value PBFT
v0.6.0 [28] applications
Hyperledger General Ordering service
Dockers Golang, Java Key-value
v1.0.0 [29] applications (Kafka)
Transaction-
Bitcoin Crypto-currency Native Golang, C++ PoW
based
Transaction-
Litecoin [30] Crypto-currency Native Golang, C++ PoW (memory)
based
Transaction-
ZCash [31] Crypto-currency Native C++ PoW (memory)
based
General
Ethereum [5] EVM Solidity, Serpent, LLL Account-based PoW
applications
Multichain Transaction- Trusted validators
Digital assets Native C++
[32] based (round robin)
General
Quorum [33] EVM Golang Account-based Raft
applications
HydraChain General Trusted validators
Python, EVM Solidity, Serpent, LLL Account-based
[34] applications (majority)
OpenChain
Digital assets - - Transaction-based Single validator
[35]
IOTA’s Tangle
IOTA [36] Digital assets - - Account-based
Consensus
BigchainDB Transaction Trusted validators
Digital assets Native Python, crypto-conditions
[27] based (majority)
General
Monax [22] EVM Solidity Account-based Tendermint [37]
applications
Ripple [6] Digital assets - - Account-based Ripple consensus
Kadena [26] Pact applications Native Pact Table ScalableBFT [38]
Stellar [25] Digital assets - - Account-based Stellar consensus
General
Dfinity [39] EVM Solidity, Serpent, LLL Account-based Threshold relay
applications
General Trusted validators
Parity [11] EVM Solidity, Serpent, LLL Account-based
applications (round robin)
Michaleson
Tezos [40] Native Michaleson Account-based Proof of Stake
applications
Transaction-
Corda [41] Digital assets JVM Kotlin, Java Raft
based
Sawtooth General
Native Python Key-value Proof of elapsed time
Lake [42] applications

Bitcoin key management (or wallet): local storage, pass-

term keys are derived. Before a piece of code is loaded,
word protected storage, offline storage, air-gapped storage,
the hardware measures it by hashing the code content and
password-derived keys, and hosted storage. The authors
signing it with one of the keys. The signed measurement
found that none of these approaches is satisfactorily
together with the key certificate attest to a remote party
usable, due to misuse of metaphors to traditional
what is being run in the local device. This protocol requires
currencies, and also due to confusing abstractions.
a certificate authority that maintains and endorses a list of
known certificates and a list of revoked certificates. Highly
Trusted hardware
complex attestation schemes, for example direct anonymous
Recent distributed systems are leveraging trusted attestation [48], offer hardware anonymity without a certifi-
hardware such as Intel SGX and ARM TrustZone to improve cate authority.
perfor- mance at a slight cost of security [45], [46].
Sawtooth Lake proposes proof of elapsed time (PoET) as an Transaction privacy
efficient re- placement for proof-of-work. TownCrier [47]
Most blockchains are designed to protect transaction in-
employs SGX to implement a trusted party for vetting
tegrity, but they do not consider transaction privacy. A
external contents and importing them to the blockchain.
blockchain is said to have transaction privacy when (1)
These systems are based on a trust model that is weaker
transactions cannot be linked from one to another, and (2)
than that of a purely cryptographic system. In particular,
the transaction content is known only to its participants.
their security is dependent upon a trusted computing base
In private settings, complete transparency of transaction
(TCB) that is running inside the trusted hardware. Smaller
history may not be a problem. Either transparency is desir-
TCBs mean better security.
able for the applications, such as financial auditing, or it is
All systems based on trusted hardware rely on remote
straightforward to add an access control layer to protect
attestation protocols. A key pair, called Endorsement Key
the blockchain data. In public settings, on the other hand,
(EK), is burnt into each device during manufacturing. Such
the need for transaction privacy is driven by two factors.
a key pair serves as the root of trust, from which other
First, deanonymization attacks have successfully recovered
short-
the
underlying structure of the Bitcoin network [49], and even ond, transaction linkability can undermine the currency’s fungibility,
linked Bitcoin addresses to real-world identities [50]. Sec- rendering some coins more valuable than others due to their histories.
 Zerocoin [51] is the first blockchain providing transac-
tion unlinkability. It extends Bitcoin to allow for trading Table 3 summarizes key properties of the major protocols
between bitcoins and special coins called zerocoins. Ze- which we now explore in detail.
rocoin essentially implements a cryptographic mixer that
hides linkability between zerocoins and the corresponding Proof of Work variants
bitcoins. Each zerocoin is a cryptographic commitment to All PoW protocols require miners to find solutions to
two random values (s, r). When redeeming a zerocoin, the crypto- graphic puzzles based on cryptographic hashes.
owner reveals s as the proof that the coin has not been Specifically, the solution is a random nonce n such that:
spent, and a zero-knowledge proof of r. Transaction
unlinkability is derived from the fact that the coin being H(n H(b)) < t
redeemed can be any of the many unspent zercoins.
Zerocash [31] extends Zerocoin by improving efficiency for a cryptographic hash function H , a threshold t and
of the latter’s cryptographic operations. It functions as a the current block content b. The original protocol imple-
stand-alone blockchain, as opposed to Zercoin being an ex- mented in Bitcoin uses SHA-256 as the hash function.
tension of Bitcoin. Transactions in Zerocash, including split The availability of custom hardware (ASIC) that speeds
and merge transactions, are fully private. They are based up hash computation prompts other crypto-currencies to
on complex zero-knowledge proofs which only reveal the adopt memory-hard hash functions. Ethereum uses Dagger-
fact that there exists unredeemed coins whose sum is a Hashimoto function, Litecoin and Dodgecoin use scrypt,
specific value. Both Zerocoin and Zerocash are and ZCash uses Equihash function. These functions are
implemented using zkSNARK [52] and carry large resistant to ASIC, as they demand large investment in
overheads due to the un- derlying zero-knowledge memory, but are easy to verify.
protocols. Zerocash, for instance, requires a trusted party How fast a block is created is dependent on how hard
to securely create and distribute some public parameters the puzzle is. Bitcoin sets t to a value equivalent of 10
whose sizes are in hundreds of megabytes. minutes per block. Litecoin, Dodgecoin and ZCash decrease
t to achieve lower average block time to several minutes.
Advanced signatures t cannot be arbitrary small because it leads to unnecessary
Bitcoin supports multi-signatures, in which a transaction forks in the blockchain. Forks not only lead to wastage of
resources but have security implication since they make it
can be redeemed when at least t out of n valid signatures
possible to double spend. Ethereum adopts GHOST [72]
are available. Multi-signatures are resilient against corrupt
protocol which helps bring down block generation time to
individuals by virtue of spreading the decryption or singing
tens of seconds without compromising much security. In
capabilities to a group of users. Lightning Network [53],
GHOST, the blockchain is allowed to have branches as long
an extension of Bitcoin with near-instant payment con-
as the branches do not contain conflicting transactions.
firmation, relies on multi-signatures to first deposit some
mutual funds on the blockchain. Once confirmed, payments
Proof of Stake
from the funds can happen outside of the blockchain with
immediate confirmation. Finally, the funds can be closed PoW mining is hugely expensive. The process is partic-
with corresponding transactions signed with all the ularly energy intensive, and has been estimated to con-
required signatures. Extensions of Bitcoin multi-signature sume enough electricity to power a small country like
scheme can be built directly on top of ECDSA [54]. More Denmark [73]. PoS is proposed to substantially reduce the
advanced schemes, e.g. [55], [56], can be employed (albeit cost of mining. Unlike Ethereum’s GHOST, PoS maintains a
not without major changes in the current design). single branch but changes the puzzle’s difficulty to be
Byzcoin [57] uses a group signature scheme called inversely proportional to the miner’s stake in the network.
Cosi [58] to reduce communication overhead in PBFT. Cosi A stake is essentially a locked account with a certain
involves four rounds of communication at the end of which balance representing the miner’s commitment to keep the
a collective signature is generated and verified by all mem- network healthy. Let s be the function that returns the
bers of the group. The signature is structured as a tree stake, then a miner M can generate a new block by solving
of Schnorr signatures. It significantly reduces the size of the puzzle of the following form:
messages broadcast in the network during the prepare and
commit phase of PBFT, because each node no longer needs H(n H(b)) < s(M ).t
individual signatures from all the other for verification.
It can be seen that the greater the stake s(M ), the easier it
is to find n.
4.3 Consensus Peercoin, forked from Bitcoin, is among the first systems
Recall that there is a spectrum of consensus protocols be- with PoS. It bootstraps by running PoW to generate coins.
hind blockchain systems, starting from purely computation The function s(.) in Peercoin takes a coin C as input and
bound like PoW to purely communication bound like PBFT. returns C.age(C) where age(C) is the coin’s age. Nxt [74],
another PoS system, bootstraps by selling its tokens. The
function s(.) in Nxt considers both the miner’s balance and
the elapsed time from the last block. The longer it is since
the last block, the easy it is to solve the puzzle. In particular:

s(M, bh) = bal(M ).age(bh−1)

 TABLE 3: Comparison of consensus protocols.
Consensus Protocol Network Settings Description
Hyperledger uses the original PBFT [15]. Tendermint [37] en-
hances it by assigning unequal weights to votes. Other variants
PBFT-based Private
include Scalable BFT [59], Parallel BFT [60], Optimistic BFT [61],
etc.
Stellar network [62] proposes its own consensus protocol where
Stellar Federated the nodes form intersecting groups (federates). Consensus is
agreed in each group, then propagated to the rest of the network.
Ripple payment system [6] proposes a variant of PBFT where the
Ripple Federated nodes belong to intersecting groups, and in each group there is a
large majority of non-Byzantine nodes.
Bitcoin uses pure proof-of-work, which leads to scalability issues.
Proof-of-Work (PoW) Public Bitcoin-NG [63], Byzcoin [57] separate leader election from trans-
action validation in PoW, thus increase the overall performance.
Tendermint [37] uses PoS, in which a node’s ability to create new
block is determined by its stake in the blockchain, e.g. the
Proof-of-Stake (PoS) Public amount of currencies it owns [64]. A set of high-stake owners
uses another consensus mechanism, which is usually faster than
PoW, to reach agreement on a new block.
Dfinity [ 3 9 ] proposes threshold relay in which nodes form random
group based on a public verifiable random function (Byzcoin [57]
Threshold Relay Public
and Elastico [65] adopt similar approaches). The nodes in the
group create a new block by signing it using threshold signature.
Parity [11] uses PoA, in which some pre-defined nodes are consid-
ered trusted authorities and they can propose the next blocks. It
Proof-of-Authority (PoA) Private
then uses round-robin scheduling to assign every authority node
a time window during which it can propose blocks.
Slimcoin [66] uses PoB, in which a node destroys some base
currencies it owns in another blockchain in order to get a chance
Proof-of-Burn (PoB) Public
of proposing a new block. Slimcoin supports PoB based on
Peercoin [67].
Sawtooth [42] uses PoET, in which each node runs a trusted
hardware, for example Intel SGX [68], that generates random
Proof-of-Elapsed Time (PoET) Private
timers. The first node whose timer has expired can propose the
next block.
Other protocols based on PoW are of the form proof-of-X, for
Others Public examples: Proof-of-Activity [69], Proof-of-Space [70], Proof-of-
Luck [71], etc.

where bh is the current block at height h, bal(M ) returns Bitcoin [75] exploit this probabilistic guarantee to allow
how many coins in M ’s account, and age returns how double spending. In contrast, the original PBFT protocol
much time has passed since the creation of a block at a [15] is deterministic. Implemented in the earlier version of
certain height. Hy- perledger (v0.6), the protocol ensures that once a block
Ethereum’s upcoming PoS protocol is implemented as is appended, it is final and cannot be replaced or modified.
a smart contract. Referred to as Casper, it allows miners
It incurs O(N 2) network messages for each round of agree-
to become validators by depositing Ethers to the Casper
ment where N is the number of nodes in the network. In
account. The contract then picks a validator to propose
practice, however, the original protocol scales poorly and
the next block according to the deposit amount. Its unique
collapses even before reaching the network limit [76]. We
feature, however, is to force validators to behave correctly
observe the same scalability issues in our evaluation of
or else risk losing the entire deposit. In particular, each
Hyperledger with BLOCKBENCH.
validator places a bet on whether a certain block will be
Tendermint proposes a small modification on top of
confirmed in the future. If the block is confirmed, the
PBFT. Instead of each node having an equal vote, in Ten-
validator gets a small reward. But if it is not, the validator
dermint each node may have different voting power, pro-
loses its deposit. This mechanism avoids the nothing-at-stake
portional to their stake in the network. To reach agreement
problem in which validators can propose blocks in different
in Tendermint it is necessary to only gather over 2/3 of the
branches. Tezos implements a simplified version of Casper
total voting power. This may be cheaper than waiting for
in which the nodes buy in to become authorities which can
2/3 of the network to response when there is a small number
then approve changes to the underlying blockchain. Tezos
of nodes with high stakes.
aims to provide an amendable blockchain in which soft
Recent works on improving PBFT have mainly focused
forks and hard forks are inherent features of the blockchain.
on its performance. Zyzzyva [77] optimizes for normal
PBFT variants cases (when there are no failures) via speculative
execution. XFT [78], assumes a network less hostile than
PoW suffers from non-finality, that is a block appended
purely Byzan- tine, and demonstrates better performance
to a blockchain is not confirmed until it is extended by
by reducing the number of network messages.
many other blocks. Even then, its existence in the block-
HoneyBadger [79], on the other hand, focuses on
chain is only probabilistic. For example, eclipse attacks on
improving security under asyn- chronous networks. It
employs a randomized agreement pro-

tocol which achieves safety with overwhelming probability even under network asynchrony. By optimizing the network layer, it is
shown to outperform PBFT even when the net- work is
synchronous. Both Zyzzyva, XFT and HoneyBadger hold every epoch. In particular, a node identity is its solution to
great promise, but they have not been integrated into any a cryptographic puzzle. In the second phase, the selected
blockchains. nodes perform PBFT to determine the next block. The end
result is faster block confirmation time at a scale much
Trusted hardware greater than traditional PBFT (over 1000 nodes).
Most overheads of PoW and PBFT can be attributed to Similar to Byzcoin and Elastico, Dfinity [39] and Algo-
the assumption that nodes behave in Byzantine manners. rand [84] select at each round a random set of nodes that
The availability of Intel SGX [80] or ARM TrustZone [81], can propose blocks. Unlike the former, they dispense with
however, makes it possible to relax the trust model in the PoW and instead use verifiable random functions (VRFs) to
Byzantine settings. In particular, a node equipped with select the consensus group. In Dfinity, the VRF is based on
trusted hardware can be reliably checked for certain the threshold signature of the previous block. In Algorand,
proper- ties, for example, that it is running a specific it is based on a random seed published in the previous
software. block and the node’s secret key.
Sawtooth Lake leverages SGX to replace PoW with a
Non-Byzantine
more efficient protocol called PoET. Specifically, PoET runs
inside an enclave protected by SGX. It starts by taking a The systems described so far in this section tolerate Byzan-
block number as input and generating a timer of a random tine failures, rendering them attractive for public settings
duration t. Afterward, it can produce certificates indicating and for private settings where the cost of engaging trusted
how much time has passed since the timer starts. A node parties (for example, for escrowing assets) is high. Some
whose PoET generates the smallest t can append the block blockchains, however, assume trusted parties in order to
when the timer expires. In particular, the node attaches its simplify their designs. These blockchains have no safety
PoET certificate to the block, and as long as t is smaller guarantees when any of such parties behaves maliciously.
than what generated by any other node the block is Openchain [35] relies on a single trusted party (called
accepted. validator) that determines the next block. Consequently, it
A2M [82] and Hybster [83] both exploit trusted is most vulnerable to attacks as the validator is the single
hardware to reduce the number of replicas needed to point of failure. Multichain and Parity have more than one
tolerate f failures from 3f + 1 to 2f + 1. This means an N trusted party which is referred to as authority in their
-node network can now tolerate up to N/2 adversarial systems. Each authority is given a time slice, via round-
nodes, as opposed to N/3 adversarial nodes in the robin scheduling, during which it can append new blocks to
original PBFT. A2M’s and Hybster’s safety are the chain. This simple proof-of-authority (PoA) protocol
dependent on the trusted code bases (TCBs) that avoids single point of failure while ensuring balanced
implement simple functions: a log data structure in the workloads among the authorities. HydraChain and
former and a monotonic counter in the latter. BigChainDb also have multi- ple authorities, but one
authority cannot unilaterally decide the next blocks.
Federated Instead, the block is decided via majority voting. Quorum
Despite numerous improvements to the original protocol, [33] employs Raft [20] as the consensus pro- tocol among its
PBFT-based consensus remains communication bound, authorities. Raft implements crash tolerant state machine
thus it ultimately fails to scale beyond a certain number of replication, which is an important building block of
nodes. To overcome this hard limit without scarifying modern distributed database systems. Using Raft, Quorum
safety, Stellar and Ripple adopt an approach that partitions is able to make safe progress even when some authority
the network into smaller groups called federates. Each nodes crash.
federate runs a lo- cal consensus protocol among its Corda’s consensus protocol is executed by a set of
members, which does not run into scalability problems trusted parties called notaries which check if a given
because of the small network size. Local agreements are transac-
then propagated to the entire network via nodes lying in tion has been executed before. By delegating this check to an
the intersections of the federates. Global consensus can be entity outside of the blockchain, Corda can justify using Raft
achieved under certain conditions. For Stellar, the for consensus. Transactions in Corda are sent to the notaries
condition is that every two federates intersect at non- before being confirmed in the blockchain. The notaries then
Byzantine nodes. Ripple’s safety conditions are that there is use Raft to ensure that the transactions are replicated among
a large majority of honest nodes in every federate, and that themselves and remain highly available despite crashes.
the intersection of any two federates contain at least one The latest release of Hyperledger (v1.0) outsources the
honest node. consensus component to Kafka — another building block of-
Both Stellar and Ripple assume federates are pre- ten found in distributed database systems. More specifically,
defined transactions are sent to a centralized Kafka service which
and their safety conditions can be enforced by a network orders them into a stream of events. Every node subscribes
administrator. In a decentralized environment where node to the same Kafka stream and therefore is notified of new
identities are unknown, such assumptions do not hold. transactions in the same order as they are published. Since
Byzcoin [57] and Elastico [65] propose novel, two-phase there is only one Kafka service, the observed transaction
protocols that combine PoW and PBFT. In the first phase, sequence is the same at every node.
PoW is used to form a consensus group. Byzcoin imple-
Others
ments this by having a sliding window over the blockchain
and selecting the miners of the blocks within the window. IOTA [36] uses its own consensus protocol called Tangle
Elastico [65] groups nodes by their identities that change in which the blocks form a direct acyclic graph (DAG) as
opposed to a chain. In addition, a block in Tangle consists
of only one transaction. When appended, the block must Hyperledger does not have its own bytecotes. Instead,
it runs its language-agnostic smart contracts inside Docker
approve two other blocks creating links to them in the DAG.
containers. Specifically, a contract can be written in any
The block is confirmed when it is approved by many other
language, which is then compiled into native code and
blocks. Targeting IoT environments, Tangle’s main goal is
packed into a Docker image. When the contract is
efficiency and low-cost payment. Although its security has
uploaded, each node starts a new container with that
not been rigorously analyzed, the low values of transactions
image. Invoking the contract is done via Docker APIs. The
(micropayments) in Tangle could in practice discourage
contract can access the blockchain states via two methods
Byzantine behavior.
getState and putState exposed by a shim layer. One benefit
Kadena [26] proposes an extension to Raft that handles
Byzantine failures. It introduces various techniques on top of Hyperledger is that it supports multiple high-level
of Raft, such as message signatures, client verification and programming languages like Go and Java. However, its key-
value interfaces with the blockchain necessitates extra
incremental hashing. However, like Tangle, it is unclear
application logics for mapping high-level data structures
whether the protocol guarantees safety and liveness.
into key-value tuples.
Sawtooth Lake supports smart contracts in the form of
4.4 Smart Contracts transaction families. Each family is a user-defined Python
class loaded into the ledger during start up. The contract
Recall that a smart contract system can be characterized by
is executed in the native runtime environment as a normal
its language expressiveness or by its execution environment.
Python program.
Except for Openchain, IOTA, Ripple and Stellar, all systems
One consequence of supporting Turing complete con-
listed in Table 2 let users customize transaction logics to suit
tracts is that software bugs are all but inevitable. While
their applications. In the following, we group them by the
empowering, the Ethereum smart contract model receives
contract language expressiveness.
strong criticism because it directly exposes Ethers against
programming bugs. The security concerns indeed material-
Scripts
ized in the DAO attack [43] in which the attacker stole $ 50M
Bitcoin provides approximately 200 opcodes, but many of worth of asset. The attack exploits a concurrency bug in the
them are disabled in the latest implementation. Users can DAO smart contract which allows one to repeatedly draw
write stack-based programs with the opcodes. The most more money than what is specified in the transaction. Such
popular contracts in Bitcoin are related to multi-signatures. bugs are inherent in a language like EVM which has weak
One example is the escrow contract that requires 2 out of 3 or no formal specifications of its semantics. OYENTE [86]
signatures before a coin can be released. The language can presents three major causes of security bugs: transaction
also implement bounty-hunting style contracts, for order dependencies, timestamp dependencies and mishan-
example, one that releases the reward coins when the pre- dled exceptions. It formalizes Ethereum semantics and pro-
image of a hash value is found. poses a tool for checking bugs directly on EVM bytecodes.
BigchainDB [27] adopts a more expressive language The tool discovered over 8000 Ethereum contracts (worth
called crypto-condition. Developed as part of the Interledger over $60M ) with potential security bugs.
Protocol project [85], crypto-condition allows specifying Like any other transactions on the blockchain, smart
complex boolean expressions over many types of contract executions are transparent. It means the inputs,
signatures. A crypto-condition script contains conditions outputs and the states of the contract are visible to the
and fulfillments which are treated as inputs and output of network. Hawk [87] extends Zerocash to provide transaction
the script. The available conditions include timeout which privacy for smart contracts. The main challenge compared
enables time- release contracts. Crypto-condition’s to Zerocash lies in the arbitrary transaction logics, whereas
encoding is higher level than Bitcoin opcodes, making it in Zerocash the logics are constrained by a small set of oper-
easy to express com- plex logics. ations. Another challenge is to protect local states, which is
not applicable in Zerocash. Given a contract, Hawk compiles
Turing complete it with zkSNARK to make it privacy preserving. Transaction
inputs and outputs are pre- and post-processed via Hawk to
Ethereum is among the first blockchains offering Turing-
hide the complex cryptographic details. Although the pro-
complete smart contracts. Users write their contracts in
tocols incur large overhead both in time and space, Hawk
either Solidity, Serpent or LLC language, which then get
represents a practical cryptographic system that achieves
compiled to EVM bytecodes. EVM executes normal crypto-
both transaction privacy and fairness.
currency transactions, and it treats smart contract bytecodes
as a special transaction. Specifically, each smart contract is Verifiable
given its own memory to store local states. The memory is
Even before the DAO attack, some blockchains have
exposed as a key-value storage, though Solidity provides high-
rejected the models that allow for unconstrained
level data types such as map, array and composite structures.
computations. The languages of Kadena, Tezos and Corda
Resources consumed during execution of the contract, both in
are more powerful than Bitcoin scripts, but they trade
terms of CPU and memory, are tracked by EVM and charged to
Turing completeness for safety. Kadena’s language is a Lisp-
the transaction sender’s account. EVM also keeps track of
like functional language called Pact [26]. A Pact contract is
intermediate state changes and reverse them if there are
stored in the ledger in human readable form, which is then
insufficient funds to pay for the execution.
parsed and executed in Ocaml. It is strongly typed and can
be formally verified. Similarly, Tezos’s stack-based language
called Michelson
 Application Asset Securities
Crypto-currency
management settlement ... are multiple BLOCKBENCH workloads for evaluating it
individually.
block t block t+1
The consensus layer implements the consensus
... Block header Block header ...
protocol. The data model layer contains the structure,
TransactionContractTransactionContract

CPU Storage
roothashroothashroothashroothash Network content and operations on the blockchain data. The
execution layer includes details of the runtime
Consensus
environment for execut- ing smart contracts. Finally, the
Smart contract Smart contract
application layer includes classes of blockchain
Code
input, output input, output
applications. Croman et. al. [18] pro- posed to divide
Code
State storage State storage
blockchain into several planes: network, consensus,
Execution engine Data model storage, view and side plane. While similar to
BLOCKBENCH’s four layers, the plane abstraction was
geared towards crypto-currency applications and did not
take into account the execution of smart contracts.
Fig. 3: Blockchain software stack on a fully validating node.
5.2 Implementation
YCSB,

Application BLOCKBENCH stack consists of a frontend interface for

integrating new benchmark workloads, a backend inter-
face for integrating new blockchains, and a driver for
driving the workloads. A new blockchain can be inte-
grated into the framework’s backend by implementing the
Blockchain BLOCKBENCH IBlockchainConnector interface. The interface contains
Contracts Smallbank,
layers etc. workloads
operations for deploying the smart contract application,
Compilers, VM, CPU-Heavy invoking it by sending a transaction, and for querying the
Dockers, etc. Execution Engine blockchain states. Ethereum, Parity and Hyperledger are the
current backends, while ErisDB (or Monax), Quorum and
Blocks Analytics,
Transactions, Data Model IO-Heavy Sawtooth Lake integration are under development. A new
Indexing, etc.
benchmarking workload can be added by implementing
PoW, PoS,
IWorkloadConnector interface3. The Driver takes as
Consensus Commits
PBFT, etc. input a workload and sends transactions to the blockchain
according to user-defined configurations (number of oper-
Fig. 4: Blockchain software layers and corresponding there
bench-
mark workloads.

comes with a strong type system and fully specified seman-

tics. As a result, Tezos contracts can be statically checked for
safety. In Corda, a contract is a sequence of pure functions
that do not modify the states. Because the functions are
merely constraints, the contract’s safety can be formalized
and verified.

5 BLOCKBENCH
The previous section has presented a thorough qualitative
analysis of existing blockchains. In this section, we describe
our benchmarking framework called BLOCKBENCH [10].
Designed for quantitative analysis of blockchains as
data processing platforms, the framework targets private
blockchains with Turing-complete smart contracts. BLOCK-
BENCH is open source [88] and contains data processing
workloads commonly found in database benchmarks.

5.1 Layers
BLOCKBENCH targets blockchains that function as data
processing platforms. Such a blockchain must have no re-
strictions on the application logics, thus it must support
Turing complete smart contracts. Figure 3 shows the
logical components of the blockchain software stack, from
which we refine the taxonomy described in Section 4 into
four concrete layers shown in Figure 4. For each layer
ations, number of clients, threads, etc.). It collects failure. We simulate crashes, net- work delays and
runtime statistics which are used to compute five random message corruptions.
important metrics. • Security metrics: the ratio between the total
• Throughput: the number of successful number of blocks included in the main branch and
transactions per second. A workload can the total number of confirmed blocks. The lower the
be configured with multiple clients and ratio, the less vulnerable the system is from double
threads per clients to saturate the spending or selfish mining.
blockchain throughput.
• Latency: the response time per transaction. 5.3 Workloads
Driver implements blocking BLOCKBENCH comes with macro benchmark workloads
transactions, i.e. it waits for one transaction for evaluating the application layer, and micro benchmark
to finish before starting another. workloads for analyzing the lower layers. Smart contract
• Scalability: the changes in throughput implementations of the workloads shown in Figure 4 are
and latency when increasing the number available and can be readily deployed on Ethereum, Parity
of nodes and number of concurrent and Hyperledger.
workloads.
3. We assume that the smart contract implementing the workload’s
• Fault tolerance: the changes in logic is already implemented and deployed on the blockchain.
throughput and la- tency during node
Macro benchmark workloads type account_t struct {
We port two popular database benchmark workloads into Balance int
BLOCKBENCH, namely YCSB and Smallbank. YCSB is CommitBlock int
}
widely used for evaluating NoSQL databases, for which type transaction_t {
we implement a simple smart contract which functions as From string
a key-value storage. The WorkloadClient is based on To string
Val int
the YCSB driver [89] which preloads each storage with a
}
number of records, and supports requests with different func Invoke_SendValue(from_account string,
ratios of read and write operations. For Smallbank [90], a to_account string, value int) {
popular benchmark for OLTP workload, we implement a var pending_list []transaction_t
pending_list = decode(GetState("pending_list"))
smart contract that transfers money from one account to var new_txn transaction_t
another. new_txn = transaction_t
{ from_account, to_account,
value
Besides database workloads, BLOCKBENCH also pro- }
vides three other workloads based on real Ethereum con- pending_list = append(pending_list, new_txn)
PutState(’pending_list’, encode(pending_list))
tracts. The first is EtherId, a popular contract implementing }
a domain name registrar. The second is Doubler, the func Query_BlockTransactionList(block_number int)
pyramid []transaction_t {
scheme contract shown earlier in Figure 2. The third is return decode(GetState("block:"+block_number))
}
WavesPresale that implements a crowdfunding campaign func Query_AccountBlockRange(account string,
via digital token sales. start_block int, end_block int)
[]account_t {
Micro benchmark workloads version := decode(GetState(account+":latest"))
var ret []account_t
For the consensus layer, BLOCKBENCH provides DoNoth- while true {
ing workload in which the smart contract accepts a trans- var acc account_t
action as input and simply returns. Since the contract acc = decode(GetState(account+":"+version))
if acc.CommitBlock >= start_block &&
execution involves minimal number of operations at the acc.CommitBlock < end_block {
execution and data model layer, the overall performance ret = append(ret, acc)
will be determined by the consensus layer. } else if acc.CommitBlock < start_block {
break;
For the data model layer, BLOCKBENCH provides Ana-
}
lytics workload that is similar to an OLAP workload. In par- version -= 1
ticular, it performs scan-like and aggregate queries whose
performance is determined by the system’s data model. }
Specifically, there are two queries: return ret
}
Q1: Compute the total transaction values committed between
block i and block j. tract implementation in Hyperledger. To support historical data
Q2: Compute the largest transaction value involving a given lookup, the contract appends a counter to the key of each account. To
state (account) between block i and block j. fetch a specific version of an account, the key account:version is
used. The latest version is stored at the key account:latest. The
For Ethereum and Parity, both queries can be implemented
via JSON-RPC APIs that return transaction details and ac- contract also keeps keep a CommitBlock value in the data field for
count balances at a specific block. For Hyperledger, how- every version to point to the block number in which the current
ever, the second query must be implemented via a smart version is committed. To fetch the balances of a given account in a
contract (VersionKVStore), because Hyperledger has no given block range, the contract scans all versions of this account and
APIs for querying historical states. Figure 5 shows the con- returns the corresponding balance when the version’s CommitBlock
value is in the specified range. Fig. 5: Code snippet from the VersionKVStore smart
Another workload for the data model layer stresses contract for Analytics workload (Q1 and Q2).
the persistent storage. In particular, the IOHeavy workload
evaluates the blockchain’s IO performance by invoking a
contract that performs a large number of random writes execution layer for computationally heavy tasks by
and random reads to the local states. invoking a contract that executes quick sort algorithm over
Finally, for the execution layer BLOCKBENCHprovides a large array.
the CPUHeavy workload. It measures the efficiency of the

6 EVALUATION
We selected Ethereum, Parity and Hyperledger for a com-
parative study using BLOCKBENCH. They occupy different
positions in the design space, and are considered the most
mature in terms of the codebase and user base. We used
the popular Go implementation of Ethereum, geth v1.4.18,
the Parity release v1.6.0. Unless otherwise specified, the
Hyperledger version is v0.6.0-preview. We set up a private
testnet for Ethereum and Parity by defining a genesis block
and directly adding peers to the miner network. For Eth-
ereum, we manually tuned the difficulty variable in
the genesis block to ensure that miners do not diverge in
large networks. For Parity, we set the stepDuration
variable to
1. In both Ethereum and Parity, confirmationLength is
set to 5 seconds. The default batch size in Hyperledger is
500.
The experiments were run on a 48-node commodity clus-
ter. Each node has an E5-1650 3.5GHz CPU, 32GB RAM, 2TB
 Throughput
Latency Throughput vs. HStore
Ethereum Parity Hyperledger
103 Ethereum Parity Hyperledger 105
104 Ethereum Parity Hyperledger142702
H-Store

102 104 21596

1273 1122
3 92 114

second
10
#tx/s

#tx/s
51
38
284 255 101 103
1273 1122

102 3 4 284255
45 46 100 102

4546
101
10−1 101
YCSB Smallbank YCSB Smallbank YCSB Smallbank

Fig. 6: Blockchain peak performance with 8 clients and

Fig. 7: Performance of the three blockchain systems versus H-
8 servers. The benchmark workload is YCSB. The perfor-
Store.
mance against Smallbank workload is similar.

hard drive, running Ubuntu 14.04 Trusty, and connected to To put their performance in context, we compare the
the other nodes via 1GB switch. For Ethereum, we reserved three blockchains against a popular in-memory database
8 cores out of the available 12 cores per machine, so that system, namely H-Store, using the YCSB and Smallbank
the periodic polls from the client’s driver process do not workload. Blockchains and databases do not necessarily
interfere with the mining process. Our main findings are as share the same design goal: the former are not designed
follows: for general data processing, nor do the latter protect data
integrity against Byzantine failures. Nonetheless, we argue
• Hyperledger performs consistently better than Eth- that the comparison offers useful insights into the design
ereum and Parity across the benchmarks. But it fails trade-offs and relative performance of the two systems. We
to scale up to more than 16 nodes. ran H-Store’s own benchmark driver and set the
• Ethereum and Parity are more resilient to node fail- transaction rate at 100,000 tx/s. Figure 7 shows at least an
ures, but they are vulnerable to security attacks that order of mag- nitude gap in throughput and two order of
forks the blockchain. magnitude in la- tency. Specifically, H-Store achieves over
• The main bottlenecks in Hyperledger and Ethereum 140K tx/s through- put while maintaining sub-millisecond
are the consensus protocols, but for Parity the latency. The gap in performance is due to the cost of
bottle- neck is caused by transaction signing. consensus protocols. For YCSB, for example, H-Store
• Ethereum and Parity incur large overheads in terms requires almost no coordination among peers, whereas
of memory and disk usage. Their execution engine is Ethereum and Hyperledger suffer the overhead of PoW and
also less efficient than that of Hyperledger. PBFT. An interesting observation is the overhead of
• Hyperledger’s data model is low level, but its flexi- Smallbank. Recall that compared to YCSB, Smallbank
bility enables customized optimization for analytical consists of more complex transactions in which multiple
queries. keys are updated in a single transaction. Smallbank is
simple but is representative of the large class of transac-
6.1 Macro benchmarks tional workloads such as TPC-C. We observe that in H-Store,
This section discusses the performance of the blockchains Smallbank achieves 6.6x lower throughput and 4x higher
at the application layer, using YCSB and Smallbank bench- latency than YCSB, which reflects the cost of distributed
marks. transaction management. In contrast, the blockchains
suffer modest degradation in performance: 10% in
Throughput and latency throughput and 20% in latency. This is because each node
Figure 6 shows the peak performance with 8 servers and 8 in the blockchains maintains the complete states, therefore
concurrent clients over the period of 5 minutes. We it pays no overhead in coordinating distributed
observe that in terms of throughput, Hyperledger transactions since the states are not partitioned.
outperforms the other two in both benchmarks. The gap
between Hyper- ledger and Ethereum is due to the Scalability
difference in the con- sensus protocols: one is based on
PBFT while the other is based on PoW. With 8 servers, the We fixed the client request rate ( 320 requests per second
communication cost from broadcasting messages is for Hyperledger, 160 requests per second for Ethereum
cheaper than block mining whose difficulty is set at roughly and Parity) and increased both the number of clients and
2.5s per block. The gap between Parity and Hyperledger is the number of servers. Figure 8 illustrates how well the
not due to consensus protocols, as Parity’s PoA protocol is three systems scale to handle larger YCSB workloads.
expected to be simpler and more efficient than both PoW Parity’s performance remains constant as the network size
and PBFT. Instead, we observe that Parity processes and offered load increase, due to the constant transaction
transactions at a constant rate, and that it enforces a pro- cessing rate at the servers. Interestingly, while
maximum client request rate at around 80 tx/s. Ethereum’s throughput and latency degrade almost linearly
beyond 8 servers, Hyperledger stops working beyond 16
servers.

Latency Ethereum Parity

timer will expire. In the first case, the PBFT leader resends
104 Throughput
Ethereum Parity Hyperledger Hyperledger messages of the current consensus round. In the second,
103 the servers start the view change phase which broadcasts
multiple rounds of consensus messages. As client requests
103 102 are still occupying the network channels, both consensus

second
or view change messages are dropped with high probability.
#tx/s

101 Consequently, the network gets stuck in perpetual

102 attempts to establish a stable view. The fact that PBFT is
100 sensitive to network conditions has been observed in the
past [76].
101 10−1 We note that in its latest release (v1.0) Hyperledger has
12 4 8 12 16 20 24 28 32 12 4 8 12 16 20 24 28 32
#nodes #nodes replaced PBFT with a global ordering service. Implemented
using Kafka, this new consensus engine may offer higher
throughput than PBFT, but it offers no protection against
Fig. 8: Performance scalability (with the same number of Byzantine failures.
clients and servers). The benchmark workload is YCSB. The results so far indicate that scaling both the number of
The scalability against Smallbank is similar, except that clients and number of servers degrades the performance and
Hyperledger fails beyond 8 nodes instead of 16. even causes Hyperledger to fail. We next examined the costs
of increasing the number of servers alone while fixing the
number of clients to 8. Figure 9 shows that the performance
1400 Throughput 180 Latency becomes worse as there are more servers, meaning that
Ethereum Parity Hyperledger
1200 160 the systems incur some network overheads. For
140 Hyperledger, having more servers means more messages
1000
120
being exchanged and higher overheads. In particular, to a
800 confirm a batch of transaction in a larger network, the
100
#tx/s

second

leader in Hyperledger needs to wait for larger sets of

600 80
messages, therefore lowering overall throughputs. We note
400
Ethereum Parity Hyperledger
60
that with a fixed number of clients Hyperledger can scale
200
40 up to 32 nodes, as oppose to failing after 16 nodes as in
20 Figure 8. This is because with fewer clients, the message
0
8 12 16 20 24 28 32 0 queues at each node are not saturated with client requests
8 12 16 20 24 28 32
#nodes #nodes and therefore consensus messages are less likely to get
dropped.
For Ethereum, even though it is computation bound,
Fig. 9: Performance scalability (with 8 clients). a sufficient number of consensus messages, either a batch timer or a
view-change

To understand why Hyperledger failed to scale beyond

16 servers and 16 clients, we examined the system logs and
find that the nodes never reached consensus on any batch of
transactions. We observe a large number of messages being
dropped even when there are fewer than 16 servers and
clients. Furthermore, the servers repeatedly triggered view
changes but never succeeded. At the client side, requests
took longer to return as time passed, suggesting that the
server or the network were saturated. Since the original
PBFT protocol guarantees both liveness and safety, we can
attribute this failure to scale to Hyperledger’s implementa-
tion. Further investigation reveals that it is indeed the case.

Hyperledger uses gRPC for communication between

servers. Each server keeps a separate message queue for
every other servers in the network. The queue size is
defined during initial setup (the default size being 1000
messages) and the default behavior is to drop messages
when the queue is full. In the current design, both client
requests (transactions) and consensus messages (pre-
prepare, pre- pare, commit, view changes) are sent on the
same channel, that is they share the same queue. For large
numbers of concurrent clients and servers, the channels
are dominated by client requests, increasing the
probability of consensus messages being dropped. Without
it still consumes a modest amount of network
resources for propagating transactions and blocks
to other nodes. Furthermore, with larger network,
the difficulty is increased to account for the longer
propagation delays. We observe that to prevent
the network from diverging, the difficulty level
increases at a higher rate than the number of
nodes. Thus, one reason for Ethereum’s
throughput degradation is due to network sizes.
Another reason is that in our settings, 8 clients
send requests to only 8 servers, but these servers
do not always broadcast transactions to each
other (they keep mining on their own transaction
pool). As a result, the network mining capability is
not fully utilized.

6.1.1 Fault tolerance and security

To evaluate how resilient the systems are to
crash failures, we ran the systems with 12 and
16 servers, with 8 clients for over 5 minutes,
during which we killed off 4 servers at 250th
second. Due to space constraints, we only
highlight key findings here and refer readers to
[10] for more details. First, Ethereum is
unaffected by the change, suggesting that the
failed servers do not contribute significantly to
the mining process. Second, Parity’s throughput
is also unaffected. It is because each node is
given equal time slice during which it can
generate block, thus failing 4 nodes in Parity
means that the remaining 8 nodes are given
bigger time slices. Third , Hyperledger stops
generating blocks after the failure in the 12-
server network, which is as expected because
the PBFT can only tolerate fewer than 4 failures
in a 12- server network. In the 16-server
network, Hyperledger still
 #Blocks generated 104 Execution time Peak memory usage
500 Ethereum Parity Hyperledger
Ethereum-bc Ethereum-totalParity-bc Parity-total
Hyperledger-bc Hyperledger-total
Ethereum Parity Hyperledger
103
400 108
232.78
102 79.61 22,819
300 24.04
#blocks

13,090

second
7
10

MB
1
10 10.52
3.01 4,150
200
1.94 2,078
100 1,353
0.33 106 718
100 0.19 376473
10−1

0 10−2 105
0 50 100 150 200 250 300 350 400 1M 10M x100M 1M 10M 100Mx
time (second) input size input size

Fig. 10: Blockchain forks caused by attacks that partitions

the network in half at 100th second and lasts for 150 sec- Fig. 11: CPUHeavy workload, ‘X’ indicates Out-of-Memory
error.
onds. X-total means the total number of blocks generated
in blockchain X, X-bc means the total number of blocks that
reach consensus in blockchain X. Execution layer
We deployed the CPUHeavy smart contract that is initial-
ized with an integer array of a given size. The array is
generates blocks but at a lower rate, which were caused by
initialized in descending order. We invoked the contract to
the remaining servers having to stabilize the network after
sort the array using quicksort algorithm, and measured the
the failures by synchronizing their views.
execution time and server’s peak memory usage. The
We next simulated the attack that could make the results for varying input sizes are shown in Figure 11.
blockchains vulnerable to double spending. The attack es- Although Ethereum and Parity use the same execution
sentially creates network partition at 100th second that engine, i.e. EVM, Parity’s implementation is more
lasts for 150 seconds. We set the partition size to be half optimized, therefore it is more computation and memory
of the original4. Figure 10 compares the vulnerability of efficient. An interesting finding is that Ethereum incurs
the three blockchains with 8 clients and 8 servers. Recall large memory overhead. In sorting 10M elements, it uses
that vulnerability is measured as the differences in the 22GB of memory, as compared to 473MB used by
number of total blocks and the number of blocks on the Hyperledger. Ethereum runs out of mem- ory when sorting
main branch. We refer to this as ∆. Both Ethereum and more than 10M elements. In Hyperledger, the smart
Parity fork at 100th seconds, and ∆ increases as time contract is compiled and runs directly on the native
passes. For the attack duration, upto 30% of the blocks are machine within Docker environment, thus it does not have
generated in the forked branch, meaning that the systems the overheads associated with executing high-level EVM
are highly exposed to double spending or selfish mining byte code. As the result, Hyperledger is much more
attacks. When the partition heals, the nodes come to efficient in term of speed and memory usage. Finally, we
consensus on the main branch and discard the forked
note that all three systems fail to make use of the multi-
blocks. As a consequence, ∆ stops increasing shortly after core architecture, i.e. they execute the contracts using only
250th second. Hyperledger, in stark contrast, has no fork one core.
which is as expected because its consensus protocol is
proven to guaranteed safety. We note, however, that
Data model - IOHeavy
Hyperledger takes longer than the other two systems to
recover from the attacks (about 50 seconds more). This is We deployed the IOHeavy smart contract that performs a
because of the synchronization protocol performed after number of read and write operations of key-value tuples.
the partitioned nodes reconnect. In particular, when the We used 20-byte keys and 100-byte values. Figure 12 re-
nodes reconnect they enter the view change phase and ports the throughput and disk usage for these operations.
exchange checkpointed states with each other in order to Ethereum and Parity use the same data model and internal
establish a new, stable view. index structure, therefore they incur similar space over-
heads. Both use an order of magnitude more storage space
than Hyperledger which employs a simple key-value data
model. Parity holds all the state information in memory, so
6.2 Micro benchmarks
it has better I/O performance but fails to handle large data
This section discusses the performance of the blockchains at (capped by over 3M states under our hardware settings).
the execution, data model and consensus layer. For the first On the contrary, Ethereum only caches only parts of the
two layers, the workloads were run with one client and one state in memory (using LRU for eviction policy), therefore it
server. For the consensus layer, 8 clients and 8 servers were can handle more data than Parity at the cost of throughput.
used. Hyperledger leverages RocksDB to manage its states, which
makes it more efficient at scale.
4. We note that partitioning a N -node network in half does not Hyperledger v1.0. We used the same IOHeavy smart con-
mean there are N/2 Byzantine nodes. In fact, Byzantine tolerance
protocols do not count network adversary as Byzantine failure
tract to compare I/O performance of Hyperledger version
v1.0 with the older version v0.6. As Figure 14 illustrates,
 Average throughput (write)
Average throughput (read) Disk usage
Ethereum Parity Hyperledger
104 5618
5411 5123 4852 Ethereum ParityHyperledger 105 EthereumParityHyperledger
4527

1049329 8785 9234 8758 8974 870086708605

1329 30,221
1181 1087
second

second

MB
103
104
5,459 5,045 4,865
377 359 337 2,337 2,086 12,804 12,104
2019 2,477
1631
1282 1,283
102 69 103 675
103
x x 512 6.4M 360
0.8M 1.6M 3.2M 6.4M 12.8M 0.8M 1.6M 3.2M 12.8M 0.8M 1.6M 3.2M 6.4M 12.8M
# tuples # tuples x x # tuples x x

(a) Write (b) Read (c) Disk usage

Fig. 12: IOHeavy workload, ‘X’ indicates Out-of-Memory error.

Latency
Latency Transaction througput
13.314 101 104
101 Ethreum Parity Hyperledger 8.901 8.465
Ethereum Parity Hyperledger 4.907 SmallBank YCSB
3.472
DoNothing

1.374
0.915 0.984
100 103 112212731285
100 0.595
0.533
0.427
second

second

#tx/s
256 284 328
0.168
0.129 0.135 0.107

1 0−1 10−1 0.091 0.076 2

10
0.032 0.034 0.038 0.033 0.031 45 45 46
0.025 0.024 0.020
0.023
0.020 0.020 0.019 0.019

10−2 1 10 100 1,000 10,000 10−2 1 10 100 1,000 10,000

101
# blocks scanned # blocks scanned Ethereum Parity Hyperledger

(a) Analytics workload (Q1)

(b) Analytics workload (Q2) (c) DoNothing workload

Fig. 13: Analytics and DoNothing workloads.

Average throughput (write)

Average throughput (read) 4 Disk usage
10
Hyperledger v0.6.0 Hyperledger v1.0.0-rc1

4
10 4
Hyperledger v0.6.0 Hyperledger v1.0.0-rc1 Hyperledger v0.6.0 Hyperledger v1.0.0-rc1
6122 5815 5618 5411 10
3
10
8559 8690 8785 8758
ops/sec

ops/sec

MB

675
506
360
3 625 625 623 625 280
10 3 197
10 2 141
10 107
627 628 631 630
63

2
10 2 1
0.2M 0.4M 0.8M 1.2M 10 0.2M 0.4M 0.8M 1.2M 10 0.2M 0.4M 0.8M 1.2M
# tuples # tuples # tuples

(a) Write
(b) Read (c) Read

Fig. 14: Hyperledger v0.6.0 vs. v1.0.0 against IOHeavy workload.

throughputs of v1.0 is an order of magnitude worse than PBFT with a centralized service not only fails to protect the
that of v0.6. Furthermore, v1.0 crashes with more than blockchain against Byzantine
0.8M operations, reporting exceptions about message
oversizes. The significant gap can be attributed to the
changes in the system architecture from v0.6 to v1.0. In
the former, the nodes take part in PBFT to confirm a block.
In this case, transactions in the IOHeavy workload incur no
consensus overhead because there is only one node. In the
latter, a new service, the orderer, is introduced into the
network to order transactions and provide the consensus.
With this new service, transactions in the IOHeavy
workload now need to communicate with the orderer for
them to be confirmed. More specifically, the nodes in v1.0
perform three more steps to finish a transaction compared
to v0.6. As communication overhead increases, the
throughputs decrease. This result suggests that replacing
failures, but it may also impair the overall performance. signing transactions when there are many accounts, we
considered transactions using only 1024 accounts. We then
Data model - Analytics executed the two queries described in Section 5.3 and
We implemented the analytics workload by measured their latencies. Fig- ure 13 shows that the
initializing the three systems with over 120, 000 performance for Q1 is similar, whereas Q2 sees a significant
accounts with a fixed balance. We then pre-loaded gap between Hyperledger and the rest. We note that the main
them with 100, 000 blocks, each contains 3 bottleneck for both Q1 and Q2 is the number of network (RPC)
transactions on average. The transaction trans- requests sent by the client. For Q1, the client sends the same
fers a value from one random account to another number of requests to all systems, therefore their
random account. Due to Parity’s overheads in performance are similar. On the other hand,

for Q2 the client sends one RPC per block to Ethereum and Usability of blockchain. Our experience in working with the three
Parity, but only one RPC to Hyperledger because of our blockchain systems confirms that in their cur- rent states, the
customized smart contract implementation. This saving in blockchains are not yet ready for mass usage. Their designs and
network roundtrip time translates to over 10x codebases are still being refined constantly, and there are no other
improvement in Q2 latency. established applications beside crypto-currency. Of the three systems,
Ethereum is more mature both in terms of its codebase, user base and
6.2.1 Consensus de- veloper community. Another usability issue we encountered
We deployed the DoNothing smart contract that accepts
a transaction and returns immediately. We measured the
throughput of this workload and compare against that of
YCSB and Smallbank. The differences compared to other
workloads, shown in Figure 13[c] is indicative of the cost
of consensus protocol versus the rest of the software stack.
In particular, for Ethereum we observe 10% increases in
throughput as compared to YCSB, which means that execu-
tion of the YCSB transaction accounts for the 10% overhead.
We observe no differences among these workloads in Parity,
because the bottleneck in Parity is due to transaction signing
(even empty transactions still need to be signed), not due to
consensus or transaction execution.

7 D ISCUSSION
In this section, we first distill the lessons learned during the
comparative studies of Ethereum, Parity and Hyperledger.
We then discuss how design principles from database sys-
tems could help improve blockchain performance.

7.1 Lessons Learned From the Performance Study

Understanding blockchain systems. BLOCKBENCH aims
to facilitate better understanding of the design and per-
formance of different private blockchains. As more and
more blockchain systems are being proposed, each offering
different sets of feature, BLOCKBENCH’s main value is that
it narrows down the design space into four distinct ab-
straction layers. The layers are derived from our taxonomy
presented in Section 4 which sufficiently captures the key
and subtle characteristics of blockchain systems. By bench-
marking these layers, one can gain insights into the design
trade-offs and performance bottlenecks. For example, us-
ing the IOHeavy workload we identify that Parity trades
performance for scalability by keeping states in memory.
In addition, the workload reveals potential performance
issues with the latest version of Hyperledger. Another ex-
ample is the Analytics workload that demonstrates trade-
offs in the data models. In particular, Hyperledger’s simple
key-value model means some analytical queries cannot be
directly supported. However, it enables optimization that
helps answering analytical queries more efficiently. Finally,
we identify that the bottleneck in Parity is not due to
the consensus protocol, but due to the server’s transaction
signing. We argue it is not easy to arrive at such insights
without a systematic analysis framework.
is in porting smart contracts from one system to for examples, have decoupled the consensus layer by
another, due of their different programming outsourcing it to separate parties. The data model layer
models. This is likely to be exacerbated as more could also be de- coupled. For instance, current systems
blockchain platforms are being proposed [6], [26], employ generic key- value storage, which may not be best
[ 4 0 ] , [91]. suited to the unique data structure and operations in
Comparison to database systems. The comparison against blockchain. UStore [92] demonstrates that a storage
H-Store presented in the previous section designed around the blockchain data structure is able to
demonstrates that blockchains perform poorly achieve better performance than ex- isting
at data processing tasks currently being handled implementations. Most importantly, we observe that
by database systems. Although databases are
5. Note that the execution phase can be considered the last part
designed without security and tolerance to
of the consensus phase, because during execution a node may detect
Byzantine failures, we remark that the gap conflicting transactions and abort the current consensus round.
remains too big for blockchains to be disruptive
to incumbent database systems. There are much
for blockchains to learn from databases in terms
of high-performance data processing, which we
discuss next. Nevertheless, there are useful
lessons that databases can take from the
popularity and success of blockchains. Perhaps
the most interesting lesson is that there is a
clear need for Byzantine tolerant data pro-
cessing systems which can accommodate a large
number of users. Distributed databases have
diverged from P2P system designs by assuming
non-Byzantine failures [16], but with the
increasing availability of faster and more
trustworthy hardware, this may be the right
time for the community to revise interest in
high-performance, decentralized, P2P database
systems.

7.2 Bringing Database Designs into Blockchains

The challenges in scaling blockchain by
optimizing the consensus protocols are being
addressed in many recent works [57], [65].
Nevertheless, we demonstrated in our com-
parative study that there are other performance
bottlenecks beside consensus. Figure 15
illustrates different stages that a transaction
goes through before it is considered committed
to the blockchain. Each stage could be a
potential bottleneck and be subject to future
optimizations. First, transactions are batch into
a block. Next, the block becomes input to the
consensus protocol, and if selected by the
protocol it is sent to the execution engine.
Finally, the engine executes the en- tire batch of
transaction, creates new states and appends the
block to the chain5. We note the striking
resemblance with the flow of a transaction in a
distributed database. In fact, the only difference
being the consensus protocol: databases use
two-phase commit or Paxos, whereas
blockchains use Byzantine tolerant protocols or
variants thereof. Given the similarity, we
propose four approaches inspired by design
principles in databases to improve blockchain
performance.

Decoupling the layers and optimize them

individually. One possible direction is to
decouple storage, execution en- gine and
consensus layer from each other, then optimize
and scale them independently. Tezos and Corda,
 Blockchain peer

execute commit
batch
...

transactions

Fig. 15: The life of a blockchain transaction.

current data models in Ethereum and Hypereldger are not

ideal for answering analytics queries. In particular, both do
8 C ONCLUSION
not support fine-grained versioning at the transaction In this paper, we have conducted a comprehensive survey
level: it is not possible to immediately query previous on blockchain technologies. We laid out four underpinning
versions of a state except for querying at block level and concepts behind blockchains and analyzed the state of the
processing the state of each block. Implementing a new art using these concepts. We presented our benchmarking
data model is less complex when the storage layer is framework, BLOCKBENCH which is designed to evaluate
decoupled from the rest of the blockchain stack. performance of blockchains as data processing platforms.
Finally, we discussed four potential research directions,
Embracing new hardware primitives. Many data inspired by database design principles, for improving
process- ing systems are taking advantage of new hardware block- hchain performance. We hope that the survey and
to boost their performance [93], [94], [95]. For blockchain, bench- marking framework would serve to guide the
using trusted hardware, the underlying Byzantine fault design and implementation of future blockchain systems
tolerance protocols can be modified to incur fewer that are not only secure, but scalable and usable in the real
network mes- sages [82]. With trusted hardware, the world.
blockchain can tol- erate more failures, and because of
fewer network messages it can scale better. Systems like
Parity and Ethereum can take advantage of multi-core ACKNOWLEDGEMENTS
CPUs and large memory to improve contract execution and
This work is funded by the National Research Foundation,
I/O performance.
Prime Ministers Office, Singapore, under its Competitive
Sharding. Blockchain is essentially a replicated state ma- Research Programme (CRP Award No. NRF-CRP8-2011-08).
chine system, in which each node maintains the same data. We would like to thank colleagues who have provided
As such, blockchains are fundamentally different to valuable feedback to help improve the paper.
database systems such as H-Store in which the data is
partitioned (or sharded) across the nodes. Sharding helps
reduce the computation cost and can make transaction
R EFERENCES
processing faster. The main challenge with sharding is to [1] S. Nakamoto, “Bitcoin: A peer-to-peer electronic cash system,”
ensure consistency among multiple shards. However, exist- 2008.
[2] Q. Lin, P. Chang, G. Chen, B. C. Ooi, K. Tan, and Z. Wang, “To-
ing consistency protocols used in database systems do not wards a non-2pc transaction management in distributed
work under Byzantine failure. Nevertheless, their designs database systems,” in Proceedings of ACM International
can offer insights into realizing a more scalable sharding Conference on Manage- ment of Data (SIGMOD), San Francisco,
protocol for blockchain. Corda partitions the ledger into CA, USA, 2016, pp. 1659– 1674.
sub-ledgers belonging to smaller group, thus avoiding the [3] A. Thomson, T. Diamond, S. Weng, K. Ren, P. Shao, and
D. J. Abadi, “Calvin: fast distributed transactions for partitioned
need to broadcast transactions to the entire network. How- database systems,” in Proceedings of ACM International Conference
ever, it still depends on an external, centralized compo- on Management of Data (SIGMOD), Scottsdale, AZ, USA, 2012, pp.
nent for consensus between sub-ledgers. Recent work [65] 1–12.
[4] P. Bailis, A. Fekete, M. J. Franklin, A. Ghodsi, J. M. Hellerstein, and
has demonstrated the feasibility of sharding the consensus I. Stoica, “Coordination avoidance in database systems,” PVLDB,
protocol, making important steps towards partitioning the vol. 8, no. 3, pp. 185–196, 2014.
entire blockchain. [5] “Ethereum blockchain app platform,” https://www.ethereum.
org/.
[6] Ripple, “Ripple,” https://ripple.com.
Supporting declarative language. Having a set of high- [7] Melonport, “Blockchain software for asset management,” http://
melonport.com.
level operations that can be composed in a declarative [8] J. Morgan and O. Wyman, “Unlocking economic advantage with
manner makes it easy to define complex smart contracts. It blockchain. a guide for asset managers.” 2016.
also opens up opportunities for low-level optimizations [9] G. S. Group, “Blockchain: putting theory into practice,” 2016.
that speed up contract execution. Hawk [87] is a recent [10] T. T. A. Dinh, J. Wang, G. Chen, L. Rui, K.-L. Tan, and B. C. Ooi,
“Blockbench: a benchmarking framework for analyzing private
exam- ple that applies compiler techniques to hide blockchains,” in SIGMOD, 2017.
complexity of implementing privacy preserving smart [11] Ethcore, “Parity: next generation ethereum browser,” https://
contracts. However, it aims to strengthen security of the ethcore.io/parity.html.
[12] Hyperledger, “Blockchain technologies for business,” https://
smart contracts rather than improve their performance.
www.hyperledger.org.
[13] M. Stonebraker, S. Madden, D. J. Abadi, S. Harizopoulos,
N. Hachem, and P. Helland, “The end of an architectural era (it’s [43] “Decentralized autonomous organization,” https://www.
time for a complete rewrite),” in Proceedings of the 33rd International ethereum.org/dao.
Conference on Very Large Data Bases (VLDB), Vienna, Austria, 2007, [44] S. Eskandari, D. Barrera, E. Stobert, and J. Clark, “A first look at
pp. 1150–1160. the usability of bitcoin key management,” in Workshop on Usable
[14] J. C. Corbett, J. Dean, M. Epstein, A. Fikes, C. Frost, J. J. Furman, Security, 2015.
S. Ghemawat, A. Gubarev, C. Heiser, P. Hochschild, W. C. Hsieh, [45] T. T. A. Dinh, P. Saxena, E. chien Chang, B. C. Ooi, and C. Zhang,
S. Kanthak, E. Kogan, H. Li, A. Lloyd, S. Melnik, D. Mwaura, “M2r: enabling stronger privacy in mapreduce computation,” in
D. Nagle, S. Quinlan, R. Rao, L. Rolig, Y. Saito, M. Szymaniak, USENIX Security, 2015.
C. Taylor, R. Wang, and D. Woodford, “Spanner: Google’s globally- [46] W. Zheng, A. Dave, J. Beekman, R. A. Popa, J. Gonzalez, and
distributed database,” in Proceedings of 10th USENIX Symposium I. Stoica, “Oqaque: an oblivious and encrypted distributed ana-
on Operating Systems Design and Implementation (OSDI), Hollywood, lytics platform,” in NSDI, 2017.
CA, USA, 2012, pp. 261–264. [47] F. Zhang, E. Cecchetti, K. Croman, and E. Shi, “Town crier: an
[15] M. Castro and B. Liskov, “Practical byzantine fault tolerance,” in authenticated data feed for smart contracts,” in IEEE Security &
Proceedings of the 3rd USENIX Symposium on Operating Systems Privacy, 2016.
Design and Implementation (OSDI), New Orleans, Louisiana, USA, [48] B. Smyth, M. Ryan, and L. Chen, “Direct anonymous attestation:
1999, pp. 173–186. ensuring privacy with corrupt administrator,” in European Work-
[16] Q. H. Vu, M. Lupu, and B. C. Ooi, Peer-to-Peer Computing shop on Security and Privacy in Ad hoc and Sensor networks, 2007.
Principles and Applications. Springer-Verlag, 2009. [49] S. Meiklejohn, M. Pomarole, G. Jordan, K. Levchenko, D. McCoy,
[17] I. Eyal and E. G. Sirer, “Majority is not enough: Bitcoin mining is G. M. Voelker, and S. Savage, “A fistful of bitcoins: characterizing
vulnerable,” in Proceedings of 18th International Conference on Finan- payments among men with no names,” in IMC, 2013.
cial Cryptography and Data Security (FC), Christ Church, Barbados, [50] “Chainalysis - blockchain analysis,” https://www.chainalysis.
2014, pp. 436–454. com.
[18] K. Croman, C. Decker, I. Eyal, A. E. Gencer, A. Juels, A. Kosba, [51] I. Miers, C. Garman, M. Green, and A. D. Rubin, “Zerocoins:
A. Miller, P. Saxena, E. Shi, and E. Gu¨ n, “On scaling decentralized anonymous distributed e-cash from bitcoin,” in IEEE Security &
blockchains,” in Proceedings of 3rd Workshop on Bitcoin and Block- Privacy, 2013.
chain Research, 2016. [52] SCIPR Lab, “libsnark: a c++ library for zkSNARK proofs,” https:
[19] F. P. Junqueira, B. C. Reed, and M. Serafini, “Zab: High- //github.com/scipr-lab/libsnark.
performance broadcast for primary-backup systems,” in Proceed- [53] “Lightning network: scalable, instance bitcoin/blockchain trans-
ings of the IEEE/IFIP International Conference on Dependable actions,” https://lightning.network/.
Systems and Networks (DSN), Hong Kong, China, 2011, pp. 245– [54] S. Goldfeder, J. Bonneau, E. W. Felten, J. A. Kroll, and
256. A. Narayanan, “Securing bitcoin wallets via threshold
[20] D. Ongaro and J. K. Ousterhout, “In search of an understandable signatures,” in ACNS, 2016.
consensus algorithm,” in Proceedings of the USENIX Annual Tech- [55] V. Shoup, “Practical threshold signatures,” in EUROCRYPT, 2000.
nical Conference (USENIX ATC), Philadelphia, PA, USA, 2014, pp. [56] D. Boneh, B. Lynn, and H. Shacham, “Short signatures from the
305–319. weil pairing,” Journal of cryptology, 2004.
[21] L. Lamport, “Paxos made simple, fast, and byzantine,” in Proced- [57] E. Kokoris-Kogias, P. Jovanovic, N. Gailly, I. Khoffi, L. Gasser, and
ings of the 6th International Conference on Principles of Distributed B. Ford, “Enhancing bitcoin security and performance with strong
Systems (OPODIS). Reims, France, 2002, pp. 7–9. consistency via collective signing,” in Proceedings of 25th USENIX
[22] “Monax: The ecosystem application platform,” https://monax.io. Security Symposium (USENIX Security), Austin, TX, USA, 2016, pp.
[23] M. Vukolic, “The quest for scalable blockchain fabric: proof-of- 279–296.
work vs. bft replication,” in Open Problems in Network Security - [58] E. Syta, I. Tamas, D. Visher, D. I. Wolinsky, P. Jovanovic, L. Gasser,
iNetSec, 2015. N. Gailly, I. Khoffi, and B. Ford, “Keeping authorities honest or
[24] “Proof of authority chains,” https://github.com/paritytech/ bust with decentralized witess cosigning,” in IEEE Security &
parity/wiki/Proof-of-Authority-Chains. Privacy, 2016.
[25] “Stellar,” https://www.stellar.org/. [59] J. Behl, T. Distler, and R. Kapitza, “Scalable BFT for multi-cores:
Actor-based decomposition and consensus-oriented paralleliza-
[26] “kadena,” http://kadena.io/.
tion,” in 10th Workshop on Hot Topics in System Dependability,
[27] “Bigchaindb: a scalable blockchain database,” https://github.
com/bigchaindb/bigchaindb. HotDep ’14, Broomfield, CO, USA., 2014.
[28] “Hyperledger fabric v0.6.0,” https://github.com/hyperledger/ [60] M. Zbierski, “Parallel byzantine fault tolerance,” in Soft
Computing in Computer and Information Science. Springer, 2015,
fabric/releases/tag/v0.6.0-preview. pp. 321–333.
[29] “Hyperledger fabric v1.0.0-rc1,” https://github.com/ [61] W. Zhao, “Optimistic byzantine fault tolerance,” International Jour-
hyperledger/fabric/releases/tag/v1.0.0-rc1. nal of Parallel, Emergent and Distributed Systems, vol. 31, no. 3, pp.
[30] “Global decentralized currency,” https://litecoin.org/. 254–267, May 2016.
[31] E. B. Sasson, A. Chiesa, C. Garman, M. Green, I. Miers, E. Tromer, [62] D. Mazieres, “The stellar consensus protocol: A federated model
and M. Virza, “Zerocash: decentralized anonymous payments for internet-level consensus,” Stellar Development Foundation,
from bitcoin,” in IEEE Security & Privacy, 2014. 2015.
[32] “Multichain:open platform for blockchain applications,” https:// [63] I. Eyal, A. E. Gencer, E. G. Sirer, and R. van Renesse, “Bitcoin-ng:
www.multichain.com/. A scalable blockchain protocol,” in NSDI, 2016.
[33] JPMorgan, “Enterprise-ready distributed ledger and smart con- [64] “Proof of stake,” https://github.com/ethereum/wiki/wiki/
tract platforms,” https://github.com/jpmorganchase/quorum. Proof-of-Stake-FAQ.
[34] “Hydrachain: Permissioned distributed ledger based on eth- [65] L. Luu, V. Narayanan, C. Zhang, K. Baweija, S. Gilbert, and
ereum,” https://github.com/HydraChain/hydrachain. P. Saxena, “A secure sharding protocol for open blockchains,” in
[35] “Openchain: Enterprise-ready blockchain,” https://github.com/ CCS, 2016.
openchain. [66] “Slimcoin a peer-to-peer crypto-currency with proof-of-burn,”
[36] “Openchain: Next generation blockchain,” https://iota.org/. www.slimcoin.club/whitepaper.pdf.
[37] “Tendermint: Blockchain app development simplified,” http:// [67] “Peercoin - secure & sustainable cryptocoin,” https://peercoin.
tendermint.com/. net.
[38] C. Copeland and H. Zhong, “Tangaroa: a byzantine fault tolerant [68] “Proof of elapsed time (poet),” https://intelledger.github.io/
raft,” https://github.com/chrisnc/tangaroa. introduction.html#proof-of-elapsed-time-poet.
[39] “Dfinity,” https://dfinity.network/. [69] I. Bentov, C. Lee, A. Mizrahi, and M. Rosenfeld, “Proof of activity:
[40] L. Goodman, “Tezos: A self-amending crypto-ledger position pa- Extending bitcoin’s proof of work via proof of stake,” IACR
per,” 2014. Cryptology ePrint Archive, vol. 2014, p. 452, 2014.
[41] R. Brown, “Introducing r3 cordatm: A distributed ledger [70] S. Park, K. Pietrzak, A. Kwon, J. Alwen, G. Fuchsbauer, and
designed for financial services,” R3CEV blog, 2016. P. Gazi, “Spacemint: A cryptocurrency based on proofs of space,”
[42] “Sawtooth lake,” https://github.com/hyperledger/ IACR Cryptology ePrint Archive, vol. 2015, p. 528, 2015.
sawtooth-core. [71] M. Milutinovic, W. He, H. Wu, and M. Kanwal, “Proof of luck: an
efficient blockchain consensus protocol,” IACR Cryptology ePrint
Archive, vol. 2017, p. 249, 2017.
[72] Y. Sompolinsky and A. Zohar, “Accelerating bitcoin’s transaction
processing: fast money grows on trees, not chains,” Cryptology
ePrint Archive, Report 2013/881, 2013, https://eprint.iacr.org/
2013/881.pdf.
[73] S. Deetman, “Bitcoin could consume as much electricity as den-
mark by 2020,” https://tinyurl.com/yd8hq6n2, 2016.
[74] “Nxt: own your data, control your vote,” https://nxt.org/.
[75] E. Heilman, A. Kendler, A. Zohar, and S. Goldberg, “Eclipse
attacks on bitcoin’s peer-to-peer network,” in Proceedings of 24th
USENIX Security Symposium (USENIX Security), Washington, D.C.,
USA, 2015, pp. 129–144.
[76] A. Clement, E. Wong, L. Alvisi, M. Dahlin, and M. Marchetti,
“Making byzantine fault tolerant systems tolerate byzantine fail-
ure,” in NSDI, 2009.
[77] R. Kotla, L. Alvisi, M. Dahlin, A. Clement, and E. Wong, “Zyzzyva:
speculative byzantine fault tolerance,” in SOSP, 2007.
[78] S. Liu, P. Viotti, C. Cachin, V. Quema, and M. Vukolic, “Xft:
Practical fault tolerance beyond crashes,” in OSDI, 2016.
[79] A. Miller, Y. Xia, K. Croman, E. Shi, and D. Song, “The honeybad-
ger of bft protocols,” in CCS, 2016.
[80] Intel, “Intel software guard extensions,” https://software.intel.
com/en-us/sgx.
[81] T. Alves and D. Felton, “Trustzone: Integrated hardware and soft-
ware security, enabling trusted computing in embedded
systems,” White paper.
[82] B. Chun, P. Maniatis, S. Shenker, and J. Kubiatowicz, “Attested
append-only memory: making adversaries stick to their word,”
in Proceedings of the 21st ACM Symposium on Operating Systems
Principles (SOSP), Stevenson, Washington, USA, 2007, pp. 189–
204.
[83] J. Behl, T. Distler, and R. Kapitza, “Hybrids on steroids: Sgx-based
high performance bft,” in Eurosys, 2017.
[84] Y. Gilad, R. Hemo, S. Micali, G. Vlachos, and N. Zeldovich,
“Algorand: Scaling byzantine agreements for cryptocurrencies,”
in SOSP, 2017.
[85] “Interledger: the protocol for connecting blockchains.” https://
interledger.org/.
[86] L. Luu, D.-H. Chu, H. Olickel, P. Saxena, and A. Hobor, “Making
smart contracts smarter,” in CCS, 2016.
[87] A. Kosba, A. Miller, E. Shi, Z. Wen, and C. Papamanthou, “Hawk:
the blockchain model of cryptography and privacy preserving
smart contracts,” in CCS, 2016.
[88] “BlockBench: private blockchains benchmarking,” https://github.
com/ooibc88/blockbench.
[89] B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears,
“Benchmarking cloud serving systems with YCSB,” in Proceedings
of the 1st ACM Symposium on Cloud Computing (SoCC),
Indianapolis, Indiana, USA, 2010, pp. 143–154.
[90] M. J. Cahill, U. Ro¨ hm, and A. D. Fekete, “Serializable isolation
for snapshot databases,” in Proceedings of the ACM SIGMOD
Interna- tional Conference on Management of Data (SIGMOD),
Vancouver, BC, Canada, 2008, pp. 729–738.
[91] Crypti, “A decentralized application platform,” https://crypti.me.
[92] A. Dinh, J. Wang, S. Wang, W.-N. Chin, Q. Lin, B. C. Ooi, P. Ruan,
K.-L. Tan, Z. Xie, H. Zhang, and M. Zhang, “UStore: a distributed
storage with rich semantics,” https://arxiv.org/pdf/1702.02799.
pdf.
[93] K. Tan, Q. Cai, B. C. Ooi, W. Wong, C. Yao, and H. Zhang, “In-
memory databases: Challenges and opportunities from software
and hardware perspectives,” SIGMOD Record, vol. 44, no. 2, pp.
35–40, 2015.
[94] H. Zhang, G. Chen, B. C. Ooi, K. Tan, and M. Zhang, “In-memory
big data management and processing: A survey,” IEEE
Transactions on Knowledge and Data Engineering, vol. 27, no. 7, pp.
1920–1948, 2015.
[95] A. Dragojevic, D. Narayanan, E. B. Nightingale, M. Renzelmann,
A. Shamis, A. Badam, and M. Castro, “No compromises: dis-
tributed transactions with consistency, availability, and perfor-
mance,” in Proceedings of the 25th Symposium on Operating
Systems Principles SOSP, Monterey, CA, USA, 2015, pp. 54–70.