See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/333590995

Security Vulnerabilities in Ethereum Smart Contracts

Conference Paper · July 2018

DOI: 10.1109/Cybermatics_2018.2018.00182

CITATIONS READS
135 6,549

2 authors, including:

Mariusz Nowostawski
Norwegian University of Science and Technology
130 PUBLICATIONS 1,969 CITATIONS

SEE PROFILE

All content following this page was uploaded by Mariusz Nowostawski on 18 November 2019.

The user has requested enhancement of the downloaded file.

 Security Vulnerabilities in Ethereum Smart Contracts

Ardit Dika and Mariusz Nowostawski

Norwegian University of Science and Technology | NTNU
Information Security and Computer Science Departments, Gjovik, Norway
e-mail: [email protected], [email protected]

Abstract— allows developers to build a variety of applications, ranging

Ethereum provides an open, global computing platform, from simple wallets to financial systems, energy trading sys-
that allows the exchange of value, automated and enforced tems or new and novel crypto currencies. Instead of building
workflows, and the development of general purpose applica- a separate blockchain for each use case or application, a
tions and libraries. Smart contracts present a foundation for variety of use cases can be done through technology known
the computational capabilities of the Ethereum network. Moti- as smart contracts. Ethereum can be considered a relatively
vated by the known security breaches and recurring financial new and highly experimental platform, both because of the
losses due to smart contracts vulnerabilities, we review the time when it was introduced (July 20152 ), as well as its abil-
field of security of smart contract programming and provide ity to create distributed applications with a Turing-complete
a comprehensive taxonomy of all known security issues. We programming language running in a decentralised, peer-to-
achieve that by a thorough review of known vulnerabilities. In peer blockchain platform. A general-purpose programming
this work we also review the security code analysis tools used language in a blockchain-based platform creates the op-
to identify known vulnerabilities. We conduct the investigation portunity for implementing a wide range of decentralised
of security code analysis tools on Ethereum by assessing their applications. On the other side, it creates opportunities for
effectiveness and accuracy on known issues on a representative abuse.
sample of vulnerable contracts. We have used 21 clean, and 24
In this work, we investigate the known security vulner-
vulnerable contracts and four security tools: Oyente, Securify,
abilities of smart contracts and provide an updated in-depth
analysis of existing smart contract vulnerabilities. In the
Remix, and SmartCheck, to assess the quality of contemporary
second part, we investigate the security code analysis tools
security analysis tools specific to Ethereum. The results indicate
used to identify vulnerabilities and bugs in smart contracts.
that there are overall inconsistencies between the tools in
To the best of our knowledge this is a unique contribution
respect to different security properties. SmartCheck outper-
to the field as the analysis of existing tools has not been
formed the other tools in terms of effectiveness, whereas Oyente
conducted before.
performed the best in terms of accuracy. Furthermore, based
on the limitations we identified, we propose improvements
within the user interfaces, interpretation of results, and, most 2. Problem Description
importantly, an enhanced list for vulnerability checks.
In addition to expressing business logic and handling
different, sometimes heavy computational tasks, based on
1. Introduction the Ether price3 , smart contracts also present a foundation
for possessing expensive digital assets. This means that there
Blockchain technology and cryptocurrencies have expe- are currently financial and semi-financial smart contracts
rienced a steady increase of attention from academia and the which are worth thousands and millions of dollars. Due to
industry alike [1]. Blockchain technology represents a fully those reasons, smart contracts and the Ethereum platform are
distributed public ledger and a peer-to-peer platform which continuously a target for adversaries and manipulators. As a
makes use of cryptography to securely host applications, result, one of the main and active research areas is within the
transfer digital currencies, messages, and store data [2]. security drawbacks of high-level programming languages
One of the most popular blockchain platforms as of used for smart contract programming. The research com-
March 2018, based on the current cryptocurrency market munity proposes further research and development work in
capitalization1 , is Ethereum. Vitalik Buterin, the main in- formal verification, techniques for analyzing smart contracts,
ventor of Ethereum, in one of the panel discussion explains and defensive programming techniques.
Ethereum as a general purpose blockchain, meaning that What Ethereum and other popular blockchain platforms
the Ethereum network is able to accomodate algorithms have in common, is the publicly visible data. This is
expressed in a general purpose programming language. This
2. Link: https://blog.ethereum.org/2015/07/30/ethereum-launches/
1. Cryptocurrency Market Capitalisation: https://coinmarketcap.com/ 3. $705 as of the time of writing: https://ethereumprice.org/
a result of having a decentralized peer-to-peer network one specific vulnerability, for example, on timed commit-
and distributed ledger among thousands of nodes4 . Hence, ments [10] and smart contract altering possibilities [11].
Ethereum is referred to as The World Computer5 . Regard- One of the most established taxonomies in this area is
less of the positive impact and many benefits that this the one provided by Atzei et al. [4]. The levels chosen to
approach has on Ethereum and generally in any other public represent the vulnerabilities are (i) focused on the program-
distributed blockchain, it presents serious challenges from ming language Solidity, (ii) specific to the underlying imple-
the security perspective. Implementing specific use cases of mentation of the EVM, and (iii) specific to the Blockchain
smart contracts, considering the fact that the complete source itself. Note, Solidity vulnerabilities are also applicable for
code of an application is publicly visible from anyone in the other high-level programming languages in Ethereum. This
network, and making sure that the code is correctly validated taxonomy seems to properly classify all vulnerabilities based
and verified. on their level, since a newly discovered vulnerability falls
One of the key characteristics of Ethereum’s platform is in one of these categories. Therefore, we have used it as a
that once you deploy your smart contract in the blockchain, basis for our own taxonomy.
you cannot modify or alter it. This characteristic can both Another taxonomy is provided by Alharby and
be seen as advantageous and disadvantageous. The advan- Moorsel [5]. This one is based on a systematic study of
tage is that it represents a trustworthy platform where the current research topics related to smart contracts, and it
developers cannot modify the smart contract once they identifies four key smart contract issues; codifying issues,
have deployed it, with the sole purpose of gaining illegal security issues, privacy issues, and performance issues. By
profit and misleading the users. The disadvantage lies in codifying issues they refer to the challenges that are related
the unusual development challenges, such as the inability to with the development of smart contracts. This could be
easily patch discovered vulnerabilities in already deployed generally called validation issues. The security issues mean
contracts. bugs or vulnerabilities (verification), and privacy issues
Due to the above issues, a significant number of smart are related to unintentional information disclosures. Lastly,
contracts are considered to be vulnerable. In 2016, a sym- performance issues are related to the challenges that affect
bolic execution analysis tool (Oyente) was developed by Luu the ability of blockchain to scale. [5]
et al. [3], which analysed all smart contracts in the Ethereum Other research articles refer to security vulnerabilities in
blockchain at that time, in order to identify potential vul- general, without any categorisation, such as, in [3], where
nerabilities6 . Their results state, that at that time, 45% of they discuss only severe vulnerabilities. In [7], where Bu-
19,366 smart contracts in total were vulnerable with at least terin with the community’s help created a crowd sourced list
one security issue [3]. of the major bugs with smart contracts, and in [6] through
Our work provides insight into the smart-contract secu- a university course for smart contract programming, they
rity domain. Through a thorough research and a comprehen- exposed numerous common pitfalls and vulnerabilities.
sive experiment on security vulnerabilities and code analysis In addition to the above, more general research on
tools we propose up-to-date taxonomy of vulnerabilities, vulnerabilities, there is research on specific vulnerability, for
their architectural classification, in conjunction with their example privacy preserving issues. It represents a category
severity level. In addition, we conduct an experiment on of development challenges to keep critical functions secret,
several security tools to assess their accuracy, effectiveness, apply correctly cryptographic protocols, and avoid disclos-
and consistency. This generates results, such as false positive ing data that should not have been public in the first place. A
and false negative rates and an overall discussion on how research on ‘replacing paper contracts with Ethereum smart
effective these tools are in analysing the smart contracts contracts’ finds out what kind of criteria Ethereum needs to
from the data collected in this study. fulfil to be properly applied on replacing paper contracts [8].
They conclude that due to a large privacy setback it is not
3. Related Work yet recommended to replace legally-enforceable agreements
with smart contract applications [8]. This is as a result of
We focus on three aspects in the area of smart contracts: the private information these papers (agreements) hold and
security vulnerabilities, smart contract attacks/incidents, and the damage that could be done if they become public or
preventive methodologies. In order to identify the current if the blockchain does not work as intended on preserving
state of the art, we have conducted literature review. Some privacy. According to [5], lack of transactional privacy and
of the existing research is focused on security vulnerabilities lack of data feeds privacy are two issues correlated with the
in general [3], [4], [5], [6], [7]. Others are focused on privacy preserving category.
specific vulnerabilities and smart contract challenges, such A similar research on the issue of privacy-preserving
as privacy [8], [9]. There are also articles focused only on is conducted by Kosba et al. [9]. They highlighted the
significant importance of privacy in smart contract appli-
4. Ethereum as of May, 2017 has nearly 25,000 nodes. Link: http://www. cations generally in blockchain technologies, not only in
trustnodes.com/2017/05/31/ethereum-now-three-times-nodes-bitcoin
Ethereum. For a solution to this issue, they have proposed a
5. Ethereum: the World Computer: https://www.youtube.com/watch?v=
j23HnORQXvs decentralized smart contract system, Hawk, which does not
6. Vulnerabilities that Oyente is able to identify: transaction-ordering store financial transactions in the blockchain and saves the
dependence, timestamp dependence, mishandled exceptions and reentrancy. developers from implementing any cryptographic function-
ality [9]. Juels et al. [12], also investigated the leakage of of contract (A). The use of checks-effects-interactions8 is
confidential information and theft of cryptographic keys for recommended as a solution to avoid this vulnerability.
smart contracts used in criminal activities. tx.origin (transaction origin) is the identity of the user
One of the most prominent vulnerabilities of Ethereum who initiated a chain of interactions between contracts. The
is considered to be the timestamp dependency. Boneh usage of tx.origin for authorisation is discouraged, as it is
and Naor [10], introduce and construct timed commitment easy for attacker to spoof that value in a contract. tx.origin
schemes which are proposed as a solution for this vulnera- must be used with extra care not to allow an attacker
bility. Their proposed solution could be applied when two obtaining leveraged privileges in the contract.
mutually suspicious parties wish to exchange signatures on Callstack depth exception. It is possible to makes an
a contract. external call to fail because it exceeds the maximum call
Another issue that has been tackled in the literature is the stack of 1024 [15]. As a result, the call will fail, and if
gas-costly pattern, more specifically, the under-optimised the exception is not properly handled by the contract, the
smart contracts that consume more gas than necessary. A attacker can force the contract to produce an output which
research investigation in this regard is done by Chen et suits them.
al. [13], in which they identified 7 gas costly patterns and Timestamp dependence presents a common vulnerabil-
grouped them into two categories. They also developed a ity favouring a malicious miner. If a contract is using it for
tool, named Gasper, focused only on identifying gas-costly a critical check, the miner can manipulate the timestamp
patterns by analysing the smart contracts’ bytecode [13]. for a few seconds by changing the output to be in its
Their results indicate that over 80% of 4240 smart contracts favour [16]. However, this vulnerability is severe only if
analysed, suffer from one of the gas-costly patterns [13]. used in critical components of a contract and requires miners
In addition, there are specific challenges in communi- to have sufficient computing facilities.
cating with external services (Oracles7 ). Zhang et al. [14] Transaction-ordering dependence refers to the idea
presented an authenticated data feed system called Town that the user can never be sure of the order of transactions.
Crier, which enables smart contracts to consume data from For example consider a smart contract which offers a reward
outside the blockchain while preserving confidentiality with for solving a puzzle. Once a user solves the puzzle and sub-
encrypted parameters. mits the transaction, at the same moment the smart contract
There is a significant number of issues, some of which owner can reduce (or completely remove) the reward. There
have known solutions. Some of the solutions proposed re- is a probability that the transaction that reduces or removes
quire for blockchain upgrades, meaning that all the nodes the reward is processed first. In this case, the owner gets an
have to upgrade their version in order to solve a particular answer for the puzzle, and the solver (user) does not get the
issue, or they are proposed as a separate platform on top of reward.
a blockchain. This makes it challenging in rolling out the The use of external calls is considered to be by default
actual fixes. risky [16], because adversaries can execute malicious code
in that external contract. Therefore, it is recommended to
possibly avoid external calls ("calls to the unknown") in
4. Ethereum Vulnerabilities general or treat those calls as potentially risky and take
precautions, such as, use send instead of call_value(), favor
pull over push for external calls, and handle errors (check
In this section we provide a brief explanation for each
the return value) [16].
of the security vulnerabilities that has been included into the
taxonomy. Some of these vulnerabilities have already been Unchecked-send bug is part of the exception disorders
known for a while, therefore we follow well-established or mishandled exceptions. This class of vulnerabilities is
naming conventions. Some small self-explanatory issues also referred to as "send instead of transfer. ’Transfer’
with the Ethereum smart contracts are excluded from the automatically checks for the return value, whereas using
list and the main focus of this section is (mostly) within the ’send’ you have to manually check for the return value, and
severe vulnerabilities. throw an exception if the send fails. Not doing so, can lead
to an attacker executing malicious code into the contract and
Reentrancy is considered to be one of the most severe draining the balance. Overall, the consequences are similar
vulnerability. It has been first recognised by the biggest at- to the reentrancy and call to the unknown vulnerability.
tack ever made (TheDAO hack). The reentrancy vulnerabil-
DoS is explained by SmartCheck9 as a situation in which
ity relies on the interaction between two smart contracts, (A)
conditional statement (if , for , while) depends on an external
and (B). If through an interaction from a contract (A) with
call: the callee may permanently fail (throw or revert),
another contract (B), (A) handing over control to contract
preventing the caller from completing the execution [17]. An
(B) makes it possible for (B) to call back into (A) before
attacker can cause inconvenience by supplying the contract
the first initiated interaction is completed, contract (B) can
with data that is expensive to process, thereby preventing
effectively retrieve multiple refunds and empty the balance
8. First subtract the value from the contracts’ balance then send the Ether,
7. Oracle: A reliable connection between Web APIs and smart contracts, and check for the return value.
since smart contracts cannot fetch external data on their own. 9. SmartCheck: https://smartcontracts.smartdec.net/
others to interact with it. This vulnerability is closely related Relying on code coverage, SolCover measures and describes
to the external calls vulnerability and to prevent this form the degree of overall testing in a smart contract. Even
of attack from happening, we need to handle properly any though, it does not serve as a mechanism to identify specific
throw exceptions from external calls, and also, avoid looping vulnerabilities, it could be argued that it creates a more
behaviour. secure environment with the philosophy that more tests
Blockhash usage similarly to the block timestamp, it is provide improved security metrics.
not recommended to be used on crucial components, for the HackThisContract12 is a crowdsourcing experimental
same reason as with the timestamp dependency, because the website that encourages developers to test smart contracts
miners, to some degree, can manipulate it and change the before deployment by uploading it on their website. Other
output to their favour. This is particularly pronounced when developers, with their own techniques, will try and exploit
blockhash is used as a source of randomness. possible vulnerabilities. Additionally, they provide a list of
Gasless send makes a transaction to fail if not enough vulnerable smart contract examples which the developers
gas is provided for a specific call. The maximum gas limit should not follow. Overall, with the sole purpose of deploy-
on the network can vary over time based on the transaction ing secure smart contracts and mitigate severe issues in a
fees10 . It is important to throw an exception if a failure pre-deployment phase.
based on the gas consumption happens. Also, it is important Security audits are considered to be the most effective
to develop functions that do not require too much gas, not way of identifying vulnerabilities in a pre-deployment phase.
only for the purpose of failing, but also for the sole purpose Experienced blockchain developers and specialised teams
of mitigating the costs of executing the contract. carefully investigate the smart contract manually and auto-
Other vulnerabilities include; immutable bugs matically to identify vulnerabilities. Despite the fact that it
(e.g. wrong constructor name) which refer to a bug or a code might be the most secure method for preventing deployment
mistake which cannot be altered after deployment/discovery, of vulnerable smart contracts, it is not popular because of
the use of untrustworthy data feeds, failure to keep the high costs and time it takes to conduct them13 . Currently
secrets or in other words failure to apply cryptography there are many firms that do smart contract security audits:
and as a result expose crucial functions or values, the Zeppelin, Solidified14 , SmartDec15 , and DejaVu16 .
challenge to generate randomness, style guide violation, Other preventive methodologies include staying up-to-
and others. date with Ethereum upgrades and especially with the attacks
Existing defensive programming as well as traditional that happen over time, since they may discover a new vul-
secure programming techniques apply to smart contract nerability. Also, it is of a vital importance to follow a list of
programming. Note however, that many of vulnerabilities recommendations for secure smart contracts once you start
are unique to smart contracts. Some of it has to do with the developing, such as the extensive list by ConsenSys [19].
novel approach of building smart contract applications in a Oyente. Oyente is known to be the first and most popular
public blockchain. Since, so far, developers are used to a security analysis tool. It was developed by Luu et al. [3]
more traditional way of developing software. Traditionally, and is one of the few tools presented in a major security
developers do not have to worry about many of these issues, conference, Ethereum Devcon17 . Oyente leverages symbolic
because the traditional systems provide certain guarantees execution to find potential security vulnerabilities, including
that a public blockchain cannot. here transaction-ordering dependence, timestamp depen-
dence, mishandled exceptions and reentrancy. The tool
5. Preventive Methodologies can analyze both Solidity, and the bytecode of a smart con-
tract. In its early stage it could have been used only through
ZeppelinOS is an operating system for smart contract
a command line interface. Currently, it provides a more user-
applications developed by Zeppelin Solutions [18]. As re-
friendly web-based interface. It is worth mentioning that
ferred to by Zeppelin Solutions, ZeppelinOS is "an open-
it is the only tool that describes its verification method to
source, distributed platform of tools and services on top of
eliminate false positives [3].
the EVM to develop and manage smart contract applications
securely". Their system is composed of four components, Securify. Securify18 is a web-based security analysis
kernel, scheduler, marketplace, and off-chain tools. In other tool and, according to their website, it is the first security
words, Zeppelin introduces a novel approach in developing analysis tool that provides automation (to enable everyone
smart contracts by using already developed and secure smart to verify smart contracts), guarantees (for finding specific
contracts (i.e. libraries). The off-chain component provides vulnerabilities), and extensibility (to capture any newly dis-
numerous tools like debugging, testing, deployment and
12. Link: http://hackthiscontract.io/
monitoring. Based on their team, these tools will enhance 13. Based on a community discussion in Reddit, a smart contract secu-
the development process (better, easier, robust), and gen- rity audit costs between $20k-$60k. Link: https://www.reddit.com/r/ethdev/
erally will help to provide a more secure smart contract comments/6pdgvd/how_much_does_a_smart_contract_audit_cost/
environment. 14. Link: https://solidified.io/
SolCover11 provides code coverage for Solidity testing. 15. Link: https://smartcontracts.smartdec.net/
16. Link: http://www.dejavusecurity.com/services/
10. Link:http://www.kingoftheether.com/contract-safety-checklist.html 17. Link: https://www.youtube.com/watch?v=bCvh6ED-cj0
11. Link: https://github.com/sc-forks/solidity-coverage 18. Link: https://securify.ch/
covered vulnerability). Securify uses formal verification but vulnerability checks are presented as work in progress, such
also relies on static analysis checks. The security issues as, timestamp dependence, transaction-ordering dependence,
that it covers are: transaction reordering, recursive calls, and information exposure25 .
insecure coding patterns, unexpected ether flows, and use Gasper. Gasper is a security tool developed by Chen
of untrusted input. However, the recursive calls, unexpected et al. [13], which is not released yet. However, from their
ether flows, and part of the insecure coding patterns checks research paper, we already know that it is focused only on
are locked (require full access)19 . identifying gas costly programming patterns in a smart
Remix. Remix20 is a web-based IDE that facilitates contract through a command line interface. It runs analysis
writing Solidity smart contracts, deploying and running only for the bytecode. Moreover, they have discovered seven
them. A debugger and a testing environment (test-blockchain gas costly patterns, and grouped them into two categories.
network) are integrated. Additionally, it serves as a security Gasper also relies on symbolic execution to cover all reach-
tool by analyzing the Solidity code only, to reduce coding able code-blocks by disassembling its bytecode using disasm
mistakes and identify potential vulnerable coding patterns. (disassembler). So far, they only cover the gas costly patterns
Some of the vulnerabilities that it identifies are: tx.origin from the first category that they have discovered, the rest is
usage, timestamp dependence, blockhash usage, gas costly work-in-progress. [13]
patterns, check effects (reentrancy)21 . Remix security anal-
ysis rely on formal verification (deductive program verifica- 6. Methodology and Experiments
tion and theorem provers).
SmartCheck. SmartCheck22 is also a web-based secu-
rity code analysis tool provided by SmartDec team23 . Smart- Vulnerability
Unpredictable state (dynamic libraries)
Severity level
2

Blockchain
Generating randomness 2-3
Dec is a company focused on security audits, analysis tools Time constrains / Timestamp dependence 1-3
Lack of transactional privacy 1-3
and web development. Recently (November, 2017), they Transaction-ordering dependence 2-3
Untrustworthy data feeds (oracles) 3
released a beta version of their security tool, SmartCheck.

EVM
Immutable bugs/mistakes 3
Ether lost in transfer 3
It automatically checks for vulnerabilities and bad coding Gas costly patterns 1-2
Call to the unknown 3
practises. In addition to that, it highlights the vulnerability Gasless send 3
Exception disorders / Mishandled exceptions / Unchecked-send bug 3
(e.g. line of code), gives an explanation of the vulnera- Type casts 2
bility, and a possible solution to avoid a particular secu- Reentrancy 3
Solidity

Unchecked math (Integer over- and underflow) 1-2

rity issue. Their analysis uses Solidity code and it is not Visbility / Exposed functions or secrets/ Failure to use cryptography
‘tx.origin’ usage
2-3
3
stated which specific methodology they use to identify the ‘blockhash’ usage
DoS
2-3
3
vulnerabilities (e.g. symbolic execution, formal verification, ‘send’ instead of ‘transfer’
Style violation
1-2
1
etc.). Each vulnerability discovered is shown in correlation Redundant fallback function
TABLE 1. TAXONOMY OF V ULNERABILITIES
1

with its severity level. Some of the severe vulnerabilities

they identify are: DoS by external contract, gas costly
patterns, locked money, reentrancy, timestamp dependency, We have assessed the smart contract security tools based
tx.origin usage, and unchecked external call. Additionally, on their:
SmartCheck identifies many other vulnerabilities with low • effectiveness - check how many smart contract prob-
severity (warnings), such as, compiler version not fixed, lems the tools were able to find from our data set
style guide violation, and redundant functions. • accuracy - assessing the correctness of the results
F* Framework. F*, from Microsoft Research, presents they produce, based on false positive and false neg-
a framework for analyzing the runtime safety and the func- ative rates
tional correctness of Ethereum smart contracts, outlined • consistency - assessed in security tools that analyse
by Bhargavan et al. [20]. It relies on formal verification, both bytecode and Solidity, and check if there is any
by translating Solidity or bytecode into F* (a functional inconsistency27 .
programming language) and then identifying potential vul-
nerabilities, such as, reentrancy and exception disorders. We collected known vulnerabilities based on literature
Mythril. Mythril is a recently released experimental search and online resources, and stayed up-to-date with any
security analysis tool from ConsenSys24 . Through a com- related web-articles or blogs and Reddit forums that were
mand line interface, it is able to analyze bytecode, and assessing smart contract security issues, between February
by installing solc (command line compiler) it also analyses and October 2017. Additionally, we have used group chats
Solidity code. So far, it is able to identify a variety of vulner- (Slack channels, Gitter) and e-mails to communicate with
abilities, such as, unprotected functions, reentrancy, integer developers or users for a specific security tool.
overflow/underflow, and tx.origin usage. Some other severe Audited Smart Contracts. In order to assess the false
positive rates we need secure/trusted and tested smart con-
19. As of October, 2017 tracts which are considered to be bug-free or at least without
20. Link: https://remix.ethereum.org/
21. Link: https://remix.readthedocs.io/en/latest/analysis_tab.html 25. Link: https://github.com/b-mueller/mythril/blob/master/security_
22. Link: http://tool.smartdec.net checks.md
23. Link: https://smartcontracts.smartdec.net/ 27. For example, if a tool produces some results with the bytecode of a
24. Link: https://github.com/b-mueller/mythril/ smart contract and other with the Solidity source code of the same contract.
 Security Timestamp Mishandled Immutable tx.orgin Gas costly Blockhash
ReEntrancy TOD26
Tool dependency exceptions Bugs usage patterns usage
Oyente X X X X X X X X
Remix X X X X X X X X
F* X X X X X X X X
Gasper X X X X X X X X
Securify X X X X X X X X
S. Analysis X X X X X X X X
SmartCheck X X X X X X X X
Imandra n/a n/a n/a n/a n/a n/a n/a n/a
Mythril X X X X X X X X
TABLE 2. T OOLS /V ULNERABILITIES M ATRIX

Smart
Source (* = https://blog.zeppelin.solutions)
any severe security vulnerability. For that, we decided to use Hacker
Contract
*/ethercamps-hacker-gold-hkg-public-code-audit-b7dd3a2fe43b
smart contracts which were previously audited. We chose Gold (HKG)
ArcadeCity
Zeppelin28 . We have collected 28 audited smart contracts (ARC)
Golem
*/arcade-city-arc-token-audit-9071fa55a4e8

*/golem-network-token-gnt-audit-edfa4a45bc32
in total from Zeppelin, starting from the one audited first, Network
ProjectKudos */ethercamps-projectkudos-public-code-audit-179ee0c6672d
up until the last one (October 23, 2017). We did a manual EtherCamp’s
DSTC
*/ethercamps-decentralized-startup-team-public-code-audit-65f4ce8f838d
check for each security audit to dismiss a smart contract SuperDAO
Promissory
*/draft-superdao-promissory-token-audit-2409e0fe776c
which had one of the following cases: SuperDAO
ConstitutionalDNA
*/draft-superdao-promissory-token-audit-2409e0fe776c
ROSCA */wetrust-rosca-contract-code-audit-928a536c5dd2
• is written in a programming language other than Matchpool
*/matchpool-gup-token-audit-852a70330f2
GUP
Solidity, iEx.ec
*/iex-ec-rlc-token-audit-80abd763709b
RLC
• is identified with severe vulnerabilities from Zep- Cosmos */cosmos-fundraiser-audit-7543a57335a4
Blockchain
pelin, and not updated afterwards, Capital (BCAP)
*/blockchain-capital-token-audit-68e882d14f0

• is used for token pre-sale29 , WingsDAO

Moeda
*/wingsdao-token-audit-f39f800a1bc1
*/moeda-token-audit-ac72944caa6f
Basic
• very recent security audit (not updated, nor pub- Attention
*/basic-attention-token-bat-audit-88bf196df64b
Storj */storj-token-audit-32a9af082797
lished). Metal */metal-token-audit-d7e4dbf17bcf
Decentraland
*/decentraland-mana-token-audit-ee56a6bca708
After doing this data clean-up, we ended up with a total MANA
Tierion
*/tierion-presale-audit-ec14b91c3140
of 21 security audited smart contracts. Table 3 provides a list Pre-sale
Serpent
*/serpent-compiler-audit-3095d1257929
of the data-set for this category, where the seven highlighted Compiler
Hubbi */hubii-token-audit-227c0adf50ea
in red are the discarded smart contracts which were not taken Tierion
Kin
*/tierion-network-token-audit-163850fd1787
*/kin-token-audit-121788c06fe
into consideration. Additionally, each one has a link to the Render
Fuel
*/render-token-audit-2a078ba6d759
*/fuel-token-audit-30cc02f257f5
corresponding source on Zeppelin’s blog. Lastly, the source Enigma
Global
*/enigma-token-audit-91111e0b7f8a

code for each smart contract is collected using EtherScan30 . Messaging

Ripio
*/global-messaging-token-audit-865e6a821cd8
*/ripio-token-audit-abe43b887664
Vulnerable Smart Contracts. Vulnerable smart con- TABLE 3. AUDITED S MART C ONTRACTS C OLLECTION
tracts are used to identify the false negative rates, as well
as the gaps, i.e. the vulnerabilities which are not covered
by the tools. We have used existing research articles, online
resources, and community discussions assessing attacks and In total, 23 vulnerable and 21 audited smart contracts
bugs in smart contracts, to assemble a list of vulnerable are analyzed with the four tools. Since each security tool
smart contracts [3], [7], [21], [22], [23]. Table 4 provides identifies different vulnerabilities, not all vulnerable smart
the list of contracts used as our data-set. contracts were fit to be tested with all the tools. However,
Smart contracts which are synthetic are labelled with we decided to analyze all vulnerable smart contracts, in
the term "Sample" on their name. Moreover, the ones high- order to capture a general analysis on how many vulner-
lighted in red are not taken into consideration, because i) abilities each tool is not able to identify. This also gives us
Suicide function has been called and their code is no longer an insight within the possible future improvements of the
available (two cases) or ii) Smart contracts that allow their security tools. The data analysis consists of four different as-
owners to withdraw the contract funds, are removed because sessments: effectiveness, accuracy, consistency, and overall
it was considered to be more of a trust issue rather than a assessment. With the exception of the Overall assessment,
bug or vulnerability. Therefore, out of 28 smart contracts all other three assessments have a clear data analysis process
in total, after clean-up we ended up with 2431 . The tools and an evaluation method.
which are chosen for the experiment are: Oyente, Securify, Effectiveness. Generally, the effectiveness of the tools
Remix and SmartCheck. All four tools have a web-based is assessed based on the percentage of the smart contracts
user interface. in total that the tools were able to analyse. The nature of
28. More than $450 million have been raised by smart contracts that the data-set consists of different type of smart contracts,
have been audited by Zeppelin. including here; secure, vulnerable, old compiler versions,
29. They are mostly temporary smart contracts used to crowd-fund an grand scale, small scale, and samples. Additionally, the
organisation.
30. Link: https://etherscan.io/
symbolic execution methodology predominantly used to
30. TOD: Transaction-ordering dependence identify vulnerabilities is rather complex, since it analyses
31. Stricly speaking, 23, since two smart contracts have either bytecode the code without any known input and also loops through
or Solidity available, not both. the blockchain to cover all possible behaviours. Therefore,
 Smart
Vulnerability
contract name
TheDao Re-entrancy
the code for that specific vulnerability or they have
SimpleDao Sample compiler version 0.3.1
Re-entrancy,
call to the unkown suppressed it – it is removed from the false positives
Re-entrancy,
SimpleDao Sample compiler version 0.4.2
call to the unkown results.
Unchecked-send
King of the Ether game (KoET)
bug, Gasless send, Mishandled exception
Gasless
• Additionally, a manual analysis following a list of
KotET Sample compiler version 0.3.1
send
Gasless
recommendations for smart contract security [19] is
KotET Sample compiler version 0.4.2

GovernMental (PonziGovernmental)
send
Unchecked-send conducted and the line where the vulnerability is
bug, Call-stack limit
GovernMental simplified sample 0.3.1
Immutable identified is checked manually to verify if it is false
bugs, exception disorder, call-stack limit, unpredictable state
Rubixi
Immutable
bugs, wrong constructor name
positive.
Type
FirePonzi

Parity Multisig 1
casts (intentional scam)
Unintended The other approach in regards to accuracy is the false
function exposure
Parity Multisig 2 - Suicide Function called
Unintended negative assessment. This is done through the vulnerable
function exposure
Parity Multisig 3- Suicide Function called
Unintended
function exposure
contracts that have at least one vulnerability. If the tools
GoodFellas
Typo
(wrong constructor name)
state that they are able to identify a specific vulnerability
StackyGame
Typo
(wrong constructor name)
and they fail to do so, it is considered a false negative. The
Contract
DynamicPyramid
that does not refund results obtained from this experiment have two possibilities
Contract
GreedPit
that does not refund
Contract
of failure:
NanoPyramid
that does not refund
Tomeka
Contract
that does not refund
• False Positive when the tool identifies a vulnera-
Double3
Allows
the contract owner to withdraw all the funds bility in an audited smart contract, and the manual
Allows
TheGame
the contract owner to withdraw all the funds inspection does not confirm it.
Call-stack
ProtectTheCastle
limit, Withdraw option
Public
• False Negative when the security tool does not find
RockPaperScissors (RPS)
moves
Blockhash
a specific vulnerability in a vulnerable contract.
SmartBillions
bug
Unchecked-send
EtherPot
bug
Timestamp Security Bytecode Solidity
TheRun Method CLI33 WUI34
dependence Tool analysis analysis
Keeping Oyente Symbolic execution X X X X
OddsAndEvents Compiler 0.3.1 Sample
secrets Remix Formal verification X X X X
Keeping F* Framework Formal verification X X X X
OddsAndEvents Compiler 0.4.2 Sample
secrets
Gasper Symbolic execution X X N/A N/A
TABLE 4. V ULNERABLE S MART C ONTRACTS C OLLECTION Securify Formal verification X X X X
Simple
Heuristics X X X X
Analysis35
SmartCheck N/A X X X X
Imandra
Formal verification N/A - paid access
Contracts
Concolic
the security tools and the methodology are themselves prone Mythril
testing (symbolic execution)
X X X X

to errors and failures. TABLE 5. TAXONOMY OF T OOLS

Accuracy. Assessing just the effectiveness of the tools
does not necessarily show us how accurate the results are. Table 5 provides an overview of the generated taxonomy
Therefore, it is crucial to assess the accuracy of the results for security code analysis tools. The categorisation is based
that the tools produce. Accuracy is assessed through the false on their similarities, such as, the methodology they use
positive and false negative rates. Initially, this assessment (highlight) to identify security issues (symbolic execution,
idea came from Zhang et al. [24], in which they evaluate the formal verification), which code analysis they are able to
anti-phishing tools with the same methodology, using 200 perform (bytecode, Solidity), and their user interface (CLI,
verified phishing URLs (in our case vulnerable contracts) WUI). As it can be seen, for some tools we have partial
and 516 legitimate URLs (in our case audited contracts), to information, either because the tool is not released yet
test the performance of 10 popular phishing tools (in our (Gasper), the methodology is not stated in their documen-
case 4 popular Ethereum security code analysis tools). tation (SmartCheck), or the tool requires paid access for
First, we ran 21 audited smart contracts in each tool. additional information and usage (Imandra Contracts).
Based on the results obtained and the severity level of Compared to the taxonomy provided by Hildenbrandt et
vulnerabilities, we decided to manually analyse only five al. [22], which covers all Ethereum software quality tools,
vulnerabilities32 . Other vulnerabilities are not considered for our taxonomy is only focused on security tools used to
manual analysis, either because they cannot be manually identify vulnerabilities/bugs in smart contracts. The security
analysed (e.g. gas costly patterns), the security audit firm tools stated here use symbolic execution and formal veri-
does not cover them, or they are vulnerabilities with low fication as a methodology to identify vulnerabilities. These
severity (e.g. useful warnings or style violations). The man- two methodologies, generally, are used interchangeably and
ual analysis is conducted as follows: in combination. Table 2 provides the generated matrix of
the security tools and the vulnerabilities they cover. The
• Check the Zeppelin source of the smart contract in
total list of vulnerabilities is extensive, where for example,
which a vulnerability is identified
• If the vulnerability is also identified by Zeppelin, 33. CLI: Command Line Interface
and the smart contract owners have not modified 34. WUI: Web-based User Interface
35. A simple program analysis tool specifically used for detect-
32. Including here: reentrancy, timestamp dependence, transaction re- ing unchecked-send bug. Link: http://hackingdistributed.com/2016/06/16/
ordering, unchecked-send bug, tx.origin usage. scanning-live-ethereum-contracts-for-bugs/
 only SmartCheck identifies 21 vulnerabilities in total, in- [6] K. Delmolino, M. Arnett, A. Kosba, A. Miller, and E. Shi, “Step
cluding here various warnings and low-risk vulnerabilities. by step towards creating a safe smart contract: Lessons and insights
from a cryptocurrency lab,” in International Conference on Financial
Therefore, due to space limitations, only the most important Cryptography and Data Security. Springer, 2016, pp. 79–94.
vulnerabilities are taken into consideration for this matrix.
[7] V. Buterin, “Thinking about smart contract security - ethereum
Most security tools (6 out of 8) identify more than one blog,” https://blog.ethereum.org/2016/06/19/thinking-smart-contract-
vulnerability, and only two tools identify one vulnerability security/, June 2016, (Accessed on 11/18/2017).
each, Gasper (gas costly patterns), and Simple analysis [8] W. Egbertsen, G. Hardeman, M. van den Hoven, G. van der Kolk,
(unchecked-send bug). Furthermore, since Imandra requires and A. van Rijsewijk, “Replacing paper contracts with ethereum smart
paid access, we do not have any information on what contracts,” 2016.
kind of vulnerabilities it covers. To simplify the matrix, in [9] A. Kosba, A. Miller, E. Shi, Z. Wen, and C. Papamanthou, “Hawk:
the mishandled exceptions we cover: exception disorders, The blockchain model of cryptography and privacy-preserving smart
unchecked-send bug, and gasless send. Whereas, in the im- contracts,” in Security and Privacy (SP), 2016 IEEE Symposium on.
IEEE, 2016, pp. 839–858.
mutable bugs category we cover, type casts and integer over-
[10] D. Boneh and M. Naor, “Timed commitments,” in Advances in
and underflow as well. Visibility (function exposure) checks Cryptology—Crypto 2000. Springer, 2000, pp. 236–254.
are omitted because they are covered only from Smart-
[11] B. Marino and A. Juels, “Setting standards for altering and undoing
Check. And since the stack-size limit is not a vulnerability smart contracts,” in International Symposium on Rules and Rule
anymore, it is eliminated from the list, even though Oyente Markup Languages for the Semantic Web. Springer, 2016, pp. 151–
still has that vulnerability check. 166.
[12] A. Juels, A. Kosba, and E. Shi, “The ring of gyges: Investigating the
future of criminal smart contracts,” in Proceedings of the 2016 ACM
7. Conclusion SIGSAC Conference on Computer and Communications Security.
ACM, 2016, pp. 283–295.
The main purpose of this article was to provide insights [13] T. Chen, X. Li, X. Luo, and X. Zhang, “Under-optimized smart
into the security vulnerabilities on Ethereum smart contracts contracts devour your money,” in Software Analysis, Evolution and
and assess the overall effectiveness of popular security code Reengineering (SANER), 2017 IEEE 24th International Conference
analysis tools used to detect those vulnerabilities. The main on. IEEE, 2017, pp. 442–446.
motivation behind this work was to contribute to a more [14] F. Zhang, E. Cecchetti, K. Croman, A. Juels, and E. Shi, “Town crier:
secure and trustworthy Ethereum environment. We have An authenticated data feed for smart contracts,” in Proceedings of the
2016 ACM SIGSAC Conference on Computer and Communications
conducted a comprehensive review on peer-reviewed pub- Security. ACM, 2016, pp. 270–282.
lications and online resources to collect available data and
[15] Solidity, “Security considerations — solidity 0.4.19 documen-
to propose two taxonomies. The first taxonomy, presented tation,” http://solidity.readthedocs.io/en/latest/security-considerations.
in Table 1, outlines already exploited vulnerabilities and html, (Accessed on 11/19/2017).
classifies them based on their architectural and severity level. [16] Ethereum-Wiki, “Safety · ethereum/wiki wiki,” https://github.com/
It serves as a list of issues that can aid developers who ethereum/wiki/wiki/Safety, (Accessed on 11/19/2017).
plan to develop smart contract applications. The second one, [17] SmartDec, “Smartcheck | knowledgebase | dos by external con-
to is a novel taxonomy of current security tools. We have tract,” https://tool.smartdec.net/knowledge/SOLIDITY_DOS_WITH_
classified the tools based on the methodology they use, the THROW, (Accessed on 12/14/2017).
user interface, and the analysis they are able to execute, [18] M. Araoz, “Introducing zeppelinos: the operating system for smart
which allows us to build a ‘state of the art’ of security tools contract applications,” https://blog.zeppelin.solutions/introducing-
zeppelinos-the-operating-system-for-smart-contract-applications-
on Ethereum. Lastly, we construct a matrix of security tools 82b042514aa8, July 2017, (Accessed on 11/20/2017).
and the vulnerabilities they cover in order to identify gaps
[19] ConsenSys, “Recommendations for smart contract security in so-
and absent vulnerability checks. lidity - ethereum smart contract best practices,” https://consensys.
github.io/smart-contract-best-practices/recommendations/, (Accessed
on 11/21/2017).
References
[20] K. Bhargavan, A. Delignat-Lavaud, C. Fournet, A. Gollamudi,
G. Gonthier, N. Kobeissi, A. Rastogi, T. Sibut-Pinote, N. Swamy,
[1] Z. Zheng, S. Xie, H.-N. Dai, and H. Wang, “Blockchain challenges and S. Zanella-Beguelin, “Formal verification of smart contracts,” in
and opportunities: A survey,” Work Pap, 2016. Proceedings of the 2016 ACM Workshop on Programming Languages
[2] C. Dannen, Introducing Ethereum and Solidity: Foundations of Cryp- and Analysis for Security-PLAS’16, 2016, pp. 91–96.
tocurrency and Blockchain Programming for Beginners, 1st ed. [21] M. Bartoletti, S. Carta, T. Cimoli, and R. Saia, “Dissecting ponzi
Berkely, CA, USA: Apress, 2017. schemes on ethereum: identification, analysis, and impact,” arXiv
[3] L. Luu, D.-H. Chu, H. Olickel, P. Saxena, and A. Hobor, “Making preprint arXiv:1703.03779, 2017.
smart contracts smarter,” in Proceedings of the 2016 ACM SIGSAC [22] E. Hildenbrandt, M. Saxena, X. Zhu, N. Rodrigues, P. Daian, D. Guth,
Conference on Computer and Communications Security. ACM, and G. Rosu, “Kevm: A complete semantics of the ethereum virtual
2016, pp. 254–269. machine,” Tech. Rep., 2017.
[4] N. Atzei, M. Bartoletti, and T. Cimoli, “A survey of attacks on [23] CryptoNews, “CCN: Bitcoin, Ethereum, NEO, ICO & Cryp-
ethereum smart contracts (sok),” in International Conference on tocurrency News,” https://www.cryptocoinsnews.com/, (Accessed on
Principles of Security and Trust. Springer, 2017, pp. 164–186. 11/25/2017).
[5] M. Alharby and A. van Moorsel, “Blockchain-based smart contracts: [24] Y. Zhang, S. Egelman, L. Cranor, and J. Hong, “Phinding phish:
A systematic mapping study,” arXiv preprint arXiv:1710.06372, 2017. Evaluating anti-phishing tools.” ISOC, 2006.

View publication stats