Infrastructure as a Service (IaaS): A Comparative

Performance Analysis of Open-Source Cloud

Platforms

Sonia Shahzadi∗ , Muddesar Iqbal† , Zia Ul Qayyum‡ , Tasos Dagiuklas§

∗ Email: [email protected] † Email: [email protected] ‡§ Email: {m.iqbal,tdagiuklas}@lsbu.ac.uk
∗ Swan Mesh Networks Ltd
‡ Pak-UK Institute of Emerging Technologies for Disaster Management
†§ School of Engineering, London South Bank University, UK

Abstract—Recently, cloud computing has been recognized Examples of IaaS providers are Drop Box, Amazon EC2 and
as effective computing model, being adopted by both large S3, Google Compute Engine, Rackspace, IBM Smartcloud,
corporations as well as SMEs (Small and Medium Enterprises). GoGrid and Akamai [2], [12], [13].
Due to its on-demand provisioning nature number of cloud based In PaaS, CSPs offer the platform as a service where customers
services are growing gradually and rapidly. The availability of can develop and deploy their own applications using devel-
open-source cloud computing platforms plays a vital role to
enable researchers to go beyond state of the art for development
opment environment, application server technology, different
of cloud applications. In this paper, we have provided an overview supportive libraries and programming languages [1]. PaaS can
of cloud computing along with its services. We have presented be helpful where multiple clients use the same development
a comparative performance analysis of different cloud IaaS environment. Integration with the databases and web services
platforms while covering different aspects of their application through some standards is also a feature of PaaS. The testing
and service models. We have also proposed cloud test-bed using and the development can easily be automated for program-
open-source cloud platform. A comparative performance analysis mers in PaaS. The clients have full access and control over
of open-source cloud platforms will assist researchers to select an the tools and the developmental languages. PaaS provider’s
appropriate tool to validate their research and experimentation. examples includes Windows Azure, Google App Engine, Red
Keywords—IaaS, PaaS, SaaS, CSP, NASA, OpenStack, DevS- Hat OpenShift, Engine Yard, Heroku, AppFog, Caspio, MBaaS
tack, Oracle VM Virtual Box, Rally. providers and Aptana Cloud [12].
SaaS is a demand based application that are hosted, maintained
and managed by cloud service providers through a central
I. I NTRODUCTION
location. Examples of SaaS providers are Zoho, Google Docs
Cloud computing is a dynamic field with a lot of flexibility and Salesforce.com [9].
and availability of very economical and reliable on-demand In the remainder of this paper, Section II present a comparisons
computing resources. The consumer can simply connect any- of open-source cloud infrastructure platforms. In Section III,
time from anywhere to access all the resources that are implementation of cloud testbed are described. Performance
available through the cloud. These resources are managed and evaluation of two different OpenStack cloud implementations
utilized in a distributed manner. Cloud computing is a model as a testbed are presented in Section IV. The paper is con-
for enabling ubiquitous, convenient, on-demand network ac- cluded in Section V.
cess to a shared pool of configurable computing resources
(e.g. networks, servers, storage, applications, and services)
that can be rapidly provisioned and released with minimal II. L ITERATURE R EVIEW
management effort or service provider interaction [1]. Essential
A. Cloud IaaS Frameworks
characteristics of cloud computing are: on-demand self-service,
broad network access, resource pooling, rapid elasticity and Today, many cloud frameworks exist that are providing
service measurement [1]. IaaS for both private and public clouds. Several, frameworks
The main cloud computing deployment models are public, are available for this purpose including, Nimbus, Eucalyptus,
private, hybrid and community clouds [2]–[8]. By using these OpenNebula and OpenStack. A taxonomy is proposed on
models, cloud computing provides different services such as selected cloud providers and survey is also given in [14]. From
Infrastructure as a Service (IaaS), Platform as a Service (PaaS) our literature analysis, we have identified some well-known
and Software as a Service (SaaS) [1]–[4], [7]–[11]. cloud platforms that are providing IaaS. A brief overview of
In IaaS, Cloud Service Providers (CSPs) offer the infrastruc- these platforms is given below:
ture as a service where a set of virtual computing resources,
e.g. CPU, storage and network components are provided to the 1) Nimbus: Nimbus provides IaaS and it is divided into two
consumer [1]. Cloud users can deploy and run their own system phases that are: nimbus infrastructure and nimbus platform.
and the software applications using these virtual resources Nimbus infrastructure is providing IaaS implementation that is
[1] with access to the underlying hardware resources. The compatible for EC2/S3. While Nimbus platform is providing
clients of IaaS have a reliable control on their infrastructures. additional tools for simplifying the infrastructure management.
It is mostly considered for scientific cloud computing solutions • Virtual System Monitor (VWS):Used to monitor virtual
[15]. infrastructure.
2) Eucalyptus: Eucalyptus provides IaaS for private cloud. 7) XEN Cloud Platform (XCP): XCP is an open-source
It has following five components [16], [6], [17]: solution for server virtualization and cloud computing platform
includes the XEN hypervisor [24]. It doesn’t provide a com-
• Cloud Controller (CLC): It manages the virtual re-
plete cloud infrastructure solution, but it is helpful for cloud
sources.
management system [25].
• Cluster Controller (CC): It controls the execution of
VMs. B. Comparison of IaaS Frameworks
• Storage Controller (SC): It provides storage system In order to get comparison of these platforms, we have
which is called walrus. derived certain assessment criteria from different cloud fea-
• Elastic Block Storage (EBS): It provides Amazon EBS tures e.g. structure, supported cloud models and programming
semantics support. languages. To perform networking tasks in cloud computing
is also a significant part so it’s essential to understand which
• Node Controller (NC): It controls VM activities in- network service is offered in each platform. Another important
cluding VM termination and VM execution. aspect is to check these platform compatibility with other
hypervisors, and CSP’s as well as scalability, load balancing
3) OpenNebula: OpenNebula is an open source IaaS toolkit and fault-tolerance. In Table I, we have done some features
[18]. It can deal dynamic resource needs, snapshotting and live comparison that is derived from literature analysis [3], [4], [6],
migration [6]. It supports different access interfaces, including [15]–[18], [24]–[36]. The comparison provides the knowledge
OCCI service interface, REST-based interfaces and emerging and understanding which makes it much easier for users,
cloud API standards. developers and the administrator’s to choose the best suited
4) OpenStack: OpenStack provides IaaS for private and framework that is compatible for their environment. Therefore,
public clouds. OpneStack is a combination of multiple open- the presented review covers different characteristics and imple-
source projects to develop your own cloud infrastructure mentation aspects to help in developing better understanding
and it is initiated by Rackspace and NASA in 2010 [19]. of these platforms.
OpenStack main characteristics are: scalable, compatible and
flexible, and open source [13]. It exploits various open-source III. I MPLEMENTATION OF C LOUD T ESTBED
projects and manages both compute and storage resources in
the cloud [20]. OpenStack has mainly following components: In this section we have selected two of the OpenStack
Keystone (Authentication Service), Glance (Image Service), implementations i.e. deployment using DevStack and native
Nova (Compute Service), Neutron (Network Service) Horizon 3-tier OpenStack deployment. Many researchers use DevStack
(Dashboard Service), Cinder (Block Storage) and Swift (Object to quickly create an OpenStack development environment for
Storage) [21], [16], [22], [23]. experimentation, whereas 3-tier OpenStack implementations
are used for production of IaaS platform to implement as a re-
5) CloudStack: CloudStack originally developed by search testbed to perform a performance comparison analysis.
cloud.com in 2010 under GPL v3 license. In 2011, Citrix Using DevStack to substitute the OpenStack implementation
purchased Cloud.com and later on, in 2012, its license can produce different results using the deployment environ-
changed into Apache 2.0 when Citrix donated CloudStack ment. This comparative performance analysis will enable re-
to the Apache Software Foundation [16]. It consists of the searchers to understand this difference among both the options
following components [3], [16]. available for testbed deployment in terms of their performance
in lab environment for experimentation. Following tools are
• Pods: Hardware configuration into a form of clusters used to develop these testbeds.
and a pod can be consist of one or more clusters. - Cloud IaaS Platform: OpenStack and DevStack.
• Clusters: A group of multiple identical hosts and - Host OS: Ubuntu Server 14.04.
specifies for primary storage. - Hypervisor: Oracle VM Virtual Box.
• Primary Storage: Used in each cluster for host VM
IV. P ERFORMANCE E VALUATION
instances.
We have used Rally benchmark tool to measure the per-
• Secondary Storage: Used for VM images and snap-
formance of different cloud services to test and verify the
shots storage.
scalability of cloud. Although, DevStack is not applicable for
• Availability Zones: Used to represent geographical production environment, but if a user want to deploy a mini
locations which are helpful for VM allocation. private cloud to seek certain cloud performance tests, then
this work is beneficial for the purpose where user can get
6) AbiCloud: AbiCloud provides IaaS and mainly it con- an indication on how well these cloud platform services can
sists of following three components [3], [4]: perform in different environments and use-cases. Furthermore,
it will help the researchers to choose the right platform to
• AbiCloud Server: Used for computer management.
setup their testbed in order to validate their hypothesis. We
• AbiCloud Web Services (WS): Used to manage virtual have performed test using same scenarios configuration for
applications. both testbeds to generate the results.
 Scenario 1: In scenario 1, we have run 100 iterations
with 8 atomic actions to measure certain keystone services
correctness and performance. As, shown in Figure 1, x-axis
represents number of iterations while the y-axis represents
the total duration times to complete the specific iteration in
seconds and maximum time is 64.95 (sec) in DevStack case
while 19.21 (sec) in OpenStack. As, this show that OpenStack
performance is much better than as compare to DevStack in
terms of keystone services.

Figure 2. Charts for each Atomic Action in Scenario-2

Figure 1. Charts for each Atomic Action in Scenario-1

Scenario 2: In scenario 2, we have run 100 iterations with 2

atomic actions to measure certain glance services correctness
and performance. As, shown in Figure 2, x-axis represents
number of iterations while the y-axis represents the total
duration times to complete the specific iteration in seconds and
maximum time is 87.88 (sec) in DevStack case while 44.90
(sec) in OpenStack. As, this scenario also show that OpenStack
performance is much better than as compare to DevStack in
terms of creating and listing images.
Scenario 3: In scenario 3, we have run 100 iterations
with single atomic actions to measure specific nova service Figure 3. Charts for each Atomic Action in Scenario-3
correctness. As, shown in Figure 3, x-axis represents number of
iterations while the y-axis represents the total duration times to
complete the specific iteration in seconds and maximum time is the most appropriate platform for their testbed deployments.
27.90 (sec) in DevStack case while 2.63 (sec) in OpenStack.
As, this scenario also show that OpenStack performance is V. C ONCLUSION AND F UTURE WORK
much better than as compare to DevStack while listing hyper-
visors. Although many cloud platforms are available for both
academic and industrial deployments but selecting a most
Analysis of results After observing these scenarios perfor- appropriate platform based on user requirements is major
mance, we conclude that although DevStack installer provides challenge. This paper presents a detailed comparison of these
quick deployment of OpenStack and it’s also much easier to platforms while covering different deployment aspects and
deploy as compare to manual solution but its performance and characteristics such as architecture, networking, supported
response time of cloud services is much slower as compare platforms, hypervisors and languages. Although, each of these
to native OpenStack approach. As, we have tested multiple platforms has its own strengths and limitations but the pre-
scenarios from different OpenStack services (e.g. Keystone, sented comparison will help the researchers to select most
Glance, Nova) but response time in each scenario of DevStack appropriate platform as per requirement. In this paper, we
is much lower than manual OpenStack. The results shows have also implemented open-source cloud testbed using both
a noticeable difference in performance for both the testbed 3-tier OpenStack and DevStack platforms and presented per-
platforms which we believe will help the researchers to select formance comparison for both implementations using different
services scenarios. The results shows that the performance of [16] A. Beloglazov, S. F. Piraghaj, M. Alrokayan, and R. Buyya, “Deploying
OpenStack 3-tier implementation is much better as compare openstack on centos using the kvm hypervisor and glusterfs distributed
file system,” University of Melbourne, 2012.
to auto deployment method using DevStack. Therefore, the
experimentation performed on both the tools using the same [17] G. Raj, C. Kapoor, and D. Singh, “Comparative cloud deployment and
service orchestration process using juju charms,”
hardware and test scenarios may produce different results,
[18] X. Wen, G. Gu, Q. Li, Y. Gao, and X. Zhang, “Comparison of open-
which is not acceptable to validate a research hypothese in source cloud management platforms: Openstack and opennebula,” in
most of the cases and may raise concerns. Fuzzy Systems and Knowledge Discovery (FSKD), 2012 9th Interna-
In future work, we will explore the use of light-weight con- tional Conference on, pp. 2457–2461, IEEE, 2012.
tainer approaches which is a rapidly growing area of research. [19] “OpenStack.” http://www.openstack.org. Accessed: 2017-01-01.
We plan to study the performance aspects of using containers [20] A. Corradi, M. Fanelli, and L. Foschini, “Vm consolidation: A real
and compare it with virtual machines to provide cloud services. case based on openstack cloud,” Future Generation Computer Systems,
vol. 32, pp. 118–127, 2014.
[21] F. Wuhib, R. Stadler, and H. Lindgren, “Dynamic resource allocation
ACKNOWLEDGMENT with management objectives—implementation for an openstack cloud,”
in Network and service management (cnsm), 2012 8th international
The presented work is being undertaken in the context of conference and 2012 workshop on systems virtualiztion management
the ”Self-OrganizatioN towards reduced cost and eNergy per (svm), pp. 309–315, IEEE, 2012.
bit for future Emerging radio Technologies” with contract num- [22] R. Kumar, “Openstack juno release includes features of nfv, big data,”
ber 734545. The project has received research funding from the Nova, vol. 1, p. 21, 2010.
H2020-MSCA-RISE-2016 European Framework Program. [23] T. Yanagawa, “Openstack-based next-generation cloud resource man-
agement,” Fujitsu Sci. Tech. J, vol. 51, no. 2, pp. 62–65, 2015.
[24] M. Bist, M. Wariya, and A. Agarwal, “Comparing delta, open stack
R EFERENCES and xen cloud platforms: A survey on open source iaas,” in Advance
Computing Conference (IACC), 2013 IEEE 3rd International, pp. 96–
[1] P. Mell and T. Grance, “The nist definition of cloud computing,” 2011. 100, IEEE, 2013.
[2] S. N. R. J. S. Vanishree and P. Karthick, “A prototype for private cloud [25] T. Cordeiro, D. Damalio, N. Pereira, P. Endo, A. Palhares, G. Gonçalves,
implementation using open-source platform,” D. Sadok, J. Kelner, B. Melander, V. Souza, et al., “Open source cloud
[3] J. Peng, X. Zhang, Z. Lei, B. Zhang, W. Zhang, and Q. Li, “Comparison computing platforms,” in Grid and Cooperative Computing (GCC),
of several cloud computing platforms,” in Information Science and 2010 9th International Conference on, pp. 366–371, IEEE, 2010.
Engineering (ISISE), 2009 Second International Symposium on, pp. 23– [26] Q. Huang, C. Yang, K. Liu, J. Xia, C. Xu, J. Li, Z. Gui, M. Sun,
27, IEEE, 2009. and Z. Li, “Evaluating open-source cloud computing solutions for
[4] S. Wind, “Open source cloud computing management platforms: In- geosciences,” Computers & Geosciences, vol. 59, pp. 41–52, 2013.
troduction, comparison, and recommendations for implementation,” in [27] N. G. Bachiega, H. P. Martins, R. Spolon, M. A. Cavenaghi, R. S.
Open Systems (ICOS), 2011 IEEE Conference on, pp. 175–179, IEEE, Lobato, and A. Manacero, “Open source cloud computing: Character-
2011. istics and an overview,” in Proceedings of the International Conference
[5] M. Armbrust, A. Fox, R. Griffith, A. D. Joseph, R. Katz, A. Konwinski, on Parallel and Distributed Processing Techniques and Applications
G. Lee, D. Patterson, A. Rabkin, I. Stoica, et al., “A view of cloud (PDPTA), p. 237, The Steering Committee of The World Congress
computing,” Communications of the ACM, vol. 53, no. 4, pp. 50–58, in Computer Science, Computer Engineering and Applied Computing
2010. (WorldComp), 2013.
[28] H. A. Albaroodi, S. Manickam, and M. F. Aboalmaaly, “The classifica-
[6] G. Von Laszewski, J. Diaz, F. Wang, and G. C. Fox, “Comparison of
tion and arts of open source cloud computing: A review,” Advances in
multiple cloud frameworks,” in Cloud Computing (CLOUD), 2012 IEEE
Information Sciences and Service Sciences, vol. 5, no. 16, p. 16, 2013.
5th International Conference on, pp. 734–741, IEEE, 2012.
[29] F. Meireles and B. Malheiro, “Integrated management of iaas resources,”
[7] R. Buyya, S. Pandey, and C. Vecchiola, “Cloudbus toolkit for market-
in Euro-Par 2014: Parallel Processing Workshops, pp. 73–84, Springer,
oriented cloud computing,” in IEEE International Conference on Cloud
2014.
Computing, pp. 24–44, Springer, 2009.
[30] B. P. Rimal, E. Choi, and I. Lumb, “A taxonomy and survey of cloud
[8] Q. Zhang, L. Cheng, and R. Boutaba, “Cloud computing: state-of-the-art
computing systems,” in 2009 Fifth International Joint Conference on
and research challenges,” Journal of internet services and applications,
INC, IMS and IDC, pp. 44–51, Ieee, 2009.
vol. 1, no. 1, pp. 7–18, 2010.
[31] I. Nwobodo, “A comparison of cloud computing platforms,” 2015.
[9] A. Marinos and G. Briscoe, “Community cloud computing,” in Cloud
Computing, pp. 472–484, Springer, 2009. [32] M. Alam and K. A. Shakil, “Recent developments in cloud based
systems: state of art,” arXiv preprint arXiv:1501.01323, 2015.
[10] C. Vecchiola, S. Pandey, and R. Buyya, “High-performance cloud
computing: A view of scientific applications,” in Pervasive Systems, [33] B. Mohammed and M. Kiran, “Analysis of cloud test beds using open-
Algorithms, and Networks (ISPAN), 2009 10th International Symposium source solutions,” in Future Internet of Things and Cloud (FiCloud),
on, pp. 4–16, IEEE, 2009. 2015 3rd International Conference on, pp. 195–203, IEEE, 2015.
[11] D. Durkee, “Why cloud computing will never be free,” Queue, vol. 8, [34] F. Cocozza, G. López, G. Marın, R. Villalón, and F. Arroyo, “Cloud
no. 4, p. 20, 2010. management platform selection: A case study in a university setting,”
CLOUD COMPUTING 2015, p. 92, 2015.
[12] H. BH, L. Suresh, and K. Radhika, “A survey on cloud computing,”
INTERNATIONAL JOURNAL OF MATHEMATICS, vol. 1, no. 11, 2013. [35] P. T. Endo, G. E. Gonçalves, J. Kelner, and D. Sadok, “A survey on
open-source cloud computing solutions,” in Brazilian Symposium on
[13] O. Sefraoui, M. Aissaoui, and M. Eleuldj, “Openstack: toward an Computer Networks and Distributed Systems, pp. 3–16, 2010.
open-source solution for cloud computing,” International Journal of
Computer Applications, vol. 55, no. 3, 2012. [36] S. Kumar and R. Goudar, “Cloud computing-research issues, challenges,
architecture, platforms and applications: A survey,” International Jour-
[14] R. Prodan and S. Ostermann, “A survey and taxonomy of infrastructure nal of Future Computer and Communication, vol. 1, no. 4, p. 356,
as a service and web hosting cloud providers,” in Grid Computing, 2009 2012.
10th IEEE/ACM International Conference on, pp. 17–25, IEEE, 2009.
[37] L. Wang, G. Von Laszewski, A. Younge, X. He, M. Kunze, J. Tao,
[15] M. Mahjoub, A. Mdhaffar, R. B. Halima, and M. Jmaiel, “A compar- and C. Fu, “Cloud computing: a perspective study,” New Generation
ative study of the current cloud computing technologies and offers,” Computing, vol. 28, no. 2, pp. 137–146, 2010.
in Network Cloud Computing and Applications (NCCA), 2011 First
International Symposium on, pp. 131–134, IEEE, 2011.
 Table I: Comparison of IaaS Frameworks

Features Nimbus Eucalyptus Open- OpenStack CloudStack Abicloud XEN Cloud

Nebula Platform
Year 2009 2008 2005 2010 2010 2009 2012
Developed University University European Rackspace, Cloud.com Abiquo Citrix Xen
by of Chicago of Union NASA Server
California,
Santa
Barbara,
Eucalyptus
system
company
Main pur- Scientific so- Private Private Provides − Tool for Manage
pose lution Cloud Cloud Cloud automatic cloud
Computing cloud computing
services. management platform
Architecture Centralized Hierarchical Centralized Distributed Hierarchical Centralized Centralized
/Structure
Cloud Ser- IaaS IaaS IaaS IaaS IaaS IaaS IaaS
vice Model
Managed − Eucalyptus C12G Labs OpenStack Apache − −
by Systems Foundation Software
Foundation
Hypervisors KVM, XEN KVM, XEN, KVM, KVM, KVM, KVM, Xen
VMWare VMWare, VMware, VMWare, VMWare,
in the XEN XEN, LXC, XEN, Oracle XEN,
enterprise QEMU, VM Virtual Box
edition. Vsphere,
HyperV,
Microsoft
Hyper -
v, UML,
Virtual Box
Interface CLI CLI CLI CLI − Web CLI
interface
Compatible AWS EC2, AWS EC2, AWS EC2, AWS EC2, − AWS EC2 −
with S3 S3 S3, Native S3, EBS and
XML/RPC, OCCI
OCCI
Supported Linux Linux Linux Linux Linux Linux Linux
Platform
Supported Public Private and Private, pub- Private, pub- − Private, pub- Public and
Cloud hybrid lic, and hy- lic, and hy- lic, and hy- private
Types brid brid brid clouds
Programming Java, Python Java, C, C, C++, Python Java C++, Java, C, Python
Language Python Java, Ruby Ruby,
Python
VMs loca- Physical Node Cluster node Compute − Clouds XCP Host
tion nodes controller Node nodes
Networking IP assigned Managed, VLAN Flat, Flat VLAN WSManagemenVLAN,
using a managed- DHCP, Open
DHCP novLAN, VLAN vSwitch
server that system, DHCP
can be static
configured
in two
ways i.e.
centralized
and local.
Storage − Volume and Volume stor- Volume and Volume and NFS Shared stor-
Object stor- age Object stor- Object stor- age
age age age
DevOps De- No Chef, Chef, Puppet Chef, − − −
ployment Puppet, Puppet,
Crowbar Crowbar
Authentication X509, Grids X509 X509, X509, − Code −
LDAP, LDAP Access Sys-
SSH RSA, tem(CAS)
key pair,
Password
Users Scientific Enterprise Researchers Enterprises, − Enterprise −
communities researchers
and
developers.
Scalability Yes Yes Yes − − Yes Yes
Storage Cumulus Walrus NFS Swift − ZFS −
(Image
Transfer-
ence)
License Apache Li- GPL Apache Li- Apache Li- Apache Li- GPL GPL
cense 2.0 cense 2.0 cense 2.0 cense 2.0
VM Migra- No No Yes Yes Yes No Yes
tion/Live
Migration
Load Yes − Yes Yes Yes No Yes
Balancing
Fault Toler- Yes − Yes Yes Yes No Yes
ance