Available online at www.sciencedirect.

com

ScienceDirect
Procedia Computer Science 50 (2015) 482 – 489

2nd International Symposium on Big Data and Cloud Computing (ISBCC’15)

Cloud Testing Tools and Its Challenges: A

Comparative Study
S. Nachiyappana, S. Justusb
a
Assistant Professor, VIT University, Chennai
b
Associate Professor, VIT University, Chennai

Abstract

Cloud computing has emerged as a new technology across organization and cooperates that impacts several
different research fields, including software testing. To provide a cloud service and sharing resources successfully,
the cloud must be tested before it comes into offering services. Testing the applications has their own testing tools
and testing methodologies. In this paper we provide an overview regarding cloud computing trends, types,
challenges, tools and the comparison of tools for cloud testing.

© 2015 The
© 2015 The Authors.
Authors. Published
PublishedbybyElsevier
ElsevierB.V.
B.V.This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).
Peer-review under responsibility of scientific committee of 2nd International Symposium on Big Data and Cloud Computing
Peer-review under responsibility of scientific committee of 2nd International Symposium on Big Data and Cloud Computing
(ISBCC’15).
(ISBCC’15)
Keywords: Software Testing; SOASTA; Functional Testing; Performance Testing; Service Oriented Architecture; Cloud Testing

1. Introduction

Traditional approaches to software testing have become costly in terms of effort, cost and overheads.
Hence testing practices have shifted gears to strategies that would benefit the organizations in terms of business and
profits. Leading companies such as IBM, Microsoft, Google, and Amazon have a vested interest in the biz word
“CLOUD”. As computations, storage and customer interactions have already started migrating to cloud
environment, software testing is also getting migrated to Cloud. Testing new software requires costly server, storage
and network devices only for a limited time. These computing resources are not used after testing, thus incurring
extra cost on budget. To ensure a reliable service, providers have to test their services on all platforms.

1877-0509 © 2015 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).
Peer-review under responsibility of scientific committee of 2nd International Symposium on Big Data and Cloud Computing (ISBCC’15)
doi:10.1016/j.procs.2015.04.018
 S. Nachiyappan and S. Justus / Procedia Computer Science 50 (2015) 482 – 489 483

Fig.1. Types of Cloud Services

Cloud testing is a form of evaluation methodology in which the applications to be tested uses cloud as a
computing environment and its infrastructure to simulate real world traffic by using existing cloud computing
technologies. Cloud testing basically aligns with the concepts of cloud and Software as a service (SaaS). Cloud
testing provides the ability to test cloud by using cloud infrastructure such as hardware, network bandwidth, and
workload that more closely simulate real world conditions and parameters. In simple words, verification and
validation of applications, environments and infrastructure.

Testing a cloud includes availability, security, performance, interoperability, disaster recovery and
multitenancy testing. Cloud testing are challenged by several problems such as limited budget, meeting deadlines,
High cost per test, large number of test cases, little reuse of tests and geographical distributions of users. The aim of
cloud testing is to ensures high quality service delivery and avoiding data outages requires testing inside data centre
or outside the data centre or in both places

Fig.2. Cloud Testing

2. Cloud Testing Types

In this section we briefly describe various Cloud platform testing dimensions applicable in general

2.1. Elasticity and Scalability Testing

Elasticity and Scalability Testing one of the major characteristics of Cloud is its support for auto-elasticity. i.e.
resource demand can be satisfied on-the-go elastically. Hence vertical and horizontal scalability need to be tested.
Vertical scalability means that, replacing current resource with a more powerful resource should satisfy the
increasing demand. On the other hand, horizontal scalability means that adding more resources of same type should
484 S. Nachiyappan and S. Justus / Procedia Computer Science 50 (2015) 482 – 489

increase the performance of the platform as per the requirements. Similarly, resources should be provisioned in a
seamless manner whenever the requirements arise. We should also test if load balancing works properly with
changes in demand. Finally, elastic load testing need to be performed for various
possible user scenarios. [1]

2.2. Security Testing

Cloud platforms and applications are exposed to several security vulnerabilities. One important security concern
is traversal vulnerability. It means that one tenant (could be an intruder) is able to traverse from one Virtual Machine
(VM) client environment to other client environments being managed by the same hypervisor. This vulnerability
might allow a customer to access the virtual instances of other customer’s applications. So adequate testing is
required to make sure that the platform is not vulnerable to such situations. With multi-tenant environments,
penetration testing is very important to simulate a malicious user and to test for all vulnerabilities such as SQL
injection, Cross-Site scripting (XSS) etc. We need to test with a malicious user who has valid credentials to the
Platform under test as a tenant and that user’s ability to penetrate the system and view the information of other
tenants. Other security areas to be tested include checking for the capability of appropriate role-based access control,
identity federation management, and appropriate data management. [1]

2.3. Performance Testing

With the unique characteristics of Cloud environments, we need to perform the accuracy of various data present
in the Cloud, latency and throughput. Further, elastic load testing and multi-tenant performance testing are other key
items to be considered. Finally, high availability and failover testing are required to test the behavior of the platform
and applications under resilience scenarios.

2.4. Live Upgrade Testing

This is closely related to understanding the performance of the system when an upgrade of the software/platform
happens and to understand its capability to continue its business services to users even when the upgrade is going
on. We need to make sure that business continuity is there even when software/hardware maintenance/upgrade is
performed.

2.5. Stress Testing

Stress Test is used to determine ability of application to maintain a certain level of effectiveness beyond breaking
point. It is essential for any application to work even under excessive stress and maintain stability. Stress testing
assures this by creating peak loads using simulators. But the cost of creating such scenarios is enormous. Instead of
investing capital in building on-premise testing environments, cloud testing offers an affordable and scalable
alternative.

2.6. Functional Testing

Functional testing of both internet and non-internet applications can be performed using cloud testing. The
process of verification against specifications or system requirements is carried out in the cloud instead of on-site
software testing.
 S. Nachiyappan and S. Justus / Procedia Computer Science 50 (2015) 482 – 489 485

2.7. Compatibility Testing

Using cloud environment, instances of different Operating Systems can be created on demand, making compatibility
testing effortless. [2]

2.8. Browser Performance

To verify application's support for various browser types and performance in each type can be accomplished with
ease. Various tools enable automated website testing from the cloud.

2.9. Latency Testing

Cloud testing is utilized to measure the latency between the action and the corresponding response for any
application after deploying it on cloud.

Fig.3. Types of Cloud Testing

3. Testing Tools

Various tools are available to test cloud based systems at various layers including hardware interface,
platform interface, storage system and application system.

3.1. SOASTA

SOASTA is motivated by the necessity to test in production, rather than in a laboratory environment. Today’s
web applications usually follow agile practices with frequent builds and high change rates. Load testing with legacy
tools in the laboratory can be significantly different from testing in the production environment in terms of scale,
configuration, user profiles and network environment. Running tests against production websites thus can achieve
higher degree of accuracy and confidence, compared with lab practices. [3] SOASTA CloudTest is a production
performance testing tool for Web applications. It can simulate thousands of virtual. Public cloud infrastructure
service. The worker nodes can be distributed across public and private clouds to cooperate in a large load testing. [7]
486 S. Nachiyappan and S. Justus / Procedia Computer Science 50 (2015) 482 – 489

Test results from distributed test agents are integrated for analysis. Memory-based analytic techniques are
implemented to handle, in real-time, the huge data produced by large-scale testing. Provisioning data are displayed
via analytic dashboard on a synchronized time-line. Through an Ajax based web UI, testers can operate and
supervise the whole process including launching hundreds of load generation servers, creating and running test
agents geographically distributed, and analyzing test results.

3.2. ITKO LISA

ITKO's LISA product suite is designed to improve the effectiveness of application development teams, especially
those involved in custom applications, SOA, and cloud computing. [6] iTKO LISA aims to provide a cloud-based
environment and virtual services for composite application development, verification and validation. It claims to
reduce software delivery timeline by 30% or more using its innovative approach to support continuous integration
for development and testing. Central to LISA architecture is its virtualization technology. For unavailable or
inaccessible resources, LISA provides virtualized services by simulating the target system’s dynamic behaviour so
that they can respond as live systems. In this way, it breaks Dependence constraints of system integration and
supports continuous testing. [3]

3.3. Load Runner

HP LoadRunner is an automated performance and test automation product from Hewlett-Packard for application
load testing: examining system behavior and performance, while generating actual load. HP acquired LoadRunner as
part of its acquisition of Mercury Interactive in November 2006. A software testing tool, HP LoadRunner works by
creating virtual users who take the place of real users' operating client software, such as Internet Explorer, sending
requests using the HTTP protocol to IIS or Apache web servers.[10] HP LoadRunner can simulate thousands of
concurrent users to put the application through the rigors of real-life user loads, while collecting information from
key infrastructure components (Web servers, database servers etc.) The results can then be analyzed in detail to
explore the reasons for particular behaviour. HP LoadRunner supports various protocol bundles for load testing:
.NET Record/Replay, Database, DCOM, GUI Virtual Users, Java Record/Replay, Network, Oracle E-Business,
Remote Access, Remote Desktop, Rich Internet Applications, SAP, SOA, Web 2.0, Web and Multimedia and
Wireless.

HP LoadRunner can be run standalone geared towards one or two persons using each controller or as part of HP
Performance Centre (which pools together several controllers, all load generators, adding a web site, a schedulers
and more to facilitate sharing of the LoadRunner resources among many persons.[6][9]

3.4. Blitz

Blitz is a load-testing tool from the cloud to the cloud. Blitz customers tend to be application and website
developers who use the service throughout the iterative build process of mobile applications, websites, and APIs
Blitz provides developers with several capabilities throughout the build process [6]:

x Load testing for Web apps and APIs to test scalability

x Integration with PaaS providers, continuous integration tools, and browsers
x Scales testing up to 50,000 simultaneous virtual users on a pay-per-test model
x Cloud-based, no client to install. However, this means it is unable to test applications behind firewalls or
otherwise protected from the Internet.
 S. Nachiyappan and S. Justus / Procedia Computer Science 50 (2015) 482 – 489 487

3.5. Blaze Meter

Blaze Meter’s Load Testing Platform for Developers is designed for professional use, is equipped with a self-
service, on-demand platform and advanced scripting capabilities leveraging JMeter and Selenium (Web
Driver). Blaze Meter can run multiple load tests that easily simulate load of up to 1,000,000 concurrent users from
both the public cloud or inside the corporate firewall, enabling its customers to quickly locate and fix performance
bottlenecks.[6]

You can create proprietary test scripts and load scenarios using a graphical web environment. Blaze Meter offers
web-based test management, archiving, repository, cloud-based monitoring, rich scripting language, and
supports HTTP/S, web-services, XML, TCP, SQL, Login (Flash, images, streaming) and more. Blaze Meter enables
you to write load test-scripts using JMeter and user-experience test-scripts using Selenium .Blaze Meter will
generate a load based on the JMeter script. The Selenium script is used during the load to automate the launch of
real browsers to measure the real end-user experience. The load and monitoring is using a pre-configured distributed
load testing environment. The environment is ready to use and available at all times. Table1. Shows the comparison
of testing tools on cloud we have discussed the pros and cons of the tools with pricing.

Tool Pros Cons Pricing

A complete environment to setup Requires some setup: you have to “CloudTest is sold based on server
Soasta tests spin up a VM to load the test hours, including support. With no
Add test users locations seamless environment limits on the number of testers or
A good coverage around the globe software access, you select
A comprehensive real-time interface to Plans based on how many test
metrics server hours you’ll run. Plans are
Drill down analysis on web requests A available for coverage of a few
Lite (Free) version is available with a tests each month up to thousands
limit of 100 concurrent users of tests each year”

LoadStorm Scalable up to 300,000 users Currently they only cover Monthly plans – Starting Free (100
A testing data centre in Sydney, Virginia, California, Oregon, users, 25 users per test) up to “Call
Australia Ireland, Sydney, Tokyo, Us”
Supports automatic crawling (Spider) Singapore, Sao Paulo
No scripting language is involved On-Demand (Per hour) – Starting
Nothing to install No built-in feature to collect $39 (1,000 users per test) to “Call”
server performance metrics (10,000+ users

Blitz Capacity planning tools, Optimization, Only 8 test locations around the Works with “Credits”
Performance monitoring are out-of-the- globe (Australia, Brazil, 1 credit = 1 minute and 1,000
box California Ireland, Japan, Oregon, users
Nothing to install/setup, all cloud-based Singapore, Virginia) credits = $40
Simulate up to 5,000 concurrent virtual Relatively less sophisticated credits = $135
users interface (compared to Soasta) 300 Credits = $240
It has a chrome/Firefox plugin to
monitor app performance on develop
environment
Integrated with Google Analytics
Performance Monitoring Alerts (Email,
Twitter, SMS, Pager Duty)

Blaze Meter JMeter compliant Available only in limited Monthly plans – Starting $199
Easy to use test creation interface geographical areas, (1,000 users, 20 hours) up to “Call
Nice interface for test results cover’s(Brazil, Tokyo, Oregon Us”
You can export test results and Virginia USA, Singapore, On-Demand (Per hour) – Starting
Compare two test runs California, Australia and Ireland) $19 (1,000 users, 1 server) to $299
Has a server in Australia (yes, we’re (40,000 users, 40 servers)
load-testing apps in Oz)
Test scheduling
Integrated with Google Analytics
VPN is supported
Chrome extension to record JMeter
488 S. Nachiyappan and S. Justus / Procedia Computer Science 50 (2015) 482 – 489

scripts and upload it to BlazeMeter

It has a free version that can simulate up
to 50 concurrent users

Sources: [1] http://michaelfarag.blogspot.in/2013/09/load-testing-your-cloud-app-tools_26.html

[2] http://www.soasta.com/wp/wp-content/uploads/2012/10/DS-CloudTest-100812.pdf
[3]http://loadstorm.com/load-testing-cost/
[4]https://www.blitz.io/pricing#/subscriptions
[5] http://blazemeter.com/pricing
Table1. Comparison of Testing Tools

4. Testing Challenges

4.1. Constructing Environment For On Demand Testing

For an on demand testing service, what are the steps to be taken to create a testing environment which is
systematic or automatic? Even though the present technologies in cloud support instinctive delivery of needed
computing resources for every Software as a Service (SaaS) or for an application in the cloud, for setting up the
needed test domain in a cloud there are no carrying solutions to assist engineers using a beneficial way. [5]

4.2. Testing Security Measures

In present software testing society security testing is been converting to a profound topic with lot of open queries.
Meanwhile security turn into a most important requirement within the clouds and security services converted into a
compulsory measure in present Software as a Service (SaaS) and cloud technology, the problems and challenges in
security validation and quality assurance for SaaS and clouds need to be accord by the engineers .[5]

4.3. Integration Testing

Even though we saw various research articles describing software integration testing problems and schemes, not
plentiful research outcomes are been put in in the real engineering system. The main causes is the present software
and mechanisms are created without permitting technology and solution to help and assist organized software
integration. In a cloud organization, engineers needed to be deal with combination of various SaaS and applications
inside and outside clouds in a black-box sight depending on the offered APIs and connectivity protocols.

4.4. Regression Testing

The regression testing problems and challenges caused by software modifications and bug-fixing must be
addressed the by on-demand software validation in clouds. Yet, maximum current exploration in software regression
testing grants most consideration to re-test a particular software version in a preconfigured test environment. The
multi-tenancy characteristic of clouds might origin the trouble to put on the present explorated work in cloud testing,
particularly for on-demand software regression testing service each and every time software is modified.[5] In
further, we also lack of dynamic software validation methods and solutions to address the dynamic features of SaaS
and clouds

Conclusion

In upcoming years cloud computing is emerging in IT industry the real opportunity to cut cost of test
environment in public private and hybrid environments or community clouds lies in both their management and
maintenance before deploying the cloud its essential that cloud application is thoroughly tested. In this paper we
 S. Nachiyappan and S. Justus / Procedia Computer Science 50 (2015) 482 – 489 489

have discussed different cloud testing techniques and commercial tools available in the market. Though we are in
starting stage of cloud testing we have identified some of the challenges through the analysis based on research
papers. Based on the challenges we are planning to build a new testing framework in future to test a cloud.

REFERENCES
[1] Ganesh Neelakanta Iyer, Member, IEEE, IEEE-CS,PCTF: “An Integrated, Extensible Cloud Test Framework for Testing
Cloud Platforms and Applications”;13th International Conference on Quality Software 2013.
[2] Namratha gantayat, “Testing in the cloud and its challenges”, asian journal of research in social Science & Humanities Vol.2
Issue 4,ISSN 2249 7315, April 2012.
[3] Xiaoying Bai ,“Cloud Testing Tools”, ,Proceedings of The 6th IEEE International Symposium on Service Oriented System
Engineering, SOSE 2011.
[4] A.Vanitha Katherine ,”Software Testing in Cloud Platform: A Survey”, International Journal of Computer Applications
(0975 – 8887) Volume 46– No.6, May 2012
[5] Er. Tamanna Narula1, “Framework for Analyzing and Testing Cloud based Applications” , International Journal of
Advanced Research in Computer Science and Software Engineering, Volume 4, Issue 6,ISSN: 2277 128X, , June 2014
[6] www.wikipedia.org
[7] http://www.soasta.com/wp/wp-content/uploads/2012/10/DS-CloudTest-100812.pdf
[8] http://loadstorm.com/load-testing-cost/
[9] http://www.testingthefuture.net/2011/03/reliability-cloud-recovery-testing-of-the-cloud/
[10] http://www8.hp.com/in/en/software-solutions/loadrunner-load-testing/
[11] http://www.parasoft.com/soatest
[12] Prakash.V, “Cloud Testing –Myths Facts And Challenges,” International Journal of Reviews in Computing, Vol. 9, 10th April
2012.
[13] Swapnil H. Chandane, “New Computing Paradigm: Software Testing in Cloud, Issues, Challenges and Need of Cloud
Testing in today’s World”, International Journal of Emerging Research in Management &Technology ISSN: 2278-9359,
February 2013
[14] Koray ˙Inc’ki, “A Survey of Software Testing in the Cloud”, IEEE Sixth International Conference on Software Security and
Reliability Companion, 2012
[15] Mark Sylor, New Technologist EXFO Service Assurance, “Testing the cloud”, White Paper 023, EXFO, 2012
[16] Neha Mehrotra, “Cloud-Testing Vs. Testing a Cloud”,10th Annual International Software Testing Conference,2010