International Journal of Computer Applications Technology and Research

Volume 4 Issue 5, 405 - 408, 2015, ISSN:- 23198656

Evaluation of automated web testing tools

Mohamed Monier
Information System Department,
Faculty of Computers and Informatics, Zagazig
University, Egypt.

Mahmoud Mohamed El-mahdy

Information System Department,
Faculty of Computers and Informatics, Zagazig
University, Egypt.

Abstract: Software testing is a main part of Software Development Life Cycle and one of the important aspects of Software
Engineering. There is a wide variety of testing tools which require or not the user experience in testing software products. According
to the daily use, Mobile and Web applications take the first place in development and testing. Testing automation enables developers
and testers to easily automate the entire process of testing in software development saving time and costs. This paper provide a
feasibility study for commercial and open source web testing tools helping developers or users to pick the suitable tool based on their
requirements.
Keywords: Black Box Testing; web testing tools; open source; Commercial;

1. INTRODUCTION
Software Testing aims to evaluating the software quality and
to what degree the efficiency of that product. Testing Process
including many aspects such as reliability, usability, integrity,
maintainability and compatibility [1].
The Two main types of Software testing Black Box Testing
and White Box [2].Black Box Testing concerned with the
specification of the System component under test which not
require intensive knowledge about the internal structure of the
system. White box strategy otherwise require high experience
of the internal system code for developing test suits suited the
test cases.
Web and Mobile applications have become very complex and
crucial, Most of researches focused attention to Web
application design, development, analysis, and testing, by
studying and proposing methodologies and tools [3].Mobile
applications developed over more than platform which need
more experience in the developing environment and structure
of applications to be designed and developed.
Software Testing follow two ways manual or automation.
Manual Testing has many drawbacks such as consuming time
and cost, require experience, complex reusing, less efficiency
and not provide scripting facility for code [4].Automation
testing reveal all complex Obstacles attached with manual
testing, this type of testing create a scenarios by recording the
interaction with the system under design into test cases to be
tested under many Configurations [5].
Automated testing tools exist widely in the market varying in
the capabilities and features which make the user puzzled for
which tool suitable for his testing purpose [6]. There are two
types testing tools commercial and open source tools. Open
source tools are free for users to use with open source code to
be modified. On the other hand, Commercial tools take
advantage in organizations and mentoring capabilities
providing the user with facilities needed to accomplish tasks
with extra controlled features and low efforts.
The Objective of this paper is to present feasibility study of
automated web testing tools through comparing the tools
features for helping users to select suitable tools according to
their requirements based on a study of tools major criteria.
The paper divided into sections. Section I provide a brief
overview of testing tools. Section II discuss the tools features
and criteria used as input to the model. Section III discuss the

www.ijcat.com

related work. Section IV Methodology Section V finally

provide the conclusion and future work.

2. RELATED WORK
Last researches interested in comparing the capabilities of the
testing tools by practicing them or only based features each
tool support. Harpreet Kaur, Gagan Gupta conduct a
comparative study among selenium, Test Complete and QTP
tools the study include many aspects but not drag the
automation features of tools such as record and play-back,
cross platform or browsers support features [7]. Abha Jain,
Manish Jain, Sunil Dhankar[8] compare two commercial tools
Ranorex vs. QTP including many features but the main
concern on the cost of the total project and the study not
include any open source software to compare against. Angmo,
R and Sharma, M [9] compare the performance of selenium
web driver against watir-web driver the two open source
software, Study includes performance parameters such as
execution speed which vary in the type of tested Controls.
This research is efficient but require more than one tool to
give the best judge to the user.

3. METHODOLOGY
There are a lot of web testing tools exist on the market
commercial or open source. We select the tools that perform
the automation testing using record scripts and then playback
this scripts as an important feature in testing automation.

3.1 Automated Software testing tools

3.1.1

Selenium webdriver

Selenium IDE is a one of the most popular free open-source

automated testing tool which provide a testing framework for
testing web applications and supporting multiple kind of
frameworks. It can be easily downloaded from internet as a
plug-in for some browsers. It is basically used by the web
development community to perform automated testing of web
applications. We choose in our study Selenium web-driver
because Selenium IDE not support record-playback feature
and also it most supportive for web-application testing [10].

405

International Journal of Computer Applications Technology and Research

Volume 4 Issue 5, 405 - 408, 2015, ISSN:- 23198656
3.1.2 Sahi
Sahi is an open source provide a testing framework based on
Ruby and java script supporting the most types of web
browsers and platforms. Sahi provides powerful abilities for
recording and replaying across browsers; different language
drivers for writing test scripts (Java, Ruby) and support for
AJAX and highly dynamic web application Sahi used by IBM
developers for web applications testing automation [11].

3.1.3 Watir-web driver

Watir is an abbreviation for Watir application testing in Ruby.
Is a powerful open source tool that requires programming
skills in ruby language [12].We choose Watir web-driver for
evaluation study in web automated testing as it support
record-playback capability. It is available as RubyGems and
capable of driving variety of browser including the major like
Internet Explorer, Firefox etc. [13]. Bret Pettichord and Paul
Rogers developed Watir. Watir project is composed of several
other projects of which watir-classic, watershed and watir
webdriver are important.

3.1.4

Quick Test Profession

Quick Test Profession is an automated testing tools based on

graphical interface record playback capability [14]. It works
by identifying the objects in the application user interface or
web page and performing desired operations (such as mouse
and keyboard events).QTP uses a VBScript scripting language
to specify test procedures and manipulate activities.
Automated testing tool QTP provides the industrys good
solution for functional test and regression test automation
addressing every major software application and environment.
Quick Test Professional also enables testing Java applets and
applications, and multimedia objects on Applications as well
as standard Windows applications, It works by identifying the
objects in the application user interface or a web page and
performing desired operations (such as mouse clicks or
keyboard events); it can also capture objects properties [15].

3.1.5 Ranorex
Ranorex is a commercial and complete image-based detection
tool used for programmed testing [16]. Ranorex perform
testing based on Image detection and facility to record and
playback. It does not necessitate to study a scripting language,
since it is written in pure .net code using C#, VB.net and Iron
Python. Ranorex recommended for expanded projects with
new license for tools as it cost benefits but the support
restricted only to companies.

3.1.6 Test Complete

TestComplete is a testing automation tool formulated as Smart
Bear testing framework [17]. It makes available the testing of
windows and web applications and is one of the primary
functional testing tools in the world. TC is a graphical recordplayback automation tool which supports various testing types
and methodologies: unit testing, functional and GUI testing,
regression testing, distributed testing.TC provide recording
and capabilities of generation of test scripts.

www.ijcat.com

3.1.7 Telerik
Telerik is a market-leading vendor of UI controls, end-to-end
solutions for web and mobile applications development across
all major development platforms [18]. Telerik empowers over
one million developers to create compelling experiences
across web and mobile applications taking the advantage of
record and playback tested scripts to validate user interaction
with the system. [18]Telerik Perform complex UI actions like
Drag-n-drop and pure UI actions on web pages and provide
comfort and speed web application testing against many
browsers by only change browser type and settings.

3.1.8

Coded UI

Coded UI is an automated testing framework that used for

analyzing and testing user interfaces. Developers create a
coded UI test that can test the user interface for an application
functions correctly [19]. Testing performs actions on the user
interface controls for an application and verifies that the
correct controls are displayed with the correct values.
Developer create coded UI testing cases by recording the
actions of user with applications or by writing test cases using
visual studio platform and then playback this scripts for
verification of user interactions.

3.2 Tools Features

The features below used for the evaluation process for
distinguishing the capability of each tool versus the others
[20]. Each parameter are listed with the up to date value based
on intensive searching at tools support website and last
research papers. Table below list all evaluation parameters
with the meaning of parameters.
Table 1: Evaluation Parameters
Features
Explanation
Cross platforms.
To what degree tool support
operating system
Cross Browsers.
How many browsers tools able
to work with
Record-Playback.
The ability of tool to record
scripts to be run under different
conditions.
Script-language.
Programming language used to
edit testing scripts or for the
creation of testing scripts
Ease of Learning.
Working with GUI easy or not
Data-Driven Framework.
The ability of tool to reduce
efforts.
Programming skills.
Require programming skills or
based on predefined steps
Online-Support.
Provide support or not for
sudden
situations
and
troubleshooting
Training-Cost (USD).
The cost of tool training cost if
exist
Debugging support.
Does the tool has the
mechanism to handle error and
provide debug or not
Report Generation.
Effective analysis for test script

406

International Journal of Computer Applications Technology and Research

Volume 4 Issue 5, 405 - 408, 2015, ISSN:- 23198656
Table 2: Evaluation study of automated web testing tools.

4. EVALUATION STUDY
There are a number of open source and commercial
windows, web and mobile application tools available in the
software market [21]. Although the core functions of these
tools are similar, they differ in functionality, features,
usability. Keeping in view the previous mentioned aspects,
we have selected the more usable web testing tools for
comparison taking in consideration tools that support recordplayback feature which are Selenium, Quick Test
professional, Test Complete , Ranorex , Sahi ,Telerik and
CodedUI. Our research work comprises of the analysis of
different automated web testing tools based on the features
each one support. For our comparative study we use the
current version of each open source tool and the demo of
commercial one. The table below constructed based on the
features listed before providing the mainly features chained
with each tool versus other tools. We drag commercial and
open source tools for two reasons, first the main concern of
each tester is how much the tool cost? And is it fulfill his
needs? The open source tools take the advantage of it has low
or no cost for use, maintain and distribute but the main
disadvantage is it is not straight forward to use. In the other
side, the commercial tools give the user the guide steps and
full support to do his work under license agreement. The
features not contain the experimental records for the tools but
only the usability features which chained with the tools.
The evaluation study presented in a tabular form providing
the evaluation study of the tools under study according to
criteria mentioned before. The study give the user the basis
view of how to select the suitable tools based on his/her
requirement .the study list usability features of each tool
against other tools and give the user near view of how to
make a selection.

www.ijcat.com

Tools/criteria

Seleniumweb driver

Sahi

Watir-web
driver

QTP

Pricing
(USD)

Open
source

Open
Source

Open Source

8000

Cross
Platform

Windows
Only

Windows
Mac

WindowsMac-Linux

Windows
Only

Browserssupport

ChromeFirefoxIE-Opera

All
Browsers

ChromeFirefox-IEOpera

IEFirefoxChrome

RecordPlayback

Support

Support

Support

Support

ScriptLanguage

Ruby-javapythonphp- java
script

Java script
-Ruby

Ruby based

VB
Script

Ease of Use

Experience
needed

No
experience

No
experience

Easy to
learn in a
short
time
Excel
files-text
filesXMLDB files

Ranorex

Test
Complete

1855

1,069

2,999

999

Windows 7
and Higher

Windows
Vista and
Higher

Windows 7
and Higher

IE-FirefoxOperaChrome

All
Browsers

IE Only

Support

Support

Support

Support

VB script

VBScriptC#-Jscript

VB.net-C#

VB.net-C#

Experience
needed

Experience
needed

Experience
needed

Experience
needed

CSVExcel-SQL

CSV-ExcelSQL

Excel
files-text
filesXML-DB
files

CSV-ExcelSQL

Windows
Only
except XP
IEFirefoxChromeSafari

Telerik

Coded UI

Data-Driven
Framework

ExcelCSV

CSV

XML- Excel
files

Programming
skills

Required

Partially

Partially

Partially

Partially

Required

Required

Required

OnlineSupport

Strong
Support

Strong
Support

Weak
support

Licensed

Strong
Support

Strong
Support

Strong
Support

Strong
Support

TrainingCost (USD)

350

No
training
cost

No training
cost

250

1087

449

349

1251

Debugging
support

Strong

Partially

Partially

Strong

Strong

Strong

Strong

Strong

Report
Generation

HTML

HTML

HTML,XML

HTML

HTML

HTML,XML

HTML,
XLS ,
PDF, CSV

HTML

407

International Journal of Computer Applications Technology and Research

Volume 4 Issue 5, 405 - 408, 2015, ISSN:- 23198656

5. CONCLUSION AND FUTURE WORK

Our research work comprises of the analyzation of different
automated web testing tools for not also commercial but also
involve open source tools. This study helping in selecting the
suitable tools based on multiple criteria. Selecting tools in this
area, it is important to consider multiple parameters which
vary among different requirements, many requests in the
market make the cost the first target to be considered, in the
other hand some open sources software didnt provide support
for its user as it work under user experience .The study
present each tools with features which in the same and
different degree with other tools and how each tool behave
against others tools features .This comparative study can be
the basis for developing a model facilitate selecting the most
applicable tools based on the needed requirements.
Our future work will encounter more tools and more features
also that will help in building a user based requirement model.
This model also will help researches to select tools helping
their research work.

6. REFERENCES
[1] Ms. Shikha maheshwari1 A Comparative Analysis
of Different types of Models in Software
Development Life Cycle International Journal of
Advanced Research in Computer Science and
Software Engineering Volume 2, Issue 5, May
2012.
[2]

Boydens, Jeroen. Location transparency and

transactions as first-class concepts in objectoriented programming languages. Diss. PhD thesis,
KU Leuven, 2008.

[3] Bellettini, Carlo, Alessandro Marchetto, and Andrea

Trentini. "TestUml: user-metrics driven web
applications testing." Proceedings of the 2005 ACM
symposium on applied computing. ACM, 2005.
[4] Prof. (Dr.) V. N. Maurya, Er. Rajender Kumar
Analytical Study on Manual vs. Automated
Testing
Using
with
Simplistic
Cost
Model,International Journal of Electronics and
Electrical Engineering ISSN:2277-7040 Volume 2
Issue 1 (January 2012).
[5] Jomeiri, Alireza. "A SURVEY ON WINDOWSBASED WEB TESTING TOOLS." International
Journal of Academic Research 6.4 (2014).
[6] Binder, Robert. Testing object-oriented systems:
models, patterns, and tools. Addison-Wesley
Professional, 2000.
[7] Harpreet kaur et al Int. Journal of Engineering
Research and Applications ISSN : 2248-9622, Vol.
Issue 5, Sep-Oct 2013, pp.1739-1743

www.ijcat.com

[8] Abha Jain, Manish Jain, Sunil Dhankar International

Journal of Engineering, Management & Sciences
(IJEMS)ISSN-2348 3733, Volume-1, Issue-1,
January 2014
[9] Angmo, Rigzin, and Monika Sharma. "Performance
evaluation of web based automation testing tools."
Confluence The Next Generation Information
Technology Summit (Confluence), 2014 5th
International Conference-. IEEE, 2014.
[10] Bruns, Andreas, Andreas Kornstadt, and Dennis
Wichmann. "Web application tests with selenium."
Software, IEEE 26.5 (2009): 88-91
[11] http://www.ibm.com/developerworks/library/wasahi
[12] "Watir Automated testing that doesn't hurt,"
[Online]. Available: http://watir.com//
[13] B. Marick, Everyday Scripting with Ruby: For
Teams, Testers, and You, The Pragmatic
Programmers, 2007, 2007.
[14] Dustin, Elfriede, Jeff Rashka, and John
Paul. Automated software testing: introduction,
management, and performance. Addison-Wesley
Professional, 1999
[15] Nguyen, Hung Q. Testing applications on the Web:
Test planning for Internet-based systems. John
Wiley & Sons, 2001.
[16] Jain, Abha, Manish Jain, and Sunil Dhankar. "A
Comparison of RANOREX and QTP Automated
Testing Tools and their impact on Software
Testing." IJEMS 1.1 (2014): 8-12.
[17] Dubey, Neha, and Mrs Savita Shiwani. "Studying
and Comparing Automated Testing Tools; Ranorex
and TestComplete." IJECS 3.5 (2014): 5916-23.
[18] http://www.telerik.com/teststudio
[19] Nagarani, P., and R. Venkata Ramana Chary. "A
tool based approach for automation of GUI
applications." Computing
Communication
&
Networking Technologies (ICCCNT), 2012 Third
International Conference on. IEEE, 2012.
[20] Mohd. Ehmer Khan, Different Forms of Software
Testing Techniques for Finding Errors,IJCSI
International Journal of Computer Science Issues,
Vol. 7, Issue 3,No 1, May 2010.

[21]http://www.softwareqatest.com/qatweb1.html

408