INTERNATIONAL JOURNAL OF SCIENTIFIC & TECHNOLOGY RESEARCH VOLUME 9, ISSUE 04, APRIL 2020 ISSN 2277-8616

A Comparative Analysis On Software Testing

Tools And Strategies
Pramod Mathew Jacob, Priyadarsini S, Renju Rachel Varghese, Sumisha Samuel, Prasanna Mani

Abstract: Current software industry‘s prime focus is on developing quality application programs. The basic business management principle ‗Customer
acceptability is directly proportional to the quality of the product‘ is admissible for software products too. Thus, testing phase has a vital role in improving
customer satisfaction of a software application. The various research analytics claim that nearly 30% effort of entire software development is used for
performing testing activities. Every software firm or application developers follow a typical custom set of testing strategies and uses some standard
testing tools for quality assurance. The project manager has to decide the testing strategy between manual and automated testing. In automated testing,
there are many tools available with different capabilities and performance characteristics. This review analyzes the performance metrics of various
testing tools and testing strategies used for enriching the quality of the application being developed. The review result may guide the project manager to
make the trade-off decisions for choosing the appropriate testing tools and testing strategies applicable for their project domain.

Index Terms: Software testing, Test script, Test cases, Automated testing tool, Manual testing, System testing.
——————————  ——————————

1 INTRODUCTION There is a wide set of testing automation tools available. Our

Software testing Software testing [1, 2] is a quality control work evaluates the performance and characteristics of most
activity which involves defect detection and correction. Testing widely used test automation tools. This review work also
can be performed at various stages of software development focuses on analyzing the efficiency and applicability of various
process depending upon the methodology and tools being testing strategies that can be followed in a software industry.
used and usually begins after the requirement confirmation
phase. The initial phase is at unit level where it mainly focuses 2 RELATED WORKS
on coding. After coding all software units, we perform K. M. Mustafa et al has classified the software testing tools
Integration testing for finding out the bugs in the software based on software testing methods [4]. They have evaluated
application. The ultimate purpose of software testing is to nearly 135 testing tools and categorized them over three types
prevent software from failure, ensuring the quality and of software products (web application, application software,
satisfying the stakeholders. Software testing can also be network protocol). Their analysis also points out which testing
mentioned as the process of performing verification and method has limited automation tools. T. E. J. Vos, et al has
validation to a software or an application that meets the proposed a framework for evaluating software testing tools
business oriented and technical-oriented functional and techniques [5] which is successfully used for some case
requirements that guided in its design phase and studies in EvoTest and FITTEST. But this framework fails to
development. Validating and Verifying (V&V) [3] is the process evaluate the performance constraints of automated tools like
of ensuring that a software meets the requirements mentioned test script generation time, test script evaluation time and
in Software Requirement Specification (SRS) document and much more. James B. Michael et al derived object-oriented
that it fulfils its intended functionality. It can be considered as a metrics for measuring the effectiveness of the software testing
methodology to ensure software quality. The terms can be tools and is validated using three testbeds [6]. The proposed
defined as follows: computational metrics can be used in evaluating the
effectiveness, quality, accuracy, performance and much more
Verification: It is the process of ensuring whether the software for a tool. Insha Altaf, J. A. Dar et al has performed a survey
product is built in the right manner. on Selenium tool in software testing [7]. They have discussed
about the basic features and functionalities of the Selenium
Validation: It is the process of ensuring whether the testing tool along with mentioning some of its application
developed software product is as expected. Testing usually domains. R. Angmo et al has evaluated the performance of
consist of three levels called Unit testing, Integration testing web-based automation testing tools [8]. They have analyzed
and System testing each of them has many sub levels of web testing tools: Selenium and Watir. Their evaluation results
testing strategies. These testing can be performed either using claim that Selenium web driver is much better than Watir web
human resources or using automation tools. driver, if suitable Selenium browser plugins are added. In most
of the related works, the authors mainly focusing on a
_______________________________ particular framework or testing aspects. Most of the tools used
are outdated now. So, our review focuses on the most widely
• Pramod Mathew Jacob is currently working as Assistant Professor used testing tools like Selenium and HPE UFT against some
in Providence College of Engineering Chengannur, Kerala, India, E- basic standard performance metrics.
mail: [email protected]
• Priyadarsini S is currently working as Assistant Professor in
Providence College of Engineering Chengannur, Kerala, India. 3 TERMINOLOGY
• Renju Rachel Varghese is currently working as Assistant Professor In order to analyze various testing techniques, it is better to
in Providence College of Engineering Chengannur, Kerala, India. recollect the basic terminologies used in software testing:
• Sumisha Samuel is currently working as Assistant Professor in
Providence College of Engineering Chengannur, Kerala, India.
• Dr. Prasanna Mani is currently working as Associate Professor in Test case: A test case is a set of test data that is to be
Vellore Institute of Technology, Vellore, Tamil Nadu, India. inputted, along with expected results and resultant conditions,
3510
IJSTR©2020
www.ijstr.org
 INTERNATIONAL JOURNAL OF SCIENTIFIC & TECHNOLOGY RESEARCH VOLUME 9, ISSUE 04, APRIL 2020 ISSN 2277-8616

designed for a particular testing module in order to ensure prepares test plan and suitable test cases which is tested over
system functionalities [9, 10]. Consider the example test cases the application to verify the behaviour of Graphical User
for the ATM Personal Identification Number (PIN) field Interface (GUI), functional and non – functional requirements
validation in a web page. Assume that the PIN should be a as illustrated in Figure 1. But performing manual testing
four-digit integer number. Then the test case should be as requires a large amount of human intervention and the
follows: presence of an experienced - skilled person to design an
appropriate test suite.
Valid test case: PIN = 1234 // 4 –digit integer
Invalid Test cases:
PIN = 123 // non 4 –digit integer
PIN = abcd // not an integer
PIN = 12a4 // combination of character and integer
PIN = Blank // No Input data

In this case test cases can be defined using the Equivalence

class partitioning method [11]. There is one valid and two
invalid classes.
Valid class: (1000 ≤ PIN ≤ 9999) // 4-digit numbers.
Invalid classes are: (PIN < 1000) and (PIN > 9999)

Test suite: It is mathematically a set which consist of all the

test cases as set elements. A software can have infinite Figure 1: Process involved in manual testing
number of test cases. A good test suite should have minimal
test cases which covers most errors. Test suite to validate the In automated testing, execution of test cases is supported by
ATM PIN field is: automation tools. Automated testing is quite beneficial in case
{2000, 999, 10000, 12a4, abcd} of large projects. Here automation tools perform tests that
repeat predefined actions, matches the developed program‘s
Error: Error is the degree of mismatch from actual result and probable and real results [18].
expected result. It represents mistake made by code
developers [12]. Though error and bug sounds synonyms, they
are not exactly the same. Error is the mistake found by tester.
When developer accepts this mistake, then it is called as a
bug.

Fault: Fault is an incorrect code statement, function or data

interpretation in a computer program which makes the
program or software system to generate an unexpected result.
Fault leads to error.
Figure 2: Processes involved in automated testing
Bug: Bug is a fault in the code block which generates an
unintended result. It is normally the mistake accepted by the Test scripts are used by automated tools to perform software
developer. testing. The automated tool traverses through the entire code
blocks for bugs, verify each system behavior by executing test
Failure: It is the inability of a software system to perform its cases. If the expected behavior matches with the test results,
expected functional and non-functional requirements. the automation tool confirms the quality of the product. The
Execution of a fault leads to failure. Figure 2 shows the process involved in automation testing. It
is not practically feasible to automate all test cases which
Defect: A defect is a mistake committed by programmer in involves some constraints regarding aesthetics and
coding or logic that causes a program to generate incorrect or appearance. There are pros and cons for both testing
deviated results. Normally a defect is detected when a failure strategies. Choosing which one to use among the two
occurs. depends upon factors like size of the project, time, availability
of resources and much more. The performance analytics [19,
Test script: It can be defined as a collection of instructions 20] of automated and manual testing are plotted in Figure 3.
applied on the system under testing, to ensure that the system
performs its specified functionalities and behaves as expected.

4 SOFTWARE TESTING STRATEGIES

Software industry usually follows two testing approaches:
Manual testing and Automated testing. In manual testing,
testers evaluate the software manually for the faults. Here the
tester plays the role of an end user and evaluates all the
possible features and functionalities of the developed software
to ensure its behaviour and quality. The tester manually
3511
IJSTR©2020
www.ijstr.org
 INTERNATIONAL JOURNAL OF SCIENTIFIC & TECHNOLOGY RESEARCH VOLUME 9, ISSUE 04, APRIL 2020 ISSN 2277-8616

performing tests different systems

Testing GUI &
Not possible Can be performed
Aesthetics
Performing Can be easily Very difficult and
Regression tests performed boring task
Execution of Build
Too difficult to
Verification Test Easier to perform
execute BVT
(BVT)
Small projects
Complicated and
Applicable projects with limited
large projects
functionalities

5 SOFTWARE TESTING TOOLS

PERFORMANCE EVALUATION
There are lots of software testing tools available in the market.
The most successful ones include Selenium, HPE Unified
Functional Testing (UFT), IBM Rational functional tester, Silk
Figure 3: Performance analytics of automated versus Test, LoadRunner, WinRunner and Test complete. The various
manual testing software testing tools can be classified based on their
functional features and their role in testing process as shown
If the project consists of too many test modules, then it is in Table 3. There is still research undergoing in developing
better to prefer automated testing [27,28,29] than manual testing tools to address other aspects of testing and validation.
testing. Because it is not feasible to derive all the During the STLC automation phase, the project manager
combinations of test cases for a project with too many coupled should choose an automation tool which is applicable to the
modules. If a project got more time available for testing, then project being tested. The choice depends upon factors like tool
do it manually, where the possibility of finding real time errors functionalities and features, testing team, budget of the project
is comparatively high than that of automated testing tools. The being tested, performance aspects of the tools and much
prime constraint in performing manual testing is the need of more. There are testing tools which is customized for a
experienced testers. The manual testing [30] efficiency particular programming language like JUnit for Java
depends on qualitative factors like experience and knowledge application and PyUnit for Python. Some tools like web
level[31,32] of tester, nature of the software module under test, application tool, desktop application tool and mobile
category of the software etc. The analytical comparison of both application tool are domain specific. Our comparative analysis
testing strategies is shown in Table 1. From the plotted graph, mainly evaluates two widely accepted software testing tools:
it is clear that automated testing performs far better than that Selenium and HPE Unified Functional Testing (UFT). HPE
of manual testing in most evaluation parameters. It is Unified Functional Testing (UFT) [21] which was earlier known
recommended to use automated software testing tools in as HP Quick Test Professional (QTP) provides features for
certain testing scenarios. As the software industry[33] is too test automation and functional testing for various software and
competitive, there is a wide room for software quality and environments. It is widely used for enterprise quality
reliability[34] in the scenario. Deadlines and product release assurance. It provides a GUI and features for keyword and
dates are more essential in this competitive market. In this scripting interfaces. It uses VBScript to write the test scripts.
scenario an intelligent software company will choose some HP UFT is a single console for verifying the interface,
good testing tools which may increase the efficiency of testing database and service layers of a software or application.
as well as to get the product deadlines met. Selenium [22] is an open source, portable testing framework
mainly focusing on web applications. Selenium has a record-
playback tool for authorizing tests without learning to develop
TABLE 1. AUTOMATED TESTING VS MANUAL TESTING test scripts. The prime components of Selenium framework
Criteria Automated testing Manual testing include Selenium IDE, Selenese, Selenium Remote control,
Speed of testing Faster Slower Selenium Client API, Selenium web driver and Selenium Grid.
Cost effectiveness
More cost
Slightly less For our evaluation we selected Windows (Ver. 10) as
effective operating system platform. We have installed the Selenium
Flexibility Not flexible Too flexible IDE [23] and web driver add-ons in Mozilla Firefox browser.
Not practically
Reusability Easily reusable
feasible HPE UFT [24] desktop application is installed with required
Infrastructure Comparatively updates. We evaluated the performance of two testing tools by
Less testing the login pages of our VIT University [25], Rediffmail,
required high
Need of Training
Highly
Not required
Facebook and Gmail. These login pages read username,
recommended password and a CAPTCHA from user and redirects them to
Usage of Tools Too high Slightly less homepage after successful validation. The test cases matrix
Turn-around time High Less
used for validating a login page is shown in Table 4.
Comparatively
Reliability Less reliable
more
Need of TABLE 2. TEST CASE MATRIX FOR LOGIN PAGE VALIDATION
Highly needed Not needed Test Expected
programming
Quality of testing High Low case Username Password CAPTC-HA outcome
Need of human ID message
Very few Huge 1 Valid Valid Valid Login success
resources
Mode of Concurrently in Sequentially 2 Invalid Valid Valid Incorrect Emp.
3512
IJSTR©2020
www.ijstr.org
 INTERNATIONAL JOURNAL OF SCIENTIFIC & TECHNOLOGY RESEARCH VOLUME 9, ISSUE 04, APRIL 2020 ISSN 2277-8616

Code system. If we are permitted to access CAPTCHA database,

Incorrect then it is possible to store the values for respective images in
3 Valid Invalid Valid
password an array and check the image. If image match, use that value.
Incorrect
4 Valid Valid Invalid
Captcha This is not a perfect solution though, access to CAPTCHA
database is not granted always. So, in our evaluation we
We have evaluated the tools using the following performance validated only the username and password fields.
metrics [26].
1. Mean Script Creation Time (MSCT): It can be defined 6 RESULTS
as the mean time to create test scripts. It is the average We have analyzed the above-mentioned performance
time taken to generate one test script. parameters of Selenium and HPE UFT tool. The results that
2. Mean Script Execution Time (MSET): It can be defined we obtained is plotted below in Figure 5.
as the mean time to execute test scripts. It is the
average time taken to execute a single test script.
3. Tool Learning Time (TLT): It is the average time
required for a single naïve user to learn the software
tool to use. Of course, learning level is unquantifiable. It
can be redefined as the average time taken for training
naïve personnel to familiar with the software tool.
4. CPU Utilization rate: Percentage of CPU capacity being
used for running the software tool. 5. Tool Reliability
(TR): It is the probability of failure-free operation of the
testing tool for a specific time period in a particular
environment. Its value usually ranges between 0 and 1.

We evaluated the test cases in Table 2 using the Record-Play

back feature of Selenium. The test script generated by
Selenium is shown in Figure. 5.
Figure 5: Performance evaluation of Selenium & HPE UFT

The results that we mentioned is an approximate average

value, though these performance parameters depends upon
various qualitative parameters like tester‘s knowledge level
and experience, software and hardware support, level of
testing, complexity of the project being tested, language used
for scripting and much more. But our results can be used as a
reference model for software testers to choose the testing tool
which is appropriate for their domain. Though Selenium uses
Record and playback feature, Test script generation time can
be consumed. We have noted the time for generating scripts
for multiple test cases and then estimated the average number
of scripts that can be generated in an hour. Mean script
execution time is also estimated in the same manner by noting
the time taken to execute each test cases. MSET also
depends upon the complexity of the test module. For
estimating tool learning time, we choose around 10 students
with basic programming knowledge. We trained them till they
are able to generate and execute test scripts. The mean time
Figure 4: Test data output taken for each individual is measured to approximate up the
TLT value. TLT is not purely quantitative though it depends
After saving the suitable test cases, we monitored the time for upon the skill of the persons being trained. CPU utilization is
creating each scripts. We have summed up all these to an average over a period of time. At any point in time, if the
estimate the Mean Script Creation Time (MSCT). It is found CPU is either busy (100 percent) or idle (0 percent). Though
that the MSCT depends on the number of actions or activities HPE UFT is a desktop application, it consumes much CPU
performed during the recording time. Script creation time is power than Selenium which runs in a web browser. The
directly proportional to the count of activities performed during comparative analysis of both functional and performance
recording. The phase test execution takes place afterwards, aspects of Selenium and HPE UFT are summarized in Table 3.
where we loaded the test suite script. We have noted the initial
time and final time for test execution to estimate the Mean
7 CONCLUSIONS
Script Execution Time (MSET). Then we repeated the same
Software testing plays a vital role in software quality
test suite and testing procedures to evaluate HPE UFT. But in
management. Manual testing and automated testing are the
HPE UFT it is not possible to directly write the script to verify
main strategies followed. Automation testing is always
the CAPTCHAs [27] without having access to the CAPTCHA
preferred by a software industry to improve productivity and
database which contain all CAPTCHAS values present in the
3513
IJSTR©2020
www.ijstr.org
 INTERNATIONAL JOURNAL OF SCIENTIFIC & TECHNOLOGY RESEARCH VOLUME 9, ISSUE 04, APRIL 2020 ISSN 2277-8616

efficiency. Though manual testing is time consuming and [10]. SriivasanDesikan, Gopalaswamy Ramesh, ―Software
costly than automated testing, it will figure out some errors Testing: Principles and Practices‖, Pearson.
which can‘t be found through test automation. Test automation [11]. Pramod Mathew Jacob and M. Prasanna, ―A
tools are widely used in software industry to increase the Comparative analysis on black box testing strategies,‖
productivity. But there is no tool which is perfectly automates International Conference on Information Science –
software testing. There are many software tools useful for ICIS –‘16, Kochi, India, 2016
various programming domains and software applications. But [12]. Rajib Mall, ―Fundamentals of Software
Selenium and HPE UFT are widely used tools where the Engineering‖,Third edition, PHI
former is open-source and later is licensed. HPE UFT can be [13]. http://www.tutorialspoint.com/software_testing/softwar
used for both webpage and desktop applications whereas e_testing_quick_guide.htm
Selenium restricted only for web applications. Though [14]. Ian Sommeriele, ―Software Engineering‖, Addison
Selenium is freeware, we cannot chose it always, because of Wesley.
the need of an experienced test professional. HPE UFT is [15]. Pressman, ―Software Engineering –A Practitioner‘s
comparatively easier to use and develops test scripts in less Approach‖.
time. Root cause analysis and recovery strategies are much [16]. http://www.softwaretestingclass.com/software-testing-
better in HPE UFT than that of Selenium. Thus our evaluation life-cycle-stlc/
results may guide a project manager to choose the best tool [17]. http://istqbexamcertification.com/what-is-software-
applicable for their project domain. testing-life-cycle-stlc/
[18]. http://www.softwaretestingclass.com/automation-
8 ACKNOWLEDGMENT testing-vs-manual-testing/
We acknowledge Providence College of Engineering, [19]. https://www.infosys.com/it-services/validation-
Chengannur for providing all the support for implementing this solutions/white papers/documents/choosing-right-
work. automation-tool.pdf
[20]. http://www.softwaretestinghelp.com/software-test-
metrics-and measurements/
REFERENCES [21]. https://en.wikipedia.org/wiki/HP_QuickTest_Professio
[1]. B. Beizer, ―Software Testing Techniques‖. London: nal
International Thompson Computer Press, 1990. [22]. https://en.wikipedia.org/wiki/Selenium_(software)
[2]. Glenford J. Myers, ―The Art of Software testing‖, [23]. http://www.seleniumhq.org/download/
Second Edition, Wiley India Edition. [24]. https://saas.hpe.com/en-us/download/uft
[3]. B. Beizer, ―Black Box Testing‖, New York: John Wiley [25]. https://peopleorbit.vit.ac.in/hr_login.asp
& Sons, Inc., 1995. [26]. Harpeet Kaur, Gagan gupta, ―Comparative study of
[4]. K. M. Mustafa, R. E. Al-Qutaish and M. I. Muhairat, automated testing tools: Selenium, QTP and Test
"Classification of Software Testing Tools Based on the Complete‖ Journal of Engineering Research and
Software Testing Methods," Computer and Electrical Applications www.ijera.com ISSN : 2248-9622, Vol. 3,
Engineering, 2009. ICCEE '09. Second International Issue 5, Sep-Oct 2013, pp.1739-1743
Conference on, Dubai, 2009, pp. 229-233. doi: [27]. P. M. Jacob and P. Mani, ―A Reference Model for
10.1109/ICCEE.2009.9 Testing Internet of Things based Applications‖, Journal
[5]. T. E. J. Vos, B. Marín, M. J. Escalona and A. of Engineering, Science and Technology (JESTEC),
Marchetto, "A Methodological Framework for Vol. 13, No. 8 (2018) ,pp. 2504-2519.
Evaluating Software Testing Techniques and Tools," [28]. 23. P. M. Jacob and P. Mani, "Software architecture
2012 12th International Conference on Quality pattern selection model for Internet of Things based
Software, Xi'an, Shaanxi, 2012, pp. 230-239. doi: systems," in IET Software, vol. 12, no. 5, pp. 390-396,
10.1109/QSIC.2012.16 10 2018. doi: 10.1049/iet-sen.2017.0206.
[6]. J. B. Michael, B. J. Bossuyt and B. B. Snyder, [29]. 24. Pramod Mathew Jacob and M. Prasanna, ―A
"Metrics for measuring the effectiveness of software- Comparative analysis on black box testing strategies,‖
testing tools," Software Reliability Engineering, 2002. International Conference on Information Science –
ISSRE 2003. Proceedings. 13th International ICIS –‘16, Kochi, India, 2016
Symposium on, 2002, pp. 117-128. doi: [30]. 25. V. Bose, R. Roy, M. Nadirsha, B. Raj, Ajesh M and
10.1109/ISSRE.2002.1173225 P. M. Jacob, "Gesture based painting system," 2016
[7]. Altaf, J. A. Dar, F. u. Rashid and M. Rafiq, "Survey on International Conference on Information Science
selenium tool in software testing," Green Computing (ICIS), Kochi, 2016, pp. 23-27.
and Internet of Things (ICGCIoT), 2015 International [31]. 26. P. M. Jacob, Muhammed Ilyas H, J. Jose and J.
Conference on, Noida, 2015, pp. 1378-1383. doi: Jose, "An Analytical approach on DFD to UML model
10.1109/ICGCIoT.2015.7380682 transformation techniques," 2016 International
[8]. R. Angmo and M. Sharma, "Performance evaluation Conference on Information Science (ICIS), Kochi,
of web based automation testing tools," Confluence 2016, pp. 12-17.
The Next Generation Information Technology Summit [32]. 27. Pramod Mathew Jacob and Prasanna Mani, ―A
(Confluence), 2014 5th International Conference -, Performance Estimation Model for Software Testing
Noida, 2014, pp. 731-735. doi: tools‖, ,‖ International Journal of Engineering and
10.1109/CONFLUENCE.2014.6949287 Advanced Technology (IJEAT), vol. 8. no. 4, pp. 248-
[9]. Van Vleck, T., ―Three Questions About Each Bug You 253, 2019
Find‖, ACM Software Engineering Notes, vol. 14, no.
5, July 1989.
3514
IJSTR©2020
www.ijstr.org
INTERNATIONAL JOURNAL OF SCIENTIFIC & TECHNOLOGY RESEARCH VOLUME 9, ISSUE 04, APRIL 2020 ISSN 2277-8616

[33]. Jisha Mariyam John and Hariharan R.L, ―An

Intelligent Rider Assistant System using Machine
Learning for two wheel vehicles,‖ International Journal
of Engineering and Advanced Technology (IJEAT),
vol. 8, no. 6, pp. 1361-1366, 2019.
[34]. Priyadarsini S, Junie Mariam Varghese, Aparna
Mahesh and Talit Sara George, ―Shopping Spree: A
Location based Shopping Application‖, International
Journal of Engineering and Advanced Technology
(IJEAT), vol. 8, no. 6, pp. 1451-1455, 2019.

3515
IJSTR©2020
www.ijstr.org