White Paper

Test Automation
“To be or not to be”

Testing is a form of Software Business Insurance. More coverage means less risk.
And that’s good news for everyone.

Author: Adtiya Jain

 White Paper
Test Automation

Overview
To be or not to be, that is the test. It’s a dilemma that many test automation at no point can replace the importance of manual
managers face when considering test automation. Test testing or, for that matter, manual testers. Test automation
automation can be a mixed set of affairs for organizations in combats repetitive, mundane tasks and helps speed up
terms of ROI based on varying results from different companies. execution. Moreover, it is not a substitute for walk-throughs,
Given this set of mixed inspection, good project management, coding standards and

“Some consider perceptions, many

makers choose to avoid the
decision solid configuration.

test automation long-term cost, and with it, the • Immediate reduction in efforts.
a “pill for every quality benefits that test With test automation, initial phases are often intensive,
ill,” greatly automation can offer. Some requiring a great deal of effort during setup. Once the scripts and
overestimating consider test automation a “pill the optimized test suite are ready, there is a considerable
for every ill,” greatly
its role in their decrease in resources and effort required. Based on the maturity
overestimating its role in their of the product, test automation requires and involves a certain
overall test overall test strategy. Still others degree of maintenance effort.
strategy. ” understand the boundaries of
this “magic pill,” take a • 100% test coverage.
marathon approach and build their entire test automation
One cannot test anything and everything. The general
concept around the existing overall test strategy, reaping
philosophy is to primarily test what is important and repetitive.
maximum benefits. But of course, nothing is perfect. Test
automation, despite all of its glory, should only be used as a • Anyone can automate.
supportive role, and never as the primary – or only – activity in
an organization’s software improvement endeavors. When automating tests, cases are scripted; this involves coding
and technological development knowledge.
In this whitepaper, we address key concerns and challenges
around test automation to help shed light on the truth behind • Running tests quickly and more often results
this activity. in better software production.

Myths about test automation that affect It’s not about speed and quantity, but generally, running and

good decision making continuously re-evaluating tests are what result in better software.

• One can achieve 100% test automation.

What drives test automation? Is it cost or
is it quality?
The assumption or goal of 100% test automation is not just
impractical -- it’s impossible. About 90% of organizations only undertake test automation as a
response to their failure in completing testing in the assigned
• Test automation is a one-time investment. timeframe. Sometimes, despite best efforts, quality goals have
Often misconceived as a “fire and forget” tool, test automation not been met, resulting in multiple defects in production.
involves and demands continuous investment in the essential Essentially, test automation then becomes a last-resort, giving
three: people, process and tools. management three alternatives to choose from:

(a) Hire more manual testers and grow the team organically.
• Test automation will simply do away with
manual testing. (b) Grow the team inorganically by outsourcing some part of

This assumption cannot be further from the truth. test their test activities to a third-party.

1
 White Paper
Test Automation

view of performance test automation – simply because they

“A dopting test automation without
know that it is logically not feasible to do it manually.
proper practices and process (both
development & testing) in place will • Quick Tips:
1. Automating an application whose UI changes frequently
result in a negative ROI.”
requires significant maintenance. By the time necessary
changes have been made to the script, the UI will change again,
(c) Begin evaluating software test automation tools and
rendering most efforts useless.
strategy.
2. Manual testing is recommended when the lifespan of the
In the long-run, test automation not only helps organizations
application is short and not many releases are planned.
deliver quality software faster, but it enables them to do so at a
fraction of the cost. This economy of scale is not possible with
Advantages of test automation over
options (a) and (b).
manual testing
Only 10% of organizations (those with matured test processes
and practices) adopt test automation to help with cost • Improved accuracy: Even the most seasoned and
reduction. Adopting test automation without proper practices professional tester will make mistakes during monotonous manual
and process (both development & testing) in place will result in testing. A tool performs the same task precisely every time.
a negative ROI.
• Increased test coverage: Complex test cases become
What to automate and what not to easily executable with test automation, providing coverage that
automate? was previously impossible with manual tests.
Test automation is all about identifying the right things to
automate – after all, 100% adoption is impossible. Ideal
• 24/7 test environment: Test automation allows for
scheduling unattended test runs 24/7 -- the equivalent of having
candidates for test automation include:
three daily 8-hour shifts of manual testing.
• Sanity tests: Basic functionality rarely changes, making
sanity testing ideal for automation. This is particularly • Close alignment of test & development activities:
advantageous when handling multiple releases on a monthly basis. Test automation, when integrated with continuous-build
software, provides a collaborative and continuous build-and-test
• Regression tests: Save on costs and allow testers to environment. This is particularly of importance in an agile
solely focus on new functionality verification, avoiding repetitive environment. As soon as a build/code is submitted by the
tasks and eliminating risk in code change or bug fix issues. developer, tests are run automatically, notifying the developer of
any errors. This increases the efficiency and effectiveness of
• Unit tests: Inexpensive to write and maintain, unit test developers and testers alike.
automation results in the highest ROI, providing value to the
team multiple times per day. • Faster time to market: Test automation provides testers
the ability to execute tests 24/7 on multiple machines
• Performance tests: Manually performing controlled simultaneously, allowing complex and long regression testing to be
web application tests with hundreds or even thousands of users executed in a fraction of time, and faster software delivery.
is quite the challenging feat. With test automation, users can
simulate a large quantity of virtual users to check application • Increased reusability: Done properly, test automation
load capacity. Management generally possesses very optimistic offers immense benefits to testers in terms of case reusability.

2
 White Paper
Test Automation

“Test automation There are many

build (500 test cases) is released every week. During the first year, we have
only considered the last 6 months as the initial phase which will comprise
applications where
provides test engineers basic functionality
of tool/process setup and tester training.

Investment on Test Automation = Tool Cost (10 licenses) +

the space to be intuitive remains the same;
Training Cost (if training internal resources) + Machine Cost (10
take, for example, the
and allow them to focus machines)
case of e-commerce
their efforts on more applications, which
#
The cost of manually executing/running tests will increase as a result of

”
annual increases in test cycles and the number of weekly builds and/or the
rigorous tests. are 80% similar. The
number of test cases corresponding with every build.
same test scripts, with
little or no modification, can be used to test other applications, an
As seen in the table above, the difference and the cost advantage
option that is unavailable via manual testing.
comes only in terms of test execution. With test automation, the

Benefit Analysis testers will no longer be required to spend their valuable time on
retesting something that has already been tested. Test automation
In this section, we will highlight the cost benefits that test
provides test engineers the space to be intuitive and allow them to
automation has to offer in terms of real ROI. To showcase this,
focus their efforts on more rigorous tests.
we have taken some assumptions (based on experience from Table 2
several automation projects) which may vary by small 2 Year(2 cycles)
nd

percentage points from project to project. Test Automation Manual

• Number of test cases per test build: 500 (1 build every week) Tool Cost NIL NIL
• Average cost of a tool: $12,000 (single license)
Cost of scripting/test NIL NIL
design
• Hourly rate of a manual tester: $50
Cost of NIL $204,000#
• Hourly rate of a test automation engineer: $60 executing/running test
• Manual Maintenance Effort required/week: 4 hours Cost of maintenance $23 040 , NIL
• Machine cost: $400 (each)
Total $23 040 , $204,000
ROI 135%
• Time to automate a single test case (script development): 1 hour

• Maintenance Effort required/week: 8 hours A note of caution, as time passes, new development techniques
will become available and new sets of requirements will be
• Time to manually execute a single test case: 10 minutes
requested. As a result, the application will incorporate a new set
• Time to write a single test case manually: 30 minutes
of components, which the tool may no longer be able to
• Resource training cost $10,000 recognize. Consequentially, one will have to upgrade or replace
*Considering two test cycles in a year, i.e. a test cycle of 6 months each. A the tool altogether, involving re-investments such as new
Table 1
licenses, tester training and script development. This could lead
1st Year(6 months/1* cycle)
to a reduced ROI, potentially urging firms to revisit their
Test Automation Manual
automation strategy altogether.
Tool Cost $134,000 NIL
Cost of scripting/test
design
$30,000 12500 Choosing the right automation tool
Cost of NIL 102000# In the long run, selecting the right automation tool is the most
executing/running test
important factor that will ultimately decide the success of your test
Cost of maintenance $11,520 (24 weeks) NIL
automation project - poor choices unfortunately often result in
Total $175,520 $114,500
project failure. During the culling process, current application
ROI -45.53%

3
 White Paper
Test Automation

portfolios and future changes should be taken into consideration. additional or preferred functionality to the test automation engineer.
The following are some basic features and factors to think about Every test automation team/organization has their own style of
during the selection process: working and standardized processes that they like to replicate with
• Ease of use test automation. A test automation framework allows
• Support for different types of testing organizations/teams to modify or enhance the underlying tool,
• Functionality without actually making any physical changes to the tool, in order
• Performance to suit their specific requirements and style of working. As an
• Automated scripting support using different scripting languages example, a test automation tool can only provide basic reporting
• Results and reporting based on industry best practices and analysis, which a test manager may find to be inadequate. To
• Support for cross browser and cross platform testing overcome this, they may develop a program that based on the input
• Support for integration with 3rd party open & commercial tools received from the underlying tool can generate a more
(test management & continuous build environment) comprehensive report.
• Object recognition
• Inbuilt exception handling or recovery mechanism Various types of available automation framework
• Multi-user support options are:
• Parallel execution
• Keyword-driven
• Inbuilt data management
• Data-driven
• Hybrid
“Too often, customers wind up • BPT

paying more for products that The choice of a test automation framework, again, depends on the
sound new and exciting but were individual objectives that one would like to achieve vis-à-vis the

”
underlying test automation tool and the overall test automation
not required in the first place. strategy.

The list goes on, but ultimately, tool selection is one of biggest A test automation framework provides certain added advantages
challenges to tackle before going for automation. It is very important with respect to test automation:
to identify the requirements, explore various tools and its • Ease of use and reduced learning curve
capabilities, set expectations and go for a Proof of Concept (PoC). • Faster test creation
While attractive pricing, branding and long lists of features are • Easier maintenance
tempting, it is more important to first compare the tools and analyze • Comprehensive reporting based on customized metricises
its individual benefits before making the final decision. Too often,
customers wind up paying more for products that sound new and Conclusion
exciting but were not required in the first place.
Test automation, if planned and used effectively, can be a powerful
tool in an organization’s inventory to help deliver quality software
Test Automation Frameworks faster and at a fraction of cost.

Test automation tools are standard products which, depending on

But in the end, it’s important to note that test automation should
the users’ needs, may or may not be a perfect fit. As with any other
only be considered as a special set of software that works to verify
application, a need is felt to enhance or add new functionality to the
the state of another piece of software. Used properly and for its
underlying tool for ease of use. Test automation framework is a
anticipated purpose, test automation can lead to better results and
piece of software that sits on top of an automation tool, to provide
overall success all around.

4
 White Paper
Test Automation

About AgreeYa About the Author

AAgreeYa is a global systems integrator delivering competitive For more than a decade, Aditya Jain has worked on a multitude of
advantage for its customers through software, solutions, and testing projects from inception to operation. Specifically, his vast
services. Established in 1999. AgreeYa is headquartered in experience includes significant contributions to the establishment
Folsom, California, with a global footprint and a team of more than of the Test Centre of Excellence (TCoE) for several large
1,700 staff across offices. AgreeYa works with 500+ enterprises.
organizations ranging from Fortune 100 firms to small and large
Possessing strong working knowledge of multiple test automation
businesses across industries such as Telecom, BFSI, Healthcare,
tools, including open source and commercial, Aditya has guided
Pharma/Life Sciences, Utility, Technology, Public sector and
numerous clients in implementing ‘NextGen’ test automation
others.
solutions, helping them leverage the cost benefits of these tools.

AgreeYa has received considerable recognition in the industry for Aditya holds a bachelor’s degree in engineering from the College
its technology leadership, quality processes, and customer of Engineering, Roorkee.
success that includes Microsoft Gold partner, SEI CMMI, ISO
9001:2015. We engage with our customer as a trusted partner to
live our motto “…building our future on your success”.

AgreeYa’s software portfolio includes QuickApps (award winning

suite of SharePoint web apps and pre-built templates -
www.quickapps.agreeya.com), Site Administrator (gain insight for
improved compliance and governance of your SharePoint
environment - www.siteadmin.agreeya.com), Recovery Manager
(solution for rapid and scalable SharePoint content restores -
www.recoverymanager.agreeya.com), BeatBlip (simplify software
test automation - www.beatblip.com), and Cogent (comprehensive
end-to-end case management solution for collections and law
firms - www.cogentcollections.com).

As part of its solutions and services offerings, AgreeYa provides

intranet/portal, content management, and collaboration, cloud
and infrastructure solutions, business intelligence and big data
analytics, product engineering, application development and
management, independent software testing, and staffing (IT,
Non-IT, and risk/compliance/AML) solutions.

For more information, visit www.agreeya.com

© Copyright 2018. AgreeYa Solutions Inc. All rights reserved. No part of this document may be reproduced, stored in a retrieval system, transmitted in any form or by any means,
electronic, mechanical, photocopying, recording, or otherwise, without the express written permission from ‘AgreeYa’. The information contained herein is subject to change
without notice. All other trademarks mentioned herein are the property of their respective owners.