Software Automated Testing Guidelines

Malik Jahan Khan, Abdul Qadeer, Shafay Shamail

Department of Computer Science
Lahore University of Management Sciences (LUMS)
Lahore – Pakistan
[email protected], [email protected], [email protected]

Abstract previous release, to make sure that integration is

Today software marketplace is competitive and smooth. In a big enterprise application the amount of
volatile like never before. Commercial software software, to be regression tested, increases. In
complexity is increasing while time pressure to scenario two, mostly the client changes are too small
release high quality software is also on the rise. as compared to the size of the whole project. Again
On one hand more defects are probable due to
increased software complexity while on the the result is that once the change is implemented,
other hand organizations cannot afford to let slip inspection team has a lot of testing work as far as
many bugs in the field. In an effort to test an regression testing is concerned. Although software
application reliably, quickly and thoroughly, organizations try to circumvent this situation by
automated software testing has emerged as a intelligently analyzing and segregating software
viable solution to meet today’s challenges. parts which could potentially malfunction and hence
Although it is true that automated software are good candidate of regression testing. But this
testing tools have become very powerful, it method has its potential risks.
should be kept in mind that proper induction and Iterative development and changes initiated by
utilization of tool is also nothing less than a client are not the only situations when testing team
challenge. The primary focus of this paper is to
describe the common pitfalls and to recommend has lot of work to do. Different organizations
guidelines to maximize the success probability perform different testing cycles on their releases.
of tool induction. Similarly after each bug fix, localized regression is
performed. So the workload of inspection team is
1. Introduction ever increasing [5].
Most of today’s enterprise software is developed
using some variant of agile process like RUP [8]. In this paper, we are not discussing that how an
The main idea of these processes is to break-up the organization could choose from its repertoire of
whole big project into many small manageable parts. choices. Our focus is to present guidelines for an
Each part is released to the client, while the organization, which has already decided to go for
subsequent releases integrate with the older one. automated testing. Automated software testing is
These kinds of processes have their own benefits for comparatively a newer approach of testing and lot of
all the stakeholders. myths surrounds this technique. In section 2, we
Once a project is complete and the whole product have discussed various testing alternatives.
is fully deployed, the product goes into the Automated testing is one of them. In section 3, we
maintenance phase. Software in general and have discussed some benefits of automated testing.
enterprise applications in specific should not resist In section 4, a few challenges of automation have
changing. Due to this reason these days software are been discussed briefly. In rest of this paper, a
built using pluggable components. Any component prescription or guidelines for an organization which
could be changed anytime as the needs arise. There has already arrived at the decision to use automated
could be numerous reasons for the change. We are testing, have been suggested in detail.
not discussing the reasons to initiate these changes
but one thing is obvious that in today’s volatile 2. Testing Alternatives
world change is inevitable in the software. To meet a certain quality level, an organization
In the above two scenarios, it is notable that could have the following alternatives at its disposal:
software testing team has to do lot of regression
testing. In scenario one, whenever a new iteration is 2.1 Commit their inspection team to work
complete and it is merged with the previous release, longer and harder
the inspection team has to thoroughly test the new
functionality while running the regression test on the Our local software industry is notorious for longer
working hours. Ask any software engineer and he
will tell you his wish to keep a balance between his recorded script could be enhanced by means of
personal life and work. It is very common for programming. There are different automation tools
organizations to commit their current human for white box and black box testing. For example
resource to work longer and harder. But this could Rational’s Purifier is used to detect any memory
only serve as a short term solution. There is always leaks in C / C++ code. Mercury Interactive’s
the hazard that people will burn out and the turnover WinRunner and Quick Test Professional (QTP) are
rate could increase due to people leaving the examples of GUI based functional black box testing.
organization. In any case there is always high
probability of missed bugs, slipped into the field. 3. Benefits of Automated Testing
Automated testing is a very powerful technique. But
2.2 Increase the personnel in the inspection to extract this power proper induction and usage of
team tool is mandatory. Let us discuss the major benefits
of an automated testing tool.
The second option is to increase the testing
personnel in the inspection department.
Organizations need to conduct cost benefit analysis 3.1 Fast
before hiring new resources. Hiring experienced Although automated testing tools mimic or simulate
resources is more costly while hiring fresh graduates a human user but they do it very quickly as
and training them in-house has its own cost and compared to a human being. In manual testing the
risks. The direct costs are in terms of pays and rate of test case execution varies due to many human
trainings. The in-direct cost is that the experienced factors like a person is not equally productive the
people have to devote their time to train new whole day. On the other hand an automated tool
resources. There is always some risk associated, if could run the script with the same high speed each
new resources are put on work without giving them time, it is executed. It dramatically decreases the
enough time to understand the business domain of testing time and hence utilizing the inspection time
the software. to the fullest.

2.3 Give more testing time to the same team 3.2 Reliable
The third option is to give the same testing team In manual testing, there is always a chance of human
more time, so that they could test properly. But it error. Automated testing tools, on the other hand,
would be against the spirit of an agile process if run the scripts reliably each time. Exact same steps
system testing is consuming a big chunk of the total are followed every time, the script is run.
iteration time. Software business is very competitive
these days and organizations are looking ways to 3.3 Repeatable
expedite the software delivery. So this option is no
more an option for most of the organizations. The recorded script is repeatable. One can test how
a website or application reacts after repeated
execution of the same operations. Repeated
2.4 Do not do anything and rely on whatever execution becomes very cumbersome in manual
testing is possible in given time testing.
Option four is again not viable. These days,
organizations have to be proactive to stay profitable 3.4 Programmable
in the business. It should be noted that using this In almost every script, the recorded script is
option will result in decreased product quality by enhanced by means of programming elements.
slipping the bugs in the shipped release. Later, these These programming elements enhance power of the
bugs would be reported by the client. These field scripts. Now they can verify a larger range and
defects not only reduce customer satisfaction but variety of functional requirements.
also put the organization under pressure as they now
have to take care of bigger backlog of field bugs
along with new release. So neglecting problems is 3.5 Comprehensive
no solution because problems will keep on coming One can build a suite of tests that covers every
back with more intensity. feature in a website or application, hence, increasing
the code and data coverage. It is always desirable to
2.5 Use automated testing test the complete functionality of the software.
The fifth option is to use the automated testing.
Automated testing is a technique to test software by 3.6 Reusable
means of a software tool, which records the user One can reuse tests on different versions of a
behavior and later it could replay the recorded script website or application, even if the user-interface
to mimic or simulate the human behavior. The changes.
4. Challenges of Automation recommended that an organization try to buy a
Though automated testing has a lot of benefit, but it comparison report of automated tools. In market,
also has some associated challenges. Following list every vendor claims that its tool is easy to use,
includes some of the major challenges of friendly, and reliable and your non-technical people
automation. Our suggested guidelines will be helpful can easily learn and use it [9]. But you need to
to face these challenges [3]: evaluate after a comprehensive survey and then
i. Selection of Test Tool select the really stable tool.
ii. Customization of Tool
iii. Selection of Automation Level 5.4 Proof of Concept
iv. Development and Verification of Script Once the tool choice is narrowed down (not more
v. Implementation of Test Management System than two), the next task is proof of concept. It is
very important to make a dedicated team to work on
5. Guidelines [2], [7], [1] proof of concept. Their task is to verify the tool
Many expectations are not completely met in capabilities by having first hands on experience on
automated testing, but it really adds a lot of benefits, the organization software. Few automated software
even done by right people in right environment at are available for limited time trial version. Some
right time [6]. Cost/benefit analysis is very times, organizations need to buy at least one license
important before going for automated testing. for this purpose. This phase is very crucial because
Adapting automated testing for trivial and small company decision greatly depends on the results of
cases may be quite expensive. So, experienced this study. So it goes without saying that this group
decision makers play an important role here. has the responsibility to appropriately selecting the
Following suggested guidelines will be quite helpful portions of software to be tested using the tool,
to adapt a successful automation. which are representative of whole application.

5.1 Technical and Managerial Decision 5.5 Trained People

Incorporating test automation could be a To conduct proof of concept, company need some
technological decision at the beginning but people who know automated testing. So company
ultimately technical managers have to defend their could either hire people or could train their resources
proposal in front of organizational management. for automated testing. To train people could be a
Different people have their own point of views about costly option, but to conduct proof of concept,
different things. The gap between different people company needs people who are with the
could be wider depending on different organization for quite some time and they
circumstances. So it is very important to discuss the understand the company software domain well.
automation decision inside out to reach a common
understanding among stake holders. Time, cost and 5.6 Automated Test Script
resource requirements should be crystal clear among
the management levels. Writing automated test script is just like any other
development activity. The difference is that the
requirement gathering and high level design is not
5.2 Planning Automated Testing needed. Design, coding and testing of automated
Appropriate plan is required to introduce automated test script code is very important. Don’t
testing in the organization. This plan should have underestimate this point because consider how
measurable milestones, shared with the upper sarcastic it would be that an organization inducted an
management as well. automation tool to increase their testing efficiency
but they ended up with more bugs slipped into the
5.3 Market Survey field because the automated script wasn’t able to
detect it due to a bug in its script. This is a major
Market survey is necessary. This market survey has cost of test automation. Again the test script writer
two dimensions. Firstly the organization should isn’t the best person to test it.
thoroughly know the kind of software they want to
apply automation on. They should also get
automation success and experience of other 5.7 Properties of Test Script
organizations, working in similar domain. Mostly Test script needs all the qualities of a good software
other organizations don’t provide such data. So like maintainability, proper documentation etc. This
company has to invest either for a related consultant is because if test scripts could not be changed
or they need to buy the industry survey reports. It quickly and easily, then they are almost useless. All
should be noted that there are many automated this needs experienced test script writers. It is
software testing vendors in the market and it may recommended that the organizations identify the
not be feasible for the organization to try every portions of the program, which are least probable to
product to decide among the tool. It is highly change and yet important for regression test. If they
couldn’t find such functionality and they don’t have 5.12 Incorporating Intelligence
experienced test script writers, it is recommended
not to do test automation at this point. Automated testing tools can not mimic human
intelligence. When a test engineer manually executes
the steps of a test case, s/he could intelligently
5.8 Significance of Test Case change his/her actions as needed. Incorporating
It is a myth that automated testing can catch more such customization in the script is very difficult, but
bugs as compared to manual testing. It is necessary not impossible.
to understand that an automated test script could
only be as efficient in finding bugs as the original 5.13 Exhaustive Testing
test case. Test script is a translation of a test case. If
a test case skips something, the automated test script Exhaustive testing is yet not possible. Although
will miss it. code and data coverage is greatly increased, but yet
testing each branch of code with each permutation of
data is not possible.
5.9 Manual Test Cases
For test automation, detailed test script along with 5.14 Time to Run Automated Test Tool
test data is necessary. Some times, a level of
abstraction hides the detail of test case because the In GUI-based automated test tools, it is difficult to
testing engineer has the domain knowledge. If start writing scripts if code is yet not ready. That
automated test script writer is someone else, the means, it may not be possible to use the tool from
proper detail is necessary. If the organization could the very beginning of a project. The best situation is
train all of their test team for automation, so that when first release is manually tested and deployed.
they could write the test script for their own test Then while release two is under development, the
cases, that is the ideal case. If it is not possible and automated scripts of release one should be written
there are only few people who know automation for the purposes of regression testing.
well and they have to automate everyone’s test cases
then it is recommended to increase the quality and 5.15 Testability
detail of manual test cases. The automated test script Testability becomes one of the most important traits
should be tested by the person, who originally wrote after the induction of a costly tool (The cost of a
the test case. sophisticated tool is around $4000 per license).
Testability is the quality of a software that how easy
5.10 Significance of Automation Tool or difficult it is to test the software. Many third party
Give your automation tool and yourself sufficient components may not be recognized by the tool. In
time before judging its usefulness. Although it is these conditions organization should provide the
very important to quantitatively measure the developers with a list of GUI gadgets, which could
efficiency due to test tool but the conclusions should be recognized by the automation tool.
be made after sufficient time. The organization
should rely on quick tutorial based tool learning 5.16 Automation Success
approaches. Vendors provide very good tutorials The Software Test Life Cycle (STLC) is the
with tools. The best bet is to put the people to write roadmap to automation success. STLC parallels the
the real test script after learning most important and Software Development Life Cycle (SDLC), and
basic features. Advanced features could be learned includes phases planning, analysis, design,
as needed. It decreases training cost. The automated construction, testing (initial test cycles), bug fixes
test scripts may not have all those nice features like and re-testing (final testing and implementation) [4].
maintainability in the beginning, but after some Lack of planning earlier is one of the biggest reasons
time, organization would have valuable work force. of automation failures. A lot of automation work
One important factor is to hire people in inspection should be done before the software is available.
team with right qualification, so that they have “Start Early” is the big lesson, practitioners
programming background. This thing really recommend for successful automation [4].
increases the speed of script coding.
5.17 Part of Culture
5.11 Requirements Change
Make the automation part of culture. Though, it
Widespread application change could make all the can’t be guaranteed to experience a successful
test scripts useless and they may need to change. If automation in the first go, however, “experience and
some part of software is highly probable to be learn” leads to successful adaptation of this
changed, it shouldn’t be automated. phenomenon. Finally, this success will make
automation part of the organizational culture. The
complex and hardly controllable future projects need [6] Kerry, Automated Software Testing – A
a pre-existing culture of automation. perspective,
http://www.testingstuff.com/autotest.pdf
The computer which runs automated scripts may
not be used in parallel for other activities. It is a [7] Pettichord, B. Seven steps to test automation
misconception that a test engineer will manually test success, http://www.io.com/~wazmo/papers/
application while the tool is running its script in the seven_steps.html, June 2001.
background. When the automated tool mimics the
user behavior by clicking the gadgets, the operating [8] Pressman, R. S. Software Engineering, A
system thinks that it’s the active window and hence Practitioner’s Approach, Sixth Edition,
brings that window in the foreground. That means McGraw Hill Education, 2005.
that a test engineer may not do something else, while
test script is running in the background. So either [9] Zambelich, K. Totally Data-Driven Automated
separate machines are required or test engineer may Testing, A White Paper of Automated Testing
need to observe running script. Some times, un- Specialists Inc., http://www.sqa-test.com/
attended scripts are run at the close of business to w_paper1.html, 1998.
run over night. But again writing unattended scripts
is more difficult than ordinary scripts.
6. Conclusions
Every organization is unique and has its own needs.
It is difficult to recommend “one size fit all”
solution. But following the guidelines given in this
paper increases the probability of success and could
make the automation experience satisfactory for all
the stakeholders. These guidelines might prove to be
quite effective when organizations customize them
according to their own business architecture,
available human resources, nature of the project,
level of quality needed and budget constraints.
Acknowledgements
We are highly thankful to Tauqeer Hussain, Tariq
Fayyaz and Zeeshan Ali Rana for their valuable help
and guidance towards completing this work.

References
[1] AutoTester Inc. Best practices guide to
automated testing, A document of basic
description of Automated Testing,
http://www.autotester.com/content/pdfs/BPG-
pdf.PDF, 2002.
[2] Dustin, E. Lessons in test automation, Software
Testing & Quality Engineering,
September/October 1999 pp. 16-21.
[3] Hughes Software Systems Ltd. Test Automation,
http://www.hssworld.com/whitepapers/whitepap
er_pdf/test_automation.pdf, December 2002.
[4] Isenberg, H. M. The Practical Organization of
Automated Software Testing,
http://www.automated-
testing.com/PATfinal.htm, 2005.
[5] Kan, S. H. Metrics and Models in Software
Engineering, Second Edition, Pearson
Education, 2002.