Test Automation Micro-Credential

Syllabus

Copyright Notice
Copyright AT*SQA, All Rights Reserved
 Table of Contents
Test Automation
5 Introduction
6 Selecting Test Automation Candidates
9 Building Maintainable Test Automation Software
11 Benefits of Automated Testing
13 Test Automation Risks
14 Test Automation Success Factors
17 Test Automation Tools

References
18 ISO/IEC/IEEE Standards
18 Trademarks
18 Books
19 Other References

Copyright AT*SQA,
All Rights Reserved Test Automation Micro-Credential Syllabus 2
General Information
STUDY TIME – 200 MINS.

KEYWORDS

automation engineer, data-driven, emulators, keyword-driven,

Copyright AT*SQA,
All Rights Reserved Test Automation Micro-Credential Syllabus 3
LEARNING OBJECTIVES FOR TEST AUTOMATION
Introduction
Selecting Test Automation Candidates
(K1) Recall how ROI for test automation is determined
(K2) Explain the factors to be considered when determining if a project is well suited for test auto-
mation
Building Maintainable Test Automation Software
(K2) Explain the differences between data-driven and keyword- driven test automation
(K2) Understand the purpose of a test automation framework
(K1) Recall why test automation must be updated
Benefits of Automated Testing
(K1) Recall why test automation can increase test coverage
(K2) Describe the benefits of test automation
(K1) Recall how test automation can reduce costs
Test Automation Risks
(K2) Explain the risk factors for automation projects
Test Automation Success Factors
(K2) Explain the success factors for automation projects
(K1) Recall the recommended order for automation implementation
Test Automation Tools

Copyright AT*SQA,
All Rights Reserved Test Automation Micro-Credential Syllabus 4
Introduction

Test automation is a method of testing that uses automated test scripts rather
than manual test cases. The test scripts are small software programs written
in a scripting or programming language that accomplish the goals of a test by
controlling the inputs to a software module and verifying that the results match
the expectations. In its simplest form, a script mimics the user interaction with
the system under test (SUT) and is programmed to report any variances from
the expected behavior.

Test automation can be expensive to implement, but, when done correctly, can
save enormous amounts of manual testing effort. With effective use of test
automation, the quantity of tests executed can be increased, which results in
greater requirements coverage, shorter time for execution and higher reliability
and repeatability in the testing.

Copyright AT*SQA,
All Rights Reserved Test Automation Micro-Credential Syllabus 5
Selecting Test Automation Candidates
In order to maximize the efficiency of the Good project candidates for automation
automation effort, the test automation must be share some common characteristics.
designed for long term use and maintainability.
The return on investment (ROI) for test automation Expected long term usage – Because test
is based on the amount of money/time required to automation is generally expensive to develop due
build the initial software and to maintain it over to the cost of the tools and the effort to create
its lifetime (the investment), compared with the the test scripts, the resulting automated tests
time saved from the equivalent manual testing need to be run multiple times to recover the costs.
effort (the return). A test automation project A standard heuristic is that the target software
is only worthwhile if the return will be higher should remain in production for 2-5 years in order
than the investment. Not all software is suitable to regain the automation costs.
for automation and attempting to automate
unsuitable software will reduce the potential return
while increasing the investment.

Copyright AT*SQA,
All Rights Reserved Test Automation Micro-Credential Syllabus 6
Stable functionality and interface – To reduce Adequate tool support – Not everything can or
maintenance costs due to changes in the should be automated. While there are many tools,
programmed scripts, it is more cost effective to and the tool family continues to improve and
create the test automation at the point when the expand, there may be situations where the right
target software has stabilized. Stabilization is tool is not available. This sometimes happens
generally reached when a set of core functionality with code that uses unique interfaces or for
is working and will not be substantially changed. embedded software that is communicating with
It is also important for the interfaces used by the hardware. Some of these issues can be resolved
automation (e.g., the UI, the APIs) to be established with simulators (i.e., software that is created to
and unchanging. Changes to the interface will act like the software under test) or emulators (i.e.,
require updates/changes to the test automation software that is created to act like the software
scripts that access the SUT via that interface. working on the hardware under test), but
sometimes the only option is to create a custom
Need for frequent regression testing – The tool. Before this option is selected though, there
best automated tests are those that are used must be an understanding of who will provide on-
frequently. When the software under test has need going support for the tool and how much effort will
of frequent regression testing, test automation can be required to create the tool.
be utilized to effectively and quickly conduct those
tests and provide reliable and repeatable results.
The more frequently the tests are used, the higher
the return on investment.

Copyright AT*SQA,
All Rights Reserved Test Automation Micro-Credential Syllabus 7
Adequate skills in the team – Developing test Management support – Test automation can be
automation is a software development project an expensive process and requires management
and should be conducted as such, with proper support, understanding and approval. Tools can
design, architecture, development, testing and be expensive to purchase and may have license
documentation. While some tools provide a more renewal considerations. Specific programming
user- friendly interface (generally at a higher skills are required which may necessitate
cost), true programming skills are usually required hiring people with skills for the selected tools.
to either write the test automation scripts or to Because schedules can sometimes be delayed,
write “glue-ware” that will stand between the test it is important for management to have a clear
automation scripts and other capabilities (such as understanding of the work required to achieve the
opening and parsing emails). desired level of automation.

Copyright AT*SQA,
All Rights Reserved Test Automation Micro-Credential Syllabus 8
Building Maintainable Test Automation Software
Maintainable test automation starts with building with good knowledge of the domain to be tested.
an automation framework. The value of a test This allows for the best use of the programming
automation framework is that it provides a skills of the automation engineer while the testing
way to identify and control all test automation skills of the test analyst are leveraged to supply
testware. Without a framework, the result is often and control the data. This separation provides
an inconsistent and unmanageable collection of a higher level of maintainability by allowing a
automated test scripts. Building the framework single script to conduct many tests where the only
also requires implementing the right tools, variance is the test data.
selecting and training the right people and creating
the overall automation plan. The following steps Keyword-driven test automation goes one step
are needed to create an effective framework for further and uses action words, or keywords, to
test automation. describe the actions to be performed by the script.
The test analyst defines the actions that are to
Deciding on Data-Driven vs. be tested (e.g., add a user) and the data to be
used by the actions (e.g., first name, last name,
Keyword-Driven Approach and address). The automation engineer writes
a test script that will read the action and then
Data-driven test automation separates the test
perform the appropriate steps using the provided
script (the steps to be executed) from the data to
data. This type of coding allows the script and the
be used (for input and verification). The data is
data tables with action words to be reusable for
usually accessed by the script from a spreadsheet
multiple tests and limits the areas where changes
or database and is maintained by a test analyst
are needed when new functions are added.

Copyright AT*SQA,
All Rights Reserved Test Automation Micro-Credential Syllabus 9
Keyword-driven test automation is particularly
well suited for early automation development with
Building Continuously
Agile and similar SDLCs where automation is built An automation project is generally considered
while the software is still evolving. New keywords complete only when the software under test is
can be added as new functionality is developed, no longer changing and no changes to existing
allowing automation to start early without creating tests are needed. Until that time, the automation
a large maintenance effort later (i.e., accruing must be continuously monitored, maintained and
technical debt). augmented to maintain and increase coverage
of the software under test. Test automation that
Implementing the Framework is not updated will result in declining coverage
over the life of the software under test, increasing
When a framework is created, standards are
the chances of regressions escaping unnoticed.
established for the test automation development
It is important to understand the on-going
project. Naming conventions are defined and
maintenance costs of a test automation suite to
reusable functions are created to form the basis
factor into the budget.
of the library. Elements from the library can be
re-used in various scripts, reducing the need
for development and lowering the maintenance
requirements by having common shared code. A
good framework is essential for creating efficient
test automation that will be maintainable across
a set of automation engineers. A framework, once
established, also allows automation engineers
to work on adding more to the function library
as time allows, making the framework a living
structure.

Copyright AT*SQA,
All Rights Reserved Test Automation Micro-Credential Syllabus 10
Benefits of Automated Testing
There are a number of benefits to automating • Tests that are complex and difficult to
testing. The primary benefits include the following: execute manually can be good candidates
for automation, to reduce the burden on the
• Automation can execute more tests in a shorter manual test effort
period of time, which in turns helps to increase
test coverage • Using data-driven or keyword-driven
techniques allows more tests to be generated
• Scripted tests will always run the same steps by adding more actions/data with no scripting
in the same order, providing greater reliability, changes
repeatability, and improved consistency
• The same tests can be run against various
• Reusability of automated tests facilitates future hardware and software configurations resulting
regression testing in compatibility tests being automated
• Faster release cycles are possible with lower • More time is available for testers to explore new
regression risk areas of the software that may have previously
been untested, resulting in higher quality
software

Copyright AT*SQA,
All Rights Reserved Test Automation Micro-Credential Syllabus 11
• By improving the frequency of testing A well-implemented test automation program will
(particularly regression testing) the continuous result in overall improvements to the efficiency
testing required for DevOps initiatives is of the testing, which in turn results in lower
possible test execution costs. Test automation allows an
organization to move to a faster release cycle with
• By testing earlier with the use of automated
higher quality, allowing better responsiveness to
tests, defect detection and remediation is less
market needs.
expensive
• Tests can be executed at times when people do
not need to be using the system

Copyright AT*SQA,
All Rights Reserved Test Automation Micro-Credential Syllabus 12
Test Automation Risks tests (such as specifications). It is common for
automated tests to become more confirmatory
in nature as opposed to discovering new
defects.
There are risks with a test automation effort. These
• Proper tool selection is critically important.
risks include the following:
Selecting the wrong tool due to an inadequate
• Management may perceive there is less need evaluation process can create extra effort and
for manual testers when automation is in place. may render the implementation impossible.
In reality, manual testers’ roles are expanded
when test automation is introduced. • Tool support must be reliable. Open source
tools may go dormant if there is no strong
• Automated testing is not automatic testing. community support. A commercial tool’s vendor
Test automation systems can be brittle if not may go out of business or change directions.
designed and constructed properly. This is a Tools may experience unexpected changes
problem that is frequently seen with scripts that requiring unplanned changes in the testware.
have been recorded from execution rather than
having been designed for maintainability and • Automation will not solve all testing problems.
reusability. Recorded scripts can be a basis for If a good testing process is not already in place,
script development, but the recording must be the automation effort may just speed up chaos.
converted to a reusable, well-structured and • Accurate reporting can be difficult. Failures may
maintainable script. cascade causing the numbers to inaccurately
• Test automation does not necessarily improve reflect the quality of the software. Defects must
the effectiveness of testing in terms of defects still be analyzed and documented by a person.
found, as the quality of the automated tests • Poor maintainability will be expensive
depend on the quality of the test cases (such as Potentially, very expensive.
the correct test conditions) and the basis for the

Copyright AT*SQA,
All Rights Reserved Test Automation Micro-Credential Syllabus 13
Test Automation Success Factors
In addition to the characteristics of good
automation targets, there are also some common
Build Automatability into the
success factors that will help to ensure the System
automation effort achieves the defined goals.
Building a maintainable and reusable test
automation suite starts with ensuring that the
Find the Right Project design of the system to be tested supports and
facilitates test automation. One common issue
The first step in a test automation project is
in test automation efforts is trying to automate
identifying the appropriate software system
software which is inherently difficult to automate.
candidate. Systems that are near their sunset
This could be because it contains inconsistently
years and will soon be retired are generally not
named or mis-named objects. For object-based
considered good candidates for automation as
test automation (which is the standard and
the return on investment will be short-lived. An
preferred approach in test automation), object
exception to this might be starting test automation
identification is essential for creating maintainable
development on a retiring system to capture data
test automation. Limited test access to APIs, a lack
and tests that will be valid in the replacement
of observability during testing, insufficient logging,
system, albeit with adjustments for the new
etc. can also significantly complicate the test
interface.
automation effort, leading to more time and money
spent on maintenance throughout the life of the
automation. Early involvement by the automation
engineer during the design of the system to be
tested can help to ensure that the necessary
support for the automation is built into the system.

Copyright AT*SQA,
All Rights Reserved Test Automation Micro-Credential Syllabus 14
Show Early Success • Regression tests – regression tests are
generally stable and well-defined. Automation
A large automation project can take years for these tests is efficient because they will be
to complete. It is important to create interim used many times over the life of the product
milestones and demonstrate that the milestones and that automation will save significant
are being met. In general, there are three areas manual test execution time. Automated
of automation focus. In order of priority (and to regression tests allow a team to release
provide the most visible return on investment software safely, more frequently, and free
quickly), implementation should proceed as testing time for more important areas.
follows:
• Functionality tests – in general, test automation
• Create acceptance/verification tests for the is faster when the software being tested is
software build – these are positive path tests stable. This results in fewer changes to the
that are used to verify that the code added automation because the code being tested
to the build did not “break” the build. These is not changing. That said, functional testing
tests are run every time code is committed and still needs to be automated, particularly in the
provide fast feedback to the developers if any rapid SDLCs, such as Agile. In this case, the
issues have been introduced. In a frequent build code may not be stable as new features are
environment, such as Agile, or in a continuous still being added and evolved. Maintainability in
integration model, this automated testing is design is critical for the test automation effort
critical to the success of the process. to be able to make forward progress and not be
consumed by maintenance issues.

Copyright AT*SQA,
All Rights Reserved Test Automation Micro-Credential Syllabus 15
Review the Plan Define Ownership
Original estimations for an automation effort are Automation requires upkeep and analysis to
sometimes wrong. This may be due to technical ensure it is working properly. Identifying a resource
issues, frequently changing SUT, slow ramp-up that can be called upon to detect and correct
of the testing team or other reasons. Reviewing issues is critical to continued, uninterrupted use of
the plan and validating the schedule should occur automation in testing. Likewise, it is important to
on a periodic basis to understand changes to the identify who will run the test automation and when
schedules and identified risks. It is also important it will be run. Often the manual testers assume
to continually set realistic milestones and report the responsibility to execute the automation and
on the achievement of those to the management do the preliminary debugging if an issue is found
sponsors of the effort. (by manually testing any failures that occur).
This helps the manual tester to engage with
the automation and also frees the automation
engineers from debugging issues that may be due
to data or environment changes.

Copyright AT*SQA,
All Rights Reserved Test Automation Micro-Credential Syllabus 16
Test Automation Tools
Early tools depended on recognizing characters adapt to the executing software, providing greater
that a user entered onto the screen or via a coverage and less up-front programming from the
command line. With the advent of graphical automation engineer.
systems, tools evolved to support coordinate-
based positioning and text recognition of fonts Test tools cover a range of features, functions and
and screen characters. Eventually, tools further levels of customizability. For teams with strong
evolved to recognize the native objects displayed programming skills, tools that allow development
on the windows, where multiple object attributes of purpose-built functions are appropriate. For less
could be verified and the objects themselves could technical test teams, tools that require minimal
be accessed for interaction. With the advent of programming may be more appropriate. It is
interfaces such as those used for web services, important to identify a tool that meets both the
tools additionally supported the ability to bypass needs of the SUT and the skills of the team.
the UI and call the function, service, or method
directly, often through the API.

Future uses of automation tools will include robotic

Copyright AT*SQA,
All Rights Reserved Test Automation Micro-Credential Syllabus 17
References
ISO/IEC/IEEE Standards

• ISO/IEC/IEEE 12207:2017
• ISO/IEC/IEEE 15288

Trademarks
The following registered trademarks and service marks are used in this document:

• AT*SQA® is a registered trademark of the Association for Testing and Software Quality Assurance

Books
[Anderson00]: Anderson, L.W. and Krathwohl, D.R. (2000) A Taxonomy for Learning, Teaching, and
Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives, Allyn & Bacon: Boston MA, ISBN-10:
080131903X

[Firtman]: Maximiliano Firtman, “Programming the Mobile Web”, O'Reilly Media;

Second Edition (April 8, 2013), ISBN-10: 1449334970

[PMBOK] Project Management Institute, “A Guide to the Project Management Body of Knowledge (PMBOK
Guide) – Sixth Edition, 2017, ISBN-10: 9781628251845

Copyright AT*SQA,
All Rights Reserved Test Automation Micro-Credential Syllabus 18
Other References

The following references point to information available on the Internet. Even though these references were
checked at the time of publication of this syllabus, AT*SQA cannot be held responsible if the references are
not available anymore. AT*SQA is not endorsing any of these sites or their products. The references are
provided as a source of information only.

https://techcrunch.com/2013/03/25/ip-oh-my-gosh-all-that-money-just-disappeared/ https://www.reuters.
com/article/us-facebook-settlement/facebook-settles-lawsuit-over-

2012-ipo-for-35-million-idUSKCN1GA2JR

[NASDAQ] https://www.sec.gov/news/press-release/2013-2013-95htm

National Institute of Standards and Technology. Framework for Improving Critical Infrastructure
Cybersecurity. Version 1.1. 2018. https://nvlpubs.nist.gov/nistpubs/CSWP/NIST.CSWP.04162018.pdf

National Institute of Standards and Technology. Risk Management Framework for Information Systems and
Organizations: A System Life Cycle Approach for Security and Privacy. Revision 2. 2018.
https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-37r2.pdf

[WCAG] https://www.w3.org/WAI/policies/

Copyright AT*SQA,
All Rights Reserved Test Automation Micro-Credential Syllabus 19
www.atsqa.org