EBOOK

HOLISTIC APP
TESTING STRATEGIES
One of the few constants in software development is change. Languages, technologies and
methodologies are in constant flux. No matter how fast developers run, it can seem that
they’re only going backward.

A software engineering requirement that remains constant — and the one thing developers
are truly gauged on — is the quality of the work they produce. No matter what kind of fancy
technology, algorithm or language you use, the code must ultimately work.

Thus, software testing is one of the most critical development tasks, as it ensures the
application performs as expected before release to end users. A buggy application will,
at best, cost much more than was budgeted; at worst, a bad application can ruin your
reputation and alienate customers.

Testing is all about ensuring an application fulfills its requirements and provides a great user
experience. Opinions on the best software testing approach vary widely:

» Software developers generally believe that automated unit testing holds the key to
great quality.
» QA engineers tend to prefer a combination of manual test execution and exploratory
testing, along with some automated regression tests thrown into the mix.
» Project managers generally espouse automated integration testing as the only
path to perfect software quality.
In fact, the correct answer is often a combination of these methods. Thoroughly testing an
app demands more than one approach. Manual testing (including exploratory testing and
test case execution) and automated unit and integration testing all play an important role in
a holistic test strategy.

When choosing the correct testing strategy, first consider your testing velocity. If reducing
time to market is a crucial factor in your holistic test strategy, a well-honed automation

www.applause.com Page 2 of 12
 Holistic App Testing Strategies

approach can help. Unfortunately, automation is not fool-proof; it simply cannot cover
every aspect of an app and requires ongoing adjustments to accommodate new features
or changes to existing ones. Automation can, however, save time when applied to stable,
repeatable test cases.

Another question to ask when choosing a testing strategy: are you trying to secure code
coverage or test coverage? QA teams often conflate the two. Understanding how to look
at these two areas separately will go a long way in creating the strongest holistic testing
plan possible.

Examine different testing strategies, as well as how they relate to coverage and velocity.
This insight will help you create the best strategy for your organization and applications: a
holistic software testing approach that ensures the right coverage and balance.

Code Coverage vs. Test Coverage

Code coverage and test coverage are frequently confused. Understand the differences and
where your organization stands, then adjust to establish the optimal holistic test strategy.
Let’s compare the definition, goal and subtypes of each form of coverage.

Code coverage Test coverage

Definition
The percentage of code exercised while the How much the overall test plan ensures
application is running. each application feature and behavior
performs as expected.

Goal
Raise the percentage of code tested Test each feature and behavior, including
during execution to uncover as many bugs edge cases, to learn how well the written
as possible. Code coverage metrics help code adheres to application requirements.
monitor automated tests.

Subtypes
Statement coverage, condition coverage, None.
branch coverage, toggle coverage and
finite-state machine coverage.

www.applause.com Page 3 of 12
 Holistic App Testing Strategies

Code coverage is the percentage of code complex business logic or more integrated
paths covered by testing. Most often, systems a task involves, the harder it is to
code coverage uses automated testing in automate — sometimes it just cannot be
the form of developer unit tests. Common done. Automating simple, repetitive tests
metrics in coverage reports include: makes the most sense.

» Functional coverage - the The percentage of automated test

number of functions called. coverage ranges widely by organization,
» Statement coverage - the number of typically somewhere between 20% to 80%.
statements executed within the functions. Test coverage percentage usually depends
on the following factors:
» Branches coverage - the number of
logical branches executed in the code. » maturity as an organization;
» Condition coverage - the number » ability to build stable and robust test
of boolean sub-expressions automation, data and environments;
tested for a true or false value. » unique elements of your product
» Line coverage - the number of that may be difficult to automate;
source code lines tested. » product maturity.
The ideal metric for code coverage is 100%
but that may not be possible, especially in
Types of Testing
the beginning. Aim to cover at least 80%
of your code paths with automated unit Whether you want to pursue a code
testing. Going beyond 80% might not be coverage or test coverage strategy, you
cost- or time-efficient, and could require can use a few common approaches.
more maintenance than it’s worth.
In the first approach, manual test case
Unlike code coverage, which tracks the execution, testers ensure the application
percentage of the application’s code adheres to specifications and look for
executed, test coverage tracks the defects. Testers put themselves in the
percentage of use cases or features role of the end user and work through
tested, ideally via automation. Typically, the application by following a test script.
this is focused on user interface (UI) and The goal is for the tester to exercise all
user experience (UX) testing: how the application features and make sure they
application appears and how users interact work as expected. These testers expose
with it. errors by exercising edge cases that the
developer might not have considered.
Some use cases cannot be automated. For
example, test cases that change frequently In manual exploratory testing, a tester
or scenarios like multi-factor authentication, does not follow a specific test script, but
which involve multiple devices and may does have a specific focus. The goal is to
include time delays, don’t lend themselves provide information on how to improve the
to automation. In most cases, the more application functionality, rather than just

www.applause.com Page 4 of 12
 Holistic App Testing Strategies

seeing if the application meets a rigid set of specifications. For example, an organization
might add a new feature on a checkout page and deploy a team of testers to go through a
checkout cycle to confirm everything works.

Automated testing, another common approach, uses the power of machines to execute
a test script, then compares the results to expected output. After all automated tests are
complete, the results are compiled into a report which shows whether each test passes or
fails, and describes the failure reason.

User acceptance testing (UAT) is a testing approach that relies on application users
and other authorized parties. These users confirm whether software provides the
anticipated results and follows the expected business procedures or specifications. Even
if the application passes manual and automated tests, it might not meet user needs and
expectations. In many cases, UAT executes in a production-like environment, which exposes
environmental issues other tests miss.

Another QA approach is non-functional testing, which tests features like security, usability
and accessibility. The organization’s needs dictate which features call for non-functional
tests and the frequency of these tests. For example, you should test accessibility regularly,
but not necessarily with every release. However, in financial services, frequent security
testing is essential.

Refer to the image below to determine the types of testing best suited for manual or
automated assessment:

Continuous functional testing across the SDLC

User
Functional Unit Integration Smoke System Regression Exploratory Acceptance Production
Test Types Testing Testing Testing Testing Testing Testing Testing Testing

Continuous Manual Testing

Delivery Automated Testing
Pipeline

Code Test Release

Automated testing is critical at the beginning of the SDLC, while

manual testing becomes more important as you get closer to release.

www.applause.com Page 5 of 12
 Holistic App Testing Strategies

How Do We Find the Best Testing Mix?

As previously established, no one-size-fits-all testing strategy exists. So, how can you blend
different strategies into one cohesive approach?

Let’s explore how to best combine testing approaches. First, let’s review how each testing
type is used, with examples:

Test method Usage Example

Manual Testers follow a script to confirm If a streaming service adds closed
structured whether one or more features captioning, a test case would assure
testing work as specified. the captions work on live stream
and video on demand (VOD). Manual
testing can also confirm that the
closed captioning still works when
returning from commercial and when
switching from one live-streamed
program to another. These scenarios
have been anticipated and need
validation.
Manual The tester goes beyond a written Using the closed captioning example,
exploratory test case and interacts with the the manual test case might not
testing app like a real-world user. cover what happens when the user
switches profiles. Exploratory testing
helps find issues you can’t anticipate.
Test scripts run to evaluate Automated testing verifies that when
an application in areas where the same video plays, the closed
Automated the results are predictable. caption text always appears on time,
testing Automated testing can drastically with the correct text displayed.
improve regression test efficiency
and speed.
Application end users assess UAT would certify the closed caption
User whether the application font is the needed style and size, and
acceptance design and performance meet does not interfere too much with the
testing expectations. video.

A tester or a tool verifies proper A test ensures that only authorized

security measures are in place users can view subscribed videos
Non-
functional and that the application’s look and that they play effectively even
testing and feel is acceptable for the during spikes in demand.
general user group.

www.applause.com Page 6 of 12
What Is the Right Test Automation Percentage?
Test case automation has many benefits. With automation, you can execute many test
cases using minimal manpower with repeatable results, free from human error — as long
as the scripts are accurate and maintained. Test automation has its drawbacks, however:
writing and maintaining automated test scripts can be time-consuming, especially when the
automation involves integration with other applications.

While each project is different, automated testing must balance defect removal with
development efficiency. If automation requires more work to build, update and maintain than
manual efforts, test automation is not the best approach. Set a goal, such as automating 30%
to 50% of the total project effort.

Consider the example of automated regression testing. As a practice, regression testing

guarantees that new software versions do not introduce logic errors. When new features are
added, it is important to include them in an automated regression suite to ensure they don’t
get compromised by software updates. Unfortunately, regression testing does not always
translate with all areas of an application, such as validating UI changes. It is difficult for an
automated test to determine whether parts of the UI no longer look correct after an update.

A developer can and should take advantage of automated testing while coding an
application or feature. Automated unit testing checks each unit of code to assure
correctness. With unit test automation, the developer can earn a high degree of confidence
that the code works as expected without the need to run the entire application. Unit tests
also enable other programmers to corroborate that the functionality remains intact. Run
automated unit tests during the build process for confidence that the entire unit test suite
will pass before it is released to a QA or production environment.

www.applause.com Page 7 of 12
 Holistic App Testing Strategies

Where Should — and and energy spent testing something that

rarely changes.
Shouldn’t — You Automate?
The best time to automate regression tests
Test automation is a powerful tool, but
is when all the minimum viable product
can require great effort up front. Plan your
(MVP) features are built and validated.
testing strategies and approach to realize
Once the features are approved, automated
the best results from that effort.
regression test creation can begin
Manual testing is most efficient when
there are limited variations and branches When Do You Need Room
to test. As the scope expands, manual
testing becomes cumbersome to complete
for Exploratory Testing?
in a timely manner. One way to overcome While an application might meet design
this challenge is to implement globally specifications and return expected results
distributed manual testing teams. These in both manual and automated tests, it may
testing teams simultaneously assess the still have critical bugs. Gaps in written test
application on a wide variety of devices cases and unanticipated use cases mean
using different languages and locale your app may not perform effectively in the
settings. wild.
Automated testing gains efficiency and This is where exploratory testing comes
becomes cost effective with more use. If into play. Exploratory testing uses human
there are 100 features to test, and they ingenuity to evaluate how the application
all have the same features and expected functions in the real world. The exploratory
results, it makes sense to automate. Test tester looks for bugs, edge cases and
automation requires a fraction of the time it failure scenarios that the development team
would take manual testers to run the same failed to consider.
tests and gather results, without the risk of
human error. To help determine how much time to allot
for exploratory testing, think about the
Automated tests are especially useful following:
in regression testing. Imagine you’re
building an app containing a form with » How much exploratory testing can fit
10 fields. While half of the fields change into the release schedule? Consider
on a regular basis, the other half remain the full cycle, from finding the issue
the same. You might want to automate to fixing the issue and retesting.
testing the five fields that stay the same » When does regression testing fit in?
and have a regression suite that validates Exploratory testing might logically fit
that the application continues to work after between the regression testing and
changes. Test automation scripts to test release, which can leave as much
these five fields save the amount of time as a few days for testers to dig in.

www.applause.com Page 8 of 12
 Holistic App Testing Strategies

The chart below shows how the combination of an organization’s maturity, release velocity
and product value impact the percentage of exploratory, manual and automated test cases.
As the organization and product change, so will the mix of test methodologies that provide
the greatest value.

THE FUNCTIONAL TESTING MATURITY MODEL

Exploratory
Scope of Testing

Manual Test Case Execution

Automation

REGRESSIVE REPEATABLE CONSISTENT QUANTITATIVE OPTIMIZING

Maturity, Release Velocity, Value

The first product release should have a high percentage of exploratory testing, which
ensures all features and edge cases get thoroughly vetted. The first release will likely also
have a high percentage of manual test case execution and little to no automation.

As a product matures, most organizations automate more test cases, and the percentage
of manual and exploratory tests decreases. That said, you should maintain a small set of
manual test case executions for portions of the application where automation is too costly or
cumbersome. Likewise, keep a small percentage of time for exploratory testing, because you
cannot predict how everything will work in a real-world environment.

Applause’s exploratory testing teams, for example, supplement development organizations.

These trained testing teams, sourced from Applause’s global community, know how to
explore apps and websites to uncover significant, hard-to-find bugs that test cases miss.
Some organizations provide guidance on specific areas to explore, which narrows the team’s
focus while still giving testers freedom to explore unscripted paths. When a team of a dozen
testers executes scripted test cases and conducts guided exploratory testing without finding
issues, that provides a level of confidence that the app or website is ready for the real world.

www.applause.com Page 9 of 12
 Holistic App Testing Strategies

How Do You Choose and associated staged output takes 30

minutes, this translates to 6 working days
Between Performing a to update the automated test suite. Many
Manual Test or Writing an development teams neglect budgeting for
Automated Script? test maintenance costs.

Deciding which option is better, an Automated tests are useful for regression
automated or manual test, greatly depends testing. Consider automating test cases
on the test complexity. Consider how much where part of your application will remain
time and effort it will take to automate constant and any change that occurs
the test versus executing manually. Also, should be considered an error. The more
think about the likelihood that the test will stable automated regression tests you
change over time. Tests that are unlikely have, the more confident you can feel that
to change, such as those that don’t require changes did not break existing functionality
feedback from users, are better candidates — and the quicker you can get to market.
for automation.
So what is the right mix of the various
Use caution when automating tests that are test methods? Choose a blend of QA
apt to change over time, as automated tests strategies where test types complement
are coded scripts that you must maintain. each other based on the given application.
An automated test framework typically Automate as much as you can to quickly
reads a script coded in the language see if a code change breaks existing
recognized by the test harness. Not only functionality, expedite time to market and
does the script need to be coded, but the reduce costs by running more tests faster.
output of the test must be provided for Apply test automation for repeatable
the testing engine to validate results using application features that rarely change.
known outcomes. The larger the test suite, When automation does not fit, implement
the more scripts and staged results to manual tests with very well-defined test
maintain. cases. Make sure to include a combination
of exploratory and user acceptance tests to
For example, let’s say an automated test
stress new features and ensure they meet
suite has 100 tests for a given application.
user requirements. Each test type has its
The application writes to three tables in
own purpose — and each one, when used
a database, and the values for all three
correctly, complements other types of
tables need verification. This translates
testing.
into 100 test scripts and 300 staged pieces
of output to verify — 3 tables multiplied
by 100 tests. If the application changes
how it executes, it also changes the data
written to the database tables. That means
the entire automated test suite would
need to be updated. If changing each test

www.applause.com Page 10 of 12
 Holistic App Testing Strategies

Key Takeaways
A holistic testing strategy incorporates both code coverage and test coverage, as well as
multiple testing approaches, including manual, automated, exploratory, user acceptance and
non-functional. Each type of test fulfills a different role in assuring product quality. Combine
code coverage with test coverage to gain confidence that you have exercised a good
percentage of the code and thoroughly vetted application features. Combine automation
with manual tests to guarantee that the application was sufficiently regression tested and to
ensure new features meet your end users’ needs and expectations.

The types of tests you should prioritize can vary dramatically between different
organizations and products. Put some deep thought into the business, application and users
to find your ideal balance, understanding that your test strategy will evolve over time. For
more information about holistic testing, explore the Applause website.
About Applause
Applause is the worldwide leader in crowdtesting and digital quality. Software is at the heart
of how all brands engage users, and digital experiences must work flawlessly everywhere.
With highly vetted testers available on-demand around the globe, Applause provides brands
with a full suite of testing and feedback capabilities. This approach drastically improves
testing coverage, eliminates the limitations of offshoring and traditional QA labs, and speeds
time-to-market for websites, mobile apps, IoT, and in-store experiences.

Thousands of leading companies — including Ford, Fox, Google, and Dow Jones — rely on
Applause as a best practice to deliver high-quality digital experiences that customers love.

Learn more at www.applause.com

NORTH AMERICA EUROPE ISRAEL

100 Pennsylvania Avenue Obentrautstr. 72 10 HaMenofim Street
Framingham, MA 01701 10963 Berlin, Germany Herzliya, Israel 4672561
1.844.300.2777 +49.30.57700400 +972.74.757.1300