Software Testing Strategies

Chapter 13
Software Testing Strategies
Software is tested to uncover errors
A strategy for software testing is developed by the project manager,
software engineers, and testing specialists

A Strategic Approach to Software Testing

Testing is a set of activities that can be planned in advance and conducted
systematically.
For this reason a template for software testing (a set of steps into which
you can place specific test case design techniques and testing methods)
should be defined for the software process.

Generic Characteristics
To perform effective testing, you should conduct effective technical
reviews. By doing this, many errors will be eliminated before testing
commences.
Testing begins at the component level and works “outward” toward the
integration of the entire computer-based system.
Different testing techniques are appropriate for different software
engineering approaches and at different points in time.
Testing is conducted by the developer of the software and (for large
projects) an independent test group.
 Testing and debugging are different activities, but debugging must be
accommodated in any testing strategy.

Verification and Validation

Verification refers to the set of tasks that ensure that software correctly
implements a specific function.
Validation refers to a different set of tasks that ensure that the software
that has been built is traceable to customer requirements.
Acc. to Boehm,
Verification: “Are we building the product right?”
Validation: “Are we building the right product?”
Verification and validation includes a wide array of SQA activities:
technical reviews, quality and configuration audits, performance
monitoring, simulation, feasibility study, …
Testing Strategy for Conventional S/w Architectures
A strategy for software testing may also be viewed in the context of the spiral
Unit testing begins at the vortex of the spiral and concentrates on each unit (e.g.,
component, class, or WebApp content object) of the software as implemented in
source code

Testing progresses by moving outward along the spiral to integration testing,

where the focus is on design and the construction of the software architecture.
Then Validation - requirements established as part of requirements modeling are
validated against the software that has been constructed
finally System Testing - the software and other system elements are tested as a
whole.
Considering the process from a procedural point of view, testing within the context
of software engineering is actually a series of four steps that are implemented
sequentially.
Initially, tests focus on each component individually, ensuring that it functions
properly as a unit. Hence, the name unit testing.
Unit testing makes heavy use of testing techniques to ensure complete coverage and
maximum error detection.
Next, components must be assembled or integrated to form the complete software
package.

Integration testing addresses the issues associated with the dual problems of
verification and program construction.
Test case design techniques that focus on inputs and outputs are more prevalent
during integration, although techniques that exercise specific program paths may be
used to ensure coverage of major control paths.
Software, once validated, must be combined with other system elements (e.g.,
hardware, people, databases).
System testing verifies that all elements mesh properly and that overall system
function/performance is achieved.

Test Strategies for Conventional Software

Unit testing focuses verification effort on the smallest unit of software design—the
software component or module.
The unit test focuses on the internal processing logic and data structures within
the boundaries of a component .
This type of testing can be conducted in parallel for multiple components.
 Unit tests are illustrated schematically in the following Figure.
Unit-test considerations
The module interface is tested to ensure that data/information properly flows into
and out of the program unit under test.
Local data structures are examined to ensure that data stored temporarily
maintains its integrity during all steps in an algorithm’s execution.
All independent paths through the control structure are exercised to ensure that all
statements in a module have been executed at least once.
 Selective testing of execution paths is an essential task during the unit test.
Test cases should be designed to uncover errors due to erroneous computations,
incorrect comparisons, or improper control flow.
Boundary conditions are tested to ensure that the module operates properly at
boundaries established to limit or restrict processing.
Software often fails at its boundaries - errors often occur when the nth element of
an n-dimensional array is processed, when the ith repetition of a loop with i passes
is invoked, when the maximum or minimum allowable value is encountered
And finally, all error-handling paths are tested.
Unit-test Procedures
Unit testing is normally considered as an adjunct to the coding step.
The design of unit tests can occur before coding begins or after source code has been
generated
Because a component is not a stand-alone program, driver and/or stub software
must often be developed for each unit test.
A driver is nothing more than a “main program” that accepts test case data, passes
such data to the component (to be tested), and prints relevant results.
Stubs serve to replace modules that are subordinate (invoked by) the component to
be tested.
A stub or “dummy subprogram” uses the subordinate module’s interface, may do
minimal data manipulation, prints verification of entry, and returns control to the
module undergoing testing.
Drivers and stubs represent testing overhead procedures

Integration Testing
A systematic technique for constructing the software architecture while at the same
time conducting tests to uncover errors associated with interfacing.
The objective is to take unit-tested components and build a program structure that has
been dictated by design.
There is often a tendency to attempt non incremental integration to construct the
program using a “big bang” approach
All components are combined in advance. The entire program is tested as a whole.
And chaos usually results!
A set of errors is encountered.
Correction is difficult because isolation of causes is complicated by the vast expanse
of the entire program.
Once these errors are corrected, new ones appear and the process continues in a
seemingly endless loop.
 Incremental integration is the antithesis of the big bang approach.
The program is constructed and tested in small increments, where errors are
• easier to isolate and correct
• interfaces are more likely to be tested completely
• and a systematic test approach may be applied

Top-down integration
It is an incremental approach to construction of the software architecture.
Modules are integrated by moving downward through the control hierarchy,
beginning with the main control module
Modules subordinate (and ultimately subordinate) to the main control module are
incorporated into the structure in either a depth-first or breadth-first manner.
In depth-first integration, the components M1, M2 , M5 would be integrated first.
Next, M8 or (if necessary for proper functioning of M2) M6 would be integrated.
Then, the central and right-hand control paths are built.
Breadth-first integration incorporates all components directly subordinate at each
level, moving across the structure horizontally.
From the figure, components M2, M3, and M4 would be integrated first and then
M5, M6, and so on, follows.
What are the steps for top-down integration?
1. The main control module is used as a test driver and stubs are substituted for
all components directly subordinate to the main control module.
2. Depending on the integration approach selected (i.e., depth or breadth first),
subordinate stubs are replaced one at a time with actual components.
3. Tests are conducted as each component is integrated.
4. On completion of each set of tests, another stub is replaced with the real
component.
5. Regression testing (discussed later in this section) may be conducted to ensure
that new errors have not been introduced.
The process continues from step 2 until the entire program structure is built.

Bottom-up integration
Bottom-up integration testing, as its name implies, begins construction and
testing with atomic modules (i.e., components at the lowest levels in the program
structure).
Because components are integrated from the bottom up, the functionality
provided by components subordinate to a given level is always available and the
need for stubs is eliminated.
What are the steps for Bottom-Up integration?
1. Low-level components are combined into clusters (sometimes called builds)
that perform a specific software subfunction.
2. A driver (a control program for testing) is written to coordinate test case input
and output.
3. The cluster is tested.
4. Drivers are removed and clusters are combined moving upward in the
program structure.
Components are combined to form clusters 1, 2, and 3. Each of the
clusters is tested using a driver (shown as a dashed block).
Components in clusters 1 and 2 are subordinate to Ma. Drivers D1 and D2
are removed and the clusters are interfaced directly to Ma.
Similarly, driver D3 for cluster 3 is removed prior to integration with
module Mb. Both Ma and Mb will ultimately be integrated with component
Mc, and so forth.
As integration moves upward, the need for separate test drivers lessens. In
fact,
if the top two levels of program structure are integrated top down, the
number of drivers can be reduced substantially and integration of clusters is
greatly simplified.

Regression Testing
regression testing is the reexecution of some subset of tests that have
already been conducted to ensure that changes have not propagated
unintended side effects.
Regression testing may be conducted manually, by reexecuting a subset of
all test cases or using automated capture/playback tools.
Capture/playback tools enable the software engineer to capture test cases
and results for subsequent playback and comparison.
The regression test suite (the subset of tests to be executed) contains three different
classes of test cases
• A representative sample of tests that will exercise all software functions.
• Additional tests that focus on software functions that are likely to be affected by
the change.
• Tests that focus on the software components that have been changed.

Smoke testing
Smoke testing is an integration testing approach that is commonly used when
product software is developed.
It is designed for time-critical projects, allowing the software team to assess the
project on a frequent basis.
In essence, the smoke-testing approach encompasses the following activities:
1.Software components that have been translated into code are integrated into a
build.
2. A series of tests is designed to expose errors that will keep the build from
properly performing its function.
3. The build is integrated with other builds, and the entire product (in its current
form) is smoke tested daily.
Benefits
Integration risk is minimized
The quality of the end product is improved. Because the approach is
construction (integration) oriented, smoke testing is likely to uncover
functional errors as well as architectural and component-level design errors. If
these errors are corrected early, better product quality will result
Error diagnosis and correction are simplified
Progress is easier to assess

In general, the advantages of one strategy tend to result in disadvantages for
the other strategy.
The major disadvantage of the top-down approach is the need for stubs and
the attendant testing difficulties that can be associated with them.
Problems associated with stubs may be offset by the advantage of testing
major control functions early.
The major disadvantage of bottom-up integration is that “the program as an
entity does not exist until the last module is added”
This drawback is tempered by easier test case design and a lack of stubs.
Selection of an integration strategy depends upon software characteristics
and sometimes, project schedule.
In general, a combined approach (sometimes called sandwich testing)
that uses top-down tests for upper levels of the program structure, coupled
with bottom-up tests for subordinate levels may be the best compromise

Validation testing
- succeeds when software functions in a manner that can be reasonably
expected by the customer
Configuration review
The intent of the review is to ensure that all elements of the software
configuration have been properly developed, are cataloged, and have the
necessary detail to strengthen the support activities
Alpha test
The alpha test is conducted at the developer’s site by a representative
group of end users.
Beta test
The beta test is conducted at one or more end-user sites. Unlike alpha
testing, the developer generally is not present. Therefore, the beta test is a
“live” application of the software in an environment that cannot be
controlled by the developer
A variation on beta testing, called customer acceptance testing, is sometimes
performed when custom software is delivered to a customer under contract.

System testing
Software is incorporated with other system elements (e.g., hardware, people,
information), and a series of system integration and validation tests are
conducted.
A classic system-testing problem is “finger pointing.” This occurs when an
error is uncovered, and the developers of different system elements blame
each other for the problem
System testing is actually a series of different tests whose primary purpose is
to fully exercise the computer-based system

Recovery testing
It is a system test that forces the software to fail in a variety of ways and
verifies that recovery is properly performed.
If recovery is automatic (performed by the system itself), reinitialization,
checkpointing mechanisms, data recovery, and restart are evaluated for
correctness.
If recovery requires human intervention, the mean-time-to-repair (MTTR) is
evaluated to determine whether it is within acceptable limits.
Security testing
It attempts to verify that protection mechanisms built into a system will, in
fact, protect it from improper penetration
During security testing, the tester plays the role(s) of the individual who
desires to penetrate the system.
Stress testing
It executes a system in a manner that demands resources in abnormal
quantity, frequency, or volume
A variation of stress testing is a technique called sensitivity testing.
Sensitivity testing attempts to uncover data combinations within valid input
classes that may cause instability or improper processing.
Performance testing
 It is designed to test the run-time performance of software within the context
of an integrated system.
Performance testing occurs throughout all steps in the testing process.
Deployment testing
It is sometimes called configuration testing, exercises the software in each
environment in which it is to operate.
It examines all installation procedures and specialized installation software
(e.g., “installers”) that will be used by customers, and all documentation that
will be used to introduce the software to end users.
The Art of Debugging
Debugging occurs as a consequence of successful testing.
When a test case uncovers an error, debugging is the process that results in
the removal of the error.
Although debugging can and should be an orderly process, it is still very
much an art.
As a software engineer, you are often confronted with a “symptomatic”
indication of a software problem as you evaluate the results of a test.
That is, the external manifestation of the error and its internal cause may
have no obvious relationship to one another.
The poorly understood mental process that connects a symptom to a cause
is debugging.
the debugging process begins with the execution of a test case.
Results are assessed and a lack of correspondence between expected and
actual performance is encountered.
In many cases, the non-corresponding data are a symptom of an underlying
cause as yet hidden.
The debugging process attempts to match symptom with cause, thereby
leading to error correction.
The debugging process will usually have one of two outcomes:
(1)the cause will be found and corrected or
(2)the cause will not be found. In the latter case, the person performing
debugging may suspect a cause, design a test case to help validate
that suspicion, and work toward error correction in an iterative fashion.
Why is debugging so difficult?
In all likelihood, human psychology has more to do with an answer than
software technology.
However, a few characteristics of bugs provide some clues:
1. The symptom and the cause may be geographically remote.
2. The symptom may disappear (temporarily) when another error is corrected.
3. The symptom may actually be caused by nonerrors (e.g., round-off
inaccuracies).
4. The symptom may be caused by human error that is not easily traced
5. The symptom may be a result of timing problems, rather than processing
problems.
6. It may be difficult to accurately reproduce input conditions (e.g., a real-
time application in which input ordering is indeterminate).
7. The symptom may be intermittent. This is particularly common in
embedded systems that couple hardware and software inextricably.
8. The symptom may be due to causes that are distributed across a number of
tasks running on different processors.
During debugging, you’ll encounter errors that range from mildly annoying
(e.g., an incorrect output format) to catastrophic (e.g., the system fails,
causing serious economic or physical damage).
As the consequences of an error increase, the amount of pressure to find the
cause also increases.
Often, pressure forces some software developers to fix one error and at the
same time introduce two more.
Psychological Considerations
Unfortunately, there appears to be some evidence that debugging process is
an innate human trait.
Some people are good at it and others aren’t. Although experimental
evidence on debugging is open to many interpretations, large variances in
debugging ability have been reported for programmers with the same
education and experience.
Acc. to Shneiderman , “ Debugging is one of the more frustrating parts of
programming. It has elements of problem solving or brain teasers, coupled
with the annoying recognition that you have made a mistake. Heightened
anxiety and the unwillingness to accept the possibility of errors increases the
task difficulty. Fortunately, there is a great sigh of relief and a lessening of
tension when the bug is ultimately . . . Corrected”.
Although it may be difficult to “learn” debugging, a number of approaches
to the problem can be proposed.

Debugging Strategies
Regardless of the approach that is taken, debugging has one
overriding objective - to find and correct the cause of a software error
or defect.
The objective is realized by a combination of systematic evaluation,
intuition, and luck.

Bradley describes the debugging approach in this way: “Debugging is a

straightforward application of the scientific method that has been
developed over 2,500 years. The basis of debugging is to locate the
problem’s source [the cause] by binary partitioning, through working
hypotheses that predict new values to be examined”

Take a simple non-software example: A lamp in my house does not work.

If nothing in the house works, the cause must be in the main circuit breaker
or outside; I look around to see whether the neighborhood is blacked out. I
plug the suspect lamp into a working socket and a working appliance into
the suspect circuit. So goes the alternation of hypothesis and test
In general, three debugging strategies have been proposed :
(1)brute force, (2) backtracking, and (3) cause elimination.

Each of these strategies can be conducted manually, but modern debugging

tools can make the process much more effective.
Brute force
common and least efficient method
Using a “let the computer find the error” philosophy, memory dumps are
taken, run-time traces are invoked, and the program is loaded with output
statements.
Although the mass of information produced may ultimately lead to success, it
more frequently leads to wasted effort and time
Backtracking
It is a fairly common debugging approach that can be used Successfully in
small programs
source code is traced backward (manually) until the cause is found
Cause Elimination
manifested by induction or deduction and introduces the concept of binary
partitioning.
 Data related to the error occurrence are organized to isolate potential causes.
 A “cause hypothesis” is devised and the abovementioned data are used to
prove or disprove the hypothesis.
Alternatively, a list of all possible causes is developed and tests are conducted
to eliminate each. If initial tests indicate that a particular cause hypothesis
shows
promise, data are refined in an attempt to isolate the bug.
Automated debugging
Each of these debugging approaches can be supplemented with debugging
tools that can provide you with semi-automated support as debugging
strategies are attempted
Integrated development environments (IDEs) provide a way to capture some
of the language specific predetermined errors (e.g., missing end-of-statement
characters, undefined variables, and so on) without requiring compilation.”
A wide variety of debugging compilers, dynamic debugging aids (“tracers”),
automatic test-case generators, and cross-reference mapping tools are
available.
However, tools are not a substitute for careful evaluation based on a
complete design model and clear source code.

Correcting the Error

Once a bug has been found, it must be corrected. But, as we have already
noted, the correction of a bug can introduce other errors and therefore do more
harm than good.
Van Vleck [Van89] suggests three simple questions that you should ask
before making the “correction” that removes the cause of a bug:
1. Is the cause of the bug reproduced in another part of the program?
2. What “next bug” might be introduced by the fix I’m about to make?
3. What could we have done to prevent this bug in the first place?