Chapter 16

Fault based Testing

1
 Introduction
• Engineers study failures to understand how to
prevent similar failures in the future
• A model of potential program faults is a valuable
source of information for evaluating and designing
test suites.
• Some fault knowledge is commonly used in
functional and structural testing.
• Fault based testing uses fault model directly to
hypothesize (proposed explanation) potential faults in
a program under test, and to create or evaluate test
suites based on its efficiency in detecting those
hypothetical faults.
2
• Experience with common software faults
sometimes leads to improvements in design
methods and programming languages.
• For example 1, the main purpose of automatic
memory management in Java is not to spare the
programmer the trouble of releasing unused
memory, but to prevent the programmer from
making the kind of memory management errors
• (dangling pointers, redundant deallocations, and
memory leaks)
• that frequently occur in C and C++ programs.
3
• Eg2: Automatic array bounds checking cannot
prevent a programmer from using an index
expression outside array bounds, but can make
it much less likely that the fault escapes
detection in testing, as well as to limiting the
damage
• The basic concept of fault-based testing is to
select test cases that would distinguish the
program under test from alternative
programs that contain hypothetical faults.
4
• The basic concept of fault-based testing is to select test
cases that would distinguish the program under test from
alternative programs that contain hypothetical faults.

• This is usually approached by modifying the program under

test to actually produce the hypothetical faulty programs.

• Fault seeding can be used to evaluate the thoroughness of a

test suite (that is, as an element of a test adequacy criterion),
or for selecting test cases to augment a test suite, or to
estimate the number of faults in a program.

5
 Assumptions in Fault-Based Testing
• The effectiveness of fault-based testing depends
on the quality of the fault model, and on some
basic assumptions about the relation of the seeded
faults to faults that might actually be present.

• The seeded faults are small syntactic changes, like

replacing one variable reference by another in an
expression, or changing a comparison from < to <=.
6
• if the program under test has an actual fault, we
may hypothesize that it differs from another,
corrected program by only a small textual change

• If so, then we need merely distinguish the

program from all such small variants (by selecting
test cases for original or the variant program fails)
to ensure detection of all such faults.
• This is known as the competent programmer
hypothesis, an assumption that the program under
test is “close to” (in the sense of textual
difference) a correct program.
7
• Some program faults are indeed simple
typographical errors, and others that involve
deeper errors of logic may be manifest in simple
textual differences

• Sometimes an error of logic will result in much

more complex differences in program text.

• This may not invalidate fault-based testing

with a simpler fault model, provided test cases
sufficient for detecting the simpler faults are
sufficient also for detecting the more complex
fault. This is known as the coupling effect.

8
9
• Fault-based testing can guarantee fault detection
only if the competent programmer hypothesis and
the coupling effect hypothesis hold.

• This also implies that developing better fault models,

based on hard data about real faults rather than
guesses, is a good investment of effort.

10
 Mutation analysis
• Mutation analysis is the most common form of
software fault-based testing.
• A fault model is used to produce hypothetical
faulty programs by creating variants of the
program under test.
• Variants are created by “seeding” faults, that is,
by making a small change to the program under
test following a pattern in the fault model.
• The patterns for changing program text are
called mutation operators, and each variant
program is called a mutant.
11
12
• Mutant programs that are rejected by a
compiler, or which fail almost all tests, are not
good models of the faults we seek to uncover
with systematic testing.
• We say a mutant is valid if it is syntactically
correct.
• We say a mutant is useful if, in addition to
being valid, its behaviour differs from the
behaviour of the original program for no
more than a small subset of program test
cases.

13
14
• A mutant obtained from the program of
Figure 16.1 by substituting while for switch
in the statement at line 13 would not be
valid, since it would result in a compile-time
error.
• A mutant obtained by substituting 1000 for 0
in the statement at line 4 would be valid, but
not useful,
– since the mutant would be distinguished from the
program under test by all inputs and thus would
not give any useful information on the
effectiveness of a test suite.

15
• Since mutants must be valid, mutation
operators are syntactic patterns defined
relative to particular programming languages.
• Figure 16.2 shows some mutation operators
for the C language.
• Constraints are associated with mutation
operators to guide selection of test cases
likely to distinguish mutants from the original
program.

16
• Many of the mutants of Figure 16.2 can be applied equally
well to other procedural languages, but in general a mutation
operator that produces valid and useful mutants for a given
language may not apply to a different language or may
produce invalid or useless mutants for another language.

• For example, a mutation operator that removes the “friend”

keyword from the declaration of a C++ class would not be
applicable to Java, which does not include friend classes.

17
18
19
Fault-Based Adequacy Criteria

20
21
22
 Variations on Mutation Analysis
• The mutation analysis process described above, which kills
mutants based on the outputs produced by execution of test
cases, is known as strong mutation.
• It can generate a number of mutants quadratic in the size of
the program.
• Each mutant must be compiled and executed with each test
case until it is killed.
• The time and space required for compiling all mutants and
for executing all test cases for each mutant may be
impractical.
23
• The computational effort required for mutation analysis can
be reduced by reducing the number of mutants generated
and the number of test cases to be executed.

• Weak mutation analysis reduces the number of tests to

be executed by killing mutants when they produce a
different intermediate state, rather than waiting for a
difference in the final result or observable program
behaviour.

24
• With weak mutation, a single program can be
seeded with many faults.

• A “metamutant” program is divided into segments

containing original and mutated source code, with a
mechanism to select which segments to execute.

• Two copies of the meta-mutant are executed in

tandem (one after the other), one with only original
program code selected, and the other with a set of
live mutants selected.

25
• Execution is paused after each segment to compare
the program state of the two versions.

• If the state is equivalent, execution resumes with the

next segment of original and mutated code.

• If the state differs, the mutant is marked as dead,

and execution of original and mutated code is
restarted with a new selection of live mutants.

26
• Weak mutation testing does not decrease the
number of program mutants that must be
considered, but it decreases the number of test
executions and compilations.

• This performance benefit has a cost in accuracy: weak

mutation analysis may “kill” a mutant even if the
changed intermediate state would not have an effect
on the final output or observable behavior of the
program.

27
• Mutation analysis can be used either to judge the
thoroughness of a test suite or to guide selection of
additional test cases.

• If one is designing test cases to kill particular mutants, then

it may be important to have a complete set of mutants
generated by a set of mutation operators.

• If, on the other hand, the goal is a statistical estimate of the

extent to which a test suite distinguishes programs with
seeded faults from the original program, then only a much
smaller statistical sample of mutants is required.

• statistical sampling may keep the sample small enough to

permit careful examination of equivalent mutants

28
29
30