Scribd
Upload
17 views19 pages

ST MOD2 FaultBasedTesting2

This document discusses fault-based testing and mutation analysis. It introduces fault-based testing as using a fault model to hypothesize potential faults and create test cases to detect them. Mutation analysis is described as a common form where faults are seeded by making small changes to a program to create mutant versions, and test cases are run to kill mutants by causing them to fail. Weak mutation analysis is discussed as a way to reduce computational effort by killing mutants based on intermediate state differences rather than final output.

Uploaded by

Veenaxi Painginkar
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
17 views19 pages

ST MOD2 FaultBasedTesting2

This document discusses fault-based testing and mutation analysis. It introduces fault-based testing as using a fault model to hypothesize potential faults and create test cases to detect them. Mutation analysis is described as a common form where faults are seeded by making small changes to a program to create mutant versions, and test cases are run to kill mutants by causing them to fail. Weak mutation analysis is discussed as a way to reduce computational effort by killing mutants based on intermediate state differences rather than final output.

Uploaded by

Veenaxi Painginkar
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 19

Chapter 16

Fault based Testing

1
Introduction
• Engineers study failures to understand how to
prevent similar failures in the future
• A model of potential program faults is a valuable
source of information for evaluating and designing
test suites.
• Some fault knowledge is commonly used in
functional and structural testing.
• Fault based testing uses fault model directly to
hypothesize (proposed explanation) potential faults in
a program under test, and to create or evaluate test
suites based on its efficiency in detecting those
hypothetical faults.
2
• Experience with common software faults
sometimes leads to improvements in design
methods and programming languages.
• For example 1, the main purpose of automatic
memory management in Java is not to spare the
programmer the trouble of releasing unused
memory, but to prevent the programmer from
making the kind of memory management errors
• (dangling pointers, redundant deallocations, and
memory leaks)
• that frequently occur in C and C++ programs.

3
• The basic concept of fault-based testing is to select test
cases that would distinguish the program under test from
alternative programs that contain hypothetical faults.

• This is usually approached by modifying the program under


test to actually produce the hypothetical faulty programs.

• Fault seeding can be used to evaluate the thoroughness of a


test suite (that is, as an element of a test adequacy criterion),
or for selecting test cases to augment a test suite, or to
estimate the number of faults in a program.

4
Assumptions in Fault-Based Testing
• The effectiveness of fault-based testing depends
on the quality of the fault model, and on some basic
assumptions about the relation of the seeded faults to
faults that might actually be present.

• The seeded faults are small syntactic changes, like


replacing one variable reference by another in an
expression, or changing a comparison from < to <=.
5
• if the program under test has an actual fault, we
may hypothesize that it differs from another,
corrected program by only a small textual change

• If so, then we need merely distinguish the program


from all such small variants (by selecting test cases
for original or the variant program fails) to ensure
detection of all such faults.
• This is known as the competent programmer
hypothesis, an assumption that the program under
test is “close to” (in the sense of textual difference)
a correct program.
6
• Some program faults are indeed simple
typographical errors, and others that involve
deeper errors of logic may be manifest in simple
textual differences

• Sometimes an error of logic will result in much


more complex differences in program text.

• This may not invalidate fault-based testing with


a simpler fault model, provided test cases sufficient
for detecting the simpler faults are sufficient also
for detecting the more complex fault. This is
known as the coupling effect.

7
8
• Fault-based testing can guarantee fault detection
only if the competent programmer hypothesis and
the coupling effect hypothesis hold.

• This also implies that developing better fault models,


based on hard data about real faults rather than
guesses, is a good investment of effort.

9
Mutation analysis
• Mutation analysis is the most common form of
software fault-based testing.
• A fault model is used to produce hypothetical
faulty programs by creating variants of the
program under test.
• Variants are created by “seeding” faults, that is,
by making a small change to the program under
test following a pattern in the fault model.
• The patterns for changing program text are
called mutation operators, and each variant
program is called a mutant.

10
11
• Mutant programs that are rejected by a
compiler, or which fail almost all tests, are not
good models of the faults we seek to uncover
with systematic testing.
• We say a mutant is valid if it is syntactically
correct.
• We say a mutant is useful if, in addition to
being valid, its behaviour differs from the
behaviour of the original program for no
more than a small subset of program test
cases.

12
13
14
Variations on Mutation Analysis
• The mutation analysis process described above, which kills
mutants based on the outputs produced by execution of test
cases, is known as strong mutation.
• It can generate a number of mutants quadratic in the size of
the program.
• Each mutant must be compiled and executed with each test
case until it is killed.
• The time and space required for compiling all mutants and
for executing all test cases for each mutant may be
impractical.
15
• The computational effort required for mutation analysis can
be reduced by reducing the number of mutants generated
and the number of test cases to be executed.

• Weak mutation analysis reduces the number of tests to


be executed by killing mutants when they produce a
different intermediate state, rather than waiting for a
difference in the final result or observable program
behaviour.

16
• With weak mutation, a single program can be
seeded with many faults.

• A “metamutant” program is divided into segments


containing original and mutated source code, with a
mechanism to select which segments to execute.

• Two copies of the meta-mutant are executed in


tandem (one after the other), one with only original
program code selected, and the other with a set of
live mutants selected.

17
• Execution is paused after each segment to compare
the program state of the two versions.

• If the state is equivalent, execution resumes with the


next segment of original and mutated code.

• If the state differs, the mutant is marked as dead, and


execution of original and mutated code is restarted
with a new selection of live mutants.

18
• Weak mutation testing does not decrease the
number of program mutants that must be
considered, but it decreases the number of test
executions and compilations.

• This performance benefit has a cost in accuracy: weak


mutation analysis may “kill” a mutant even if the
changed intermediate state would not have an effect
on the final output or observable behavior of the
program.

19

You might also like