Unit 7

Fault-Based Testing

Mr.Sandeep kumar S
Assistant Professor
Dept. Of. CSE, MITE

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 1

 contents
• Overview
• Assumption in fault Based Testing
• Mutation Analysis
• Fault Based Adequacy Criteria
• Variation on Mutation Analysis

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 2

 Overview
• Study the failure how to prevent similar failure
in the future.
• Ex1: Failure of the Tacoma Narrow Bridge.
• Ex2: Airline crash.

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 3

 Assumption in fault Based Testing

 The basic concept of fault based testing is to

select test cases that would distinguish the
program under test from alternative program
that contains hypothetical faults.

 Mutation testing is based on two assumptions:

the competent programmer hypothesis and the
coupling effect.

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 4

• The competent programmer hypothesis
assumes that competent programmers tend to
write nearly "correct" programs .
• That is programs written by experienced
programmers may not be correct, but they will
differ from the corrected version by some
relatively simple faults such as off-by-one
fault.

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 5

• The coupling effect stated that a set of test
data that can uncover all simple faults in a
program is also capable of detecting more
complex faults.

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 6

• Competent programmer hypothesis:
– Programs are nearly correct
• Real faults are small variations from the correct program
• => Mutants are reasonable models of real buggy programs
• Coupling effect hypothesis:
– Tests that find simple faults also find more complex
faults
• Even if mutants are not perfect representatives of real
faults, a test suite that kills mutants is good at finding real
faults too

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 7

 Mutation Analysis
• Mutation Testing is a powerful error-based testing technique for
unit testing. It provides high test coverage and detects many
simple syntactic faults.

• A mutant is a copy of a program with a mutation

• A mutation is a syntactic change (a seeded bug)

– Example: change (i < 0) to (i <= 0)

• Run test suite on all the mutant programs

• A mutant is killed if it fails on at least one test case

• If many mutants are killed, infer that the test suite is also
effective at finding real bugs

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 8

• The goal of Mutation Testing is to assess the
quality of the test cases which should be robust
enough to fail mutant code.
• To maintain the effectiveness of test sets.
• Mutation was originally proposed in 1971 but
lost fervor due to high costs involved. Now,
again it has picked steam and is widely used for
languages such as Java and XML.

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 9

 Different types of Mutants

• Stillborn mutants: Syntactically incorrect, killed by

compiler, e.g., x = a ++ b
• Trivial mutants: Killed by almost any test case
• Equivalent mutant: Always acts in the same behavior
as the original program, e.g., x = a + b and x = a – (-b)
• None of the above are interesting from a mutation
testing perspective
• Those mutants are interesting which behave differently
than the original program, and we do not have test
cases to identify them (to cover those specific changes)

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 10

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 11
 Following are the steps to execute mutation testing:

Step 1: Faults are introduced into the source code of the program by
creating many versions called mutants. Each mutant should contain a
single fault, and the goal is to cause the mutant version to fail which
demonstrates the effectiveness of the test cases.
Step 2: Test cases are applied to the original program and also to the
mutant program. A test case should be adequate, and it is tweaked to
detect faults in a program.
Step 3: Compare the results of original and mutant program.
Step 4: If the original program and mutant programs generate the same
output, then that the mutant is killed by the test case. Hence the test
case is good enough to detect the change between the original and the
mutant program.
Step 5: If the original program and mutant program generate different
output, Mutant is kept alive. In such cases , more effective test cases
need to be created that kill all mutants.

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 12

 How to Create Mutant Programs?

• A mutation is nothing but a single syntactic

change that is made to the program statement.
Each mutant program should differ from the
original program by one mutation.

Original Program Mutant Program

If (x>y)
If(x)
Print “Hello”
Print “Hello”
Else
Else
Print “Hi”
Print “Hi”
 

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 13

 Automation of Mutation Testing:

• Mutation testing is extremely time consuming and

complicated to execute manually. To speed up the
process, it is advisable to go for automation tools.
Automation tools reduce cost of testing as well.
• List of tools available -
• Ninja Turtles- .net mutation testing tool
• Mutagenesis– PHP mutation testing framework
• Heckle– Ruby Mutation Testing Tool
• Jester– Mutation Testing Tool for Java

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 14

 Mutation Score:

• The mutation score is defined as the

percentage of killed mutants with the total
number of mutants.
• Mutation Score = (Killed Mutants / Total
number of Mutants) * 100
• The effectiveness of the test data set is
measured by the percentage of mutants killed.

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 15

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 16
 Example of a Program Mutation

1 int max(int x, int y) 1 int max(int x, int y)

2 { 2{
3 int mx = x; 3 int mx = x;
4
5
if (x > y)
mx = x;
4 if (x < y)
6 else 5 mx = x;
7 mx = y; 6 else
8 return mx; 7 mx = y;
9 } 8 return mx;
9}

Dependable Software Systems

(Mutation)
 Example of Testing By Mutation
function MAX(M<N:INTEGER)
return INTEGER is
begin
if M>N then
return M;
else
return N;
end if:
end MAX;

First test data set--M=1, N=2

•the original function returns 2
•mutants: replace”>“ operator in if statements by (>=,<,<=or=)
•executing each mutant:

Mutants Outputs Comparison

if M>=N then 2 dead
if M<N then 1 live
if M<=N then 1 live
if M=N then 2 dead
if M< >N then 1 live

•adding test data M=2, N=1 will eliminate the latter live mutant, but the former
live mutant remains live because it is equivalent to the original function. No test
data can eliminate it.
 Advantages of Mutation Testing:
• It is a powerful approach to attain high coverage of the source
program.
• This testing is capable comprehensively testing the mutant
program.
• Mutation testing brings a good level of error detection to the
software developer.
• This method uncovers ambiguities in the source code, and has the
capacity to  detect all the faults in the program.
• Customers are benefited from this testing by getting most reliable
and stable system.

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 19

 Disadvantages of Mutant testing:
• Mutation testing is extremely costly and time consuming since
there are many mutant programs that need to be generated.

• Since its time consuming, it’s fair to say that this testing cannot
be done without an automation tool.

• Each mutation will have the same number of test cases than that
of the original program. So, a large number of mutant programs
may need to be tested against the original test suite.

• As this method involves source code changes, it is not at all

applicable for black box testing.

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 20

 Estimating #Defects

•How many defects remain in our software?

•With mutation testing, we can make an

Estimate of remaining defects

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 21

 Letʼ s consider a lake. How many
fish are in that lake?

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 22

 Simple. We catch a number of fish
(say, 1000), tag them, and throw
them back again.

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 23

Let's assume over the next week, we ask fishermen to count the

number of tags. We find 300 untagged and 50 tagged fish.

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 24

we can thus estimate that there are about 6,000
remaining untagged fish in the lake.

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 25

Now letʼs assume our lake is not a
lake, but our program.

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 26

 Simple. We catch a number of fish
(say, 1000), tag them, and throw
them back again.

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 27

Our test suite finds 50 mutants, and
300 natural faults.

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 28

 we can again estimate that there are about 6,000 remaining
defects in our program. (A test suite finding only 50 out of 1,000
mutations is a real bad sign.)

(Seeded fault) (seeded fault detected)

(natural fault detected)

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 29

Let’s count marbles ... a lot of marbles
• Suppose we have a big
bowl of marbles. How
can we estimate how
many?

– I don’t want to count

every marble individually
– I have a bag of 100 other
marbles of the same size,
but a different color
Photo credit: (c) KaCey97007 – What if I mix them?
on Flickr, Creative Commons
license

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 30

 Estimating marbles
• I mix 100 black marbles
into the bowl
– Stir well ...
• I draw out 100 marbles at
random
• 20 of them are black

• How many marbles were

in the bowl to begin
with?

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 31

 Fault Based Adequacy Criteria
• Given a program and a test suite T. Mutation
analysis consists of the following steps

1.Select mutation operator

2.Generate mutants

3.Distinguish mutants

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 32

 Mutation operator

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 33

 Mutation Operators
• Syntactic change from legal program to legal
program
• So: Specific to each programming language. C++ mutations
don’t work for Java, Java mutations don’t work for Python
• Examples:
– crp: constant for constant replacement
• for instance: from (x < 5) to (x < 12)
• select from constants found somewhere in program text
– ror: relational operator replacement
• for instance: from (x <= 5) to (x < 5)
– vie: variable initialization elimination
• change int x =5; to int x;

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 34

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 35
 Live Mutants
• Scenario:
– We create 100 mutants from our program
– We run our test suite on all 100 mutants, plus the
original program
– The original program passes all tests
– 94 mutant programs are killed (fail at least one test)
– 6 mutants remain alive
• What can we learn from the living mutants?

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 36

 How mutants survive
• A mutant may be equivalent to the original
program
– Maybe changing (x < 0) to (x <= 0) didn’t change the
output at all! The seeded “fault” is not really a
“fault”.
• Determining whether a mutant is equivalent may be easy or
hard; in the worst case it is undecideable
• Or the test suite could be inadequate
– If the mutant could have been killed, but was not, it
indicates a weakness in the test suite
– But adding a test case for just this mutant is a bad
idea. We care about the real bugs, not the fakes!

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 37

 Variation in mutation Testing
• Since the number of mutants that can be
generated is large (the number is usually on the
order of N2, where N is the number of variable
references in the program), methods have been
suggested to reduce the computational
expenses of this testing technique.
• Methods proposed over the years to combat the
expensive computation problem include weak
mutation, Strong mutation, Statistical
mutation.

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 38

 Variations on Mutation
• Weak mutation
• Statistical mutation

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 39

 Weak mutation
• Weak mutation only requires the test data to cause a mutated
component to take on a different value in at least one execution,
instead of outputting a different value from the expected result.

• Problem: There are lots of mutants. Running each test case to

completion on every mutant is expensive
• Number of mutants grows with the square of program size
• Approach:
– Execute meta-mutant (with many seeded faults) together with
original program
– Mark a seeded fault as “killed” as soon as a difference in
intermediate state is found
• Without waiting for program completion
• Restart with new mutant selection after each “kill”
(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 40
(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 41
 Statistical Mutation
• Problem: There are lots of mutants. Running
each test case on every mutant is expensive
• It’s just too expensive to create N2 mutants for a program of
N lines (even if we don’t run each test case separately to
completion)
• Approach: Just create a random sample of
mutants
– May be just as good for assessing a test suite
• Provided we don’t design test cases to kill particular
mutants (which would be like selectively picking out black
marbles anyway)

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 42

 In real life ...
• Fault-based testing is a widely used in
semiconductor manufacturing
– With good fault models of typical manufacturing
faults, e.g., “stuck-at-one” for a transistor
– But fault-based testing for design errors is more
challenging (as in software)
• Mutation testing is not widely used in industry
– But plays a role in software testing research, to
compare effectiveness of testing techniques
• Some use of fault models to design test cases is
important and widely practiced
(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 43
 Summary
• If bugs were marbles ...
– We could get some nice black marbles to judge the
quality of test suites
• Since bugs aren’t marbles ...
– Mutation testing rests on some troubling assumptions
about seeded faults, which may not be statistically
representative of real faults
• Nonetheless ...
– A model of typical or important faults is invaluable
information for designing and assessing test suites

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 44

 Contents
Test Execution:
•Overview from test case specifications to test
cases.
•Scaffolding.
• Generic versus specific scaffolding.
• Test oracles.
•Self-checks as oracles.
•Capture and replay.

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 45

 Automating Test Execution
• Designing test cases and test suites is creative
– Like any design activity: A demanding intellectual
activity, requiring human judgment
• Executing test cases should be automatic
– Design once, execute many times
• Test automation separates the creative human
process from the mechanical process of test
execution

(c) 2007 Mauro Pezzè & Michal Young Ch 17, slide 46

 Generation: From Test Case
Specifications to Test Cases
• Test design often yields test case
specifications, rather than concrete data
– Ex: “a large positive number”, not 420023
– Ex: “a sorted sequence, length > 2”, not “Alpha,
Beta, Chi, Omega”
• Other details for execution may be omitted
• Generation creates concrete, executable test
cases from test case specifications

(c) 2007 Mauro Pezzè & Michal Young Ch 17, slide 47

Example Tool Chain for Test
Case Generation & Execution
• We could combine ...
– A combinatorial test case generation (like
genpairs.py) to create test data
• Optional: Constraint-based data generator to “concretize”
individual values, e.g., from “positive integer” to 42
– DDSteps to convert from spreadsheet data to JUnit
test cases
– JUnit to execute concrete test cases
• Many other tool chains are possible ...
– depending on application domain

Photo: (c) Scott Robinson (clearlyambiguous on Flickr) , creative commons attribution

(c) license
2007 Mauro Pezzè & Michal Young Ch 17, slide 48
 Scaffolding
• Code produced to
support development
activities (especially
testing)
– Not part of the “product”
as seen by the end user
– May be temporary (like
scaffolding in construction
of buildings
• Includes
– Test harnesses, drivers,
and stubs

Image by Kevin Dooley under Creative Commons

(c) 2007 Mauro Pezzè & Michal Young
license Ch 17, slide 49
 Scaffolding ...
• Test driver
– A “main” program for running a test
• May be produced before a “real” main program
• Provides more control than the “real” main program
– To driver program under test through test cases

• Test stubs
– Substitute for called functions/methods/objects
• Test harness
– Substitutes for other parts of the deployed
environment
• Ex: Software simulation of a hardware device

(c) 2007 Mauro Pezzè & Michal Young Ch 17, slide 50

 Controllability & Observability
Example: We want
GUI input (MVC “Controller”) automated tests, but
interactive input provides
limited control and graphical
output provides limited
observability

Program Functionality

Graphical ouput (MVC “View”)

(c) 2007 Mauro Pezzè & Michal Young Ch 17, slide 51

 Generic or Specific?
• How general should scaffolding be?
– We could build a driver and stubs for each test case
– ... or at least factor out some common code of the
driver and test management (e.g., JUnit)
– ... or further factor out some common support code,
to drive a large number of test cases from data (as in
DDSteps)
– ... or further, generate the data automatically from
a more abstract model (e.g., network traffic model)
• A question of costs and re-use
– Just as for other kinds of software

(c) 2007 Mauro Pezzè & Michal Young Ch 17, slide 52

 Comparison-based oracle

• With a comparison-based oracle, we need predicted

output for each input
– Oracle compares actual to predicted output, and reports
failure if they differ
• Fine for a small number of hand-generated test cases
– E.g., for hand-written JUnit test cases
(c) 2007 Mauro Pezzè & Michal Young Ch 17, slide 53
 Self-Checking Code as Oracle

• An oracle can also be written as self-checks

– Often possible to judge correctness without predicting results
• Advantages and limits: Usable with large, automatically
generated test suites, but often only a partial check
– e.g., structural invariants of data structures
– recognize many or most failures, but not all
(c) 2007 Mauro Pezzè & Michal Young Ch 17, slide 54
 Capture and Replay
• Sometimes there is no alternative to human
input and observation
– Even if we separate testing program functionality
from GUI, some testing of the GUI is required
• We can at least cut repetition of human testing
• Capture a manually run test case, replay it
automatically
– with a comparison-based test oracle: behavior same
as previously accepted behavior
• reusable only until a program change invalidates it
• lifetime depends on abstraction level of input and output

(c) 2007 Mauro Pezzè & Michal Young Ch 17, slide 55

 Summary
• Goal: Separate creative task of test design from
mechanical task of test execution
– Enable generation and execution of large test suites
– Re-execute test suites frequently (e.g., nightly or
after each program change)
• Scaffolding: Code to support development and
testing
– Test drivers, stubs, harness, including oracles
– Ranging from individual, hand-written test case
drivers to automatic generation and testing of large
test suites
– Capture/replay where human interaction is required
(c) 2007 Mauro Pezzè & Michal Young Ch 17, slide 56