Structural Testing

& Mutation
Filippo Ricca
DISI, Universit di Genova, Italy
[email protected]

White vs. Black box testing

A white box testing is based upon

explicit knowledge of the SUT and its
structure

Called also structural testing

Statement coverage testing is an example

Inputs

Outputs

Inputs
Called also functional testing
Usually testcases are generated starting from
requirements/specifications

Outputs

The aim is to create enough testcases to ensure

every statement is executed at least once

A black box testing approach will device

testcases without any knowledge of the
SUT or any aspect of its structure

What is adequacy?
Test suite T
P

Fault detection
what if no faults are found?
How good this test suite is? Has P been
tested thoroughly?, or Is T adequate?
Two possibilities:

Test Coverage and program mutation

Test Coverage

Test Coverage

Coverage can be based on:

source code
model

control flow graph

(extended) finite state machines
data flow graph

requirements checklist
...
5

Coverage: what to measure?

For any coverage measure, we need:

A coverage unit: an element with the properties:

We can count the total number of units in the software

We can identify which units were hit during a single
execution run

This means that we can determine the percentage

of units hit during one or more execution runs

89%

Control flow coverage

Statement coverage
Branch coverage (also called decision coverage)
Minimum coverage specified by the IEEE unit test
standard
Multiple Condition coverage
Covers combinations of condition in decisions

If (x=6) && (y=7) then

Path coverage
100% path coverage impossible in practice

Loops ...
7

Control Flow graph

1
int proc(int a, int b, int x)
{
if ((a>1) && (b==0)) // 1
{
x = x/a; // 3
}
if ((a==2)||(x>1)) // 4
{
x = x+1; // 5
}
return x; // 7
}

true

a > 1 AND
b=0

false
xx/a

4
a == 2
OR x>1

true

false

x x+1

7
8

Statement Coverage

Criterion: All statements must be covered

during test execution
This is the weakest form of coverage

Some branches may be missed

Procedure:
Find paths that cover all statements
Choose input data that will result in the
selected paths
9

Statement Coverage

The following path is sufficient

for statement coverage:
13457
int proc(int a, int b, int x)

Possible input:
a = 2, b = 0, x = 4

true

a > 1 AND
b=0

false
xx/a

4
a == 2
OR x>1

true

false

x x+1

7
10

Not covered branches

1
true

a > 1 AND
b=0

false
xx/a

4
a == 2
OR x>1

true

false

x x+1

7
11

Branch Coverage

Criterion: At any branch point, each branch

must be covered during test execution

The true and false branch of a 2-way if statement

Each case in a switch statement
Loops

While, for, goto, .

Procedure:
Find paths that cover all branches
Choose input data that will result in the
selected paths
Branch coverage includes statement coverage!

12

Branch Coverage

The following paths are

sufficient for branch
coverage:
12457
13467

a > 1 AND
b=0

false
xx/a

1.
2.

Possible input:
a = 2, b = 2, x = -1
a = 3, b = 0, x = 1

a == 2
OR x>1

true

true

false

x x+1

7
13

Multiple Condition Coverage

Criterion:

Every atomic (i.e. does not include AND or OR)

condition must be true and false at some point
during test execution
In a compound logical statement (i.e. includes
AND and OR), every combination of atomic
conditions must be covered during test execution

Achieving multiple condition coverage also

satisfies statement and branch coverage
14

Multiple Condition Coverage

int proc(int a, int b, int x)
{
Need cases where
if ( (a>1) && (b==0) )
1.
a > 1 is true and b = 0 is true
{
2.
a > 1 is true and b = 0 is false
x = x/a;
3.
a > 1 is false and b = 0 is true
}
if ( (a==2) || (x>1) )
4.
a > 1 is false and b = 0 is false
{
5.
a = 2 is true and x > 1 is true
x = x+1;
6.
a = 2 is true and x > 1 is false
}
7.
a = 2 is false and x > 1 is true
return x;
8.
a = 2 is false and x > 1 is false
}

15

Multiple Condition Coverage

1. a > 1 is true and b = 0 is true
2. a > 1 is true and b = 0 is false
3. a > 1 is false and b = 0 is true
4. a > 1 is false and b = 0 is false
5. a = 2 is true and x > 1 is true
6. a = 2 is true and x > 1 is false

Possible input:
a = 2, b = 0,
a = 2, b = 1,
a = 0, b = 0,
a = 0, b = 1,

x
x
x
x

=
=
=
=

2
0
2
0

[1][5]
[2][6]
[3][7]
[4][8]

7. a = 2 is false and x > 1 is true

8. a = 2 is false and x > 1 is false

16

Multiple Condition Coverage

Multiple condition coverage

covers all branches and
statements.
Input values:
a
a
a
a

=
=
=
=

2,
2,
0,
0,

b
b
b
b

=
=
=
=

0,
1,
0,
1,

x
x
x
x

Paths covered
13457
12457
12457
12467

=
=
=
=

2
0
2
0

false
xx/a

4
a == 2
OR x>1

Equal

true

a > 1 AND
b=0

true

false

x x+1

7
17

All Paths Coverage

Criterion:

All paths through the code must be covered

This is typically infeasible when loops are

present

A version of this coverage with loops is to treat loops

as having two paths:
1.
2.

The loop is executed (normally, once)

The loop is skipped

Some paths may also be infeasible because there is no

combination of data conditions that permit a path to be
taken

See example below

18

All Paths Coverage

Set of all paths:

12467
13467
12457
13457

=
=
=
=

0,
3,
2,
2,

b
b
b
b

=
=
=
=

false
xx/a

Input values:
a
a
a
a

1,
0,
1,
0,

x
x
x
x

true

a > 1 AND
b=0

=
=
=
=

0
0
0
0

a == 2
OR x>1

true

false

x x+1

7
19

Comparison

From the previous two examples, we

can see that:

Multiple condition coverage does not imply

all paths coverage
All paths coverage does not imply multiple
condition coverage

20

Infeasible paths

Set of paths:
12467
13467
12457
13457
To be able to take this path, we
would have to have:

a <= 1 AND a > 4

which is logically impossible!

1
true
a>1

false
xx/a

4
a>4

true

false

x x+1

7
21

Code Coverage Tools

Clover:

Emma:

http://coverlipse.sourceforge.net

Cobertura:

http://emma.sourceforge.net
http://www.eclemma.org

Coverlipse:

http://www.cenqua.com/clover

http://cobertura.sourceforge.net

....
22

Emma

Open-source tool

Supports class, method, and line coverage

Fractional line coverage supported, but not

branch coverage

Standalone version

Only part of a line covered

http://emma.sourceforge.net

Eclipse plugin EclEmma also available

Installing the plug-in In Eclipse:

Help -> install new software -> http://update.eclemma.org/

23

Testcases are written in Junit!

Emma coverage report

24

Emma source code annotations

25

Fractional line coverage

Only part
of conditional
executed
Loop increment
not executed
green for fully covered lines,
yellow for partly covered lines and
red for lines that have not been executed at all
26

EMMA example
public class Calc {
int tot;
Calc(){
tot=0;
}
int somma (int x) {
tot=tot + x;
return tot;
}
int sottrai (int x) {
tot=tot - x;
return tot;
}
int moltiplica (int x) {
tot=tot * x;
return tot;
}
}

public class CalcTest {

Calc c;
@Before
public void setUp(){
c= new Calc();
}
@Test
public void testSomma() {
assertEquals(c.somma(3), 3);
}
@Test
public void testSottrai() {
assertEquals(c.sottrai(0), 0);
}
@Test
public void testMoltiplica() {
c.somma(2);
assertEquals(c.moltiplica(3), 6);
}
}

Output EMMA

Coverage = 100%

Program Mutation

29

What is program mutation?

P is alterated in several istances (automatically)

For several reasons:

Computing test adequacy of a testsuite

Improving/completing a testsuite
Detecting new errors

30

Mutation terms

Mutant: a copy of the original program with

a small change (seeded fault)
Mutation operator: applied to make change
(automatically) to the original program

Es. + --> *

Mutant killed: if its behaviors/outputs differ

from those of the original program

Live otherwise
31

Mutation operator

32

Method-level operator
(examples)
Mutant operator

In P

In mutant

Variable replacement

z=x*y+1;

x=x*y+1;
z=x*x+1;

Relational operator
replacement

if (x<y)

if(x>y)
if(x<=y)

Off-by-1

z=x*y+1;

z=x*(y+1)+1;
z=(x+1)*y+1;

Replacement by 0

z=x*y+1;

z=0*y+1;
z=0;

Arithmetic operator
replacement

z=x*y+1;

z=x*y-1;
z=x+y-1;
33

Class-level operator
(examples)

Access modifier change:

The AMC operator changes
the access level for instance
variables and methods to
other access levels
Hiding variable deletion:
The IHD operator deletes a
hiding variable, a variable in
a subclass that has the same
name and type as a variable
in the parent class

34

Mutant score
Mutant killed: if its behaviors/outputs differ from those of the original program

testcase

35

Test adequacy using mutation

Mutant/Mutation score can be also computed

for the entire Testsuite
MS(T)=number killed mutants/number mutants

MS(T)=1 means that all the mutant are killed

=> the testsuite is 100% adequate

Best case ...

In same cases it is not possible to reach

MS(T)=1

Mutants equivalent to P!

Ex. x+y --> x+y+0

Automatic tools should avoid it!

36

Computing Adequacy
Foo

Foo
Foo

MS(FooTest)=?

Foo

Tools Usage
Mutating Foo
Tests: FooTest
Mutation points = 12, unit test time limit 2.02s
.. M FAIL: Foo:31: negated conditional
M FAIL: Foo:33: negated conditional
M FAIL: Foo:34: - -> +
M FAIL: Foo:35: negated conditional
......
Score: 67%

37

Test enhancement using mutation

One has the opportunity to enhance a test set T

after having assessed its adequacy

If the mutation score (MS) is 1, then some other

technique, or a different set of mutants, needs to be
used to help enhance T
If the mutation score (MS) is less than 1, then there
exist live mutants. Each live mutant needs to be
distinguished from P

Adding testcases to the testuite!!!!!

38

Error detection using mutation

There is no guarantee that tests derived to

distinguish live mutants will reveal a yet
undiscovered error in P
Nevertheless, empirical studies have found to be the
most powerful of all enhancement techniques
The next simple example illustrates how test
enhancement using mutation detects errors

39

Foo (1)

Consider the following function foo that is

required to return the sum of two integers x
and y. Clearly foo is incorrect
int foo(int x, y) {
return (x-y);
}

This should be return (x+y)

Now suppose that foo has been tested using

a test set T that contains two tests
T={ t1: <x=1, y=0>, t2: <x=-1, y=0>}

Foo(1,0)=1 and Foo(-1,0)=-1

Correct?

40

int foo(int x, y) {
return (x-y);
}

Foo (2)
M1:

int foo(int x, y) {
return (x+y);
}

mutants

M2: int foo(int x, y) {

return (x-0);
}

M3: int foo(int x, y) {

return (0+y);
}

Testsuite
is
not
adequate!
Next we execute each mutant against tests in T
MS(T)= 1/3
T={ t1: <x=1, y=0>, t2: <x=-1, y=0>}

Test (t)

foo(t)

M1(t)

M2(t)

M3(t)

t1

t2

-1

-1

-1

Live

Live

Killed

41

Foo (3)

int foo(int x, y) {
return (x-y);
}

M1: int foo(int x, y) {

return (x+y);
}

Let us examine M1
A test that distinguishes M1 from foo must satisfy the
following condition:
x-yx+y => -yy => -2y 0 => y 0

Hence we get t3: <x=1, y=1>

Executing foo on t3 gives us foo(t3)=0
However, according to the requirements we must
get foo(t3)=2
Thus t3 distinguishes M1 from foo and also reveals
the error
42

Tools for mutation testing: features

A typical tool for
testing offers the
features:

mutation
following

MuClipse

A selectable palette of mutation

operators
Management of test set T
Ex. Junit
Generation of mutants
Execution of the program under test
and mutants against T
Report Mutation score (adequacy)
43

Tools Free for mutation testing

Proteum is the first mutation testing tool that implements

all mutation operators designed for the ANSI C
programming language
Java (muJava) is a mutation system for Java programs

Jumble is a mutation system for Java programs with

JUnit tests. Jumble can be used from the command line
or as an Eclipse plugin

MuClipse is an Eclipse plugin verion of Java (muJava)

The bytecode is mutated

Others at:

http://www.mutationtest.net/twiki/bin/view/Resources/WebHome
44

Jumble installation
1.

Download Jumble from:

2.

3.

Unzip the downloaded file in a folder:

jumble_1_1_0
Copy or Move the folder jumble (jumble_1_1_0 >
eclipseplugin > plugin-export) to your Eclipse plugins
folder:

4.

http://jumble.sourceforge.net

Eclipse >plugins, or Eclipse > dropins > eclipse >

plugins

Restart Eclipse
45

Run default mutation testing

If :

class to be mutated

<YourClass> and your test are in

the same package
and your tests name follows this
pattern: <YourClass>Test.java,

then you
Jumble
1.
2.
3.

can

run

Right click on <YourClass>

Choose Jumble > Jumble Class
Look at the Console view to see
the mutant score
46

How Jumble work?

class "Foo
JUnit tests "FooTest"
Jumble starts by running the unit tests (in
FooTest.class) on the unmodified Foo class to check
that they all pass, and to measure the time taken by
each test
Then it will mutate Foo in various ways and run the
tests again to see if they detect the mutation
It continues this process until all mutations of Foo
have been tried
It provides the output

47

Jumble Output
Jumble found 12 different mutants of Foo

Mutating Foo
Tests: FooTest
Mutation points = 12, unit test time limit 2.02s
.. M FAIL: Foo:31: negated conditional
M FAIL: Foo:33: negated conditional
M FAIL: Foo:34: - -> +
M FAIL: Foo:35: negated conditional
......
Score: 67%
if (C) decision on line 31 was mutated to if (!C)
unit tests kill the mutant (indicated by a '.')

Overall, 67% (8/12) of the mutations were killed by the unit

tests, which means that they probably need to be improved

48

Jumble mutations

Conditionals: Jumble replaces each condition with its negation

the condition x > y would mutate to become !(x > y)

Arithmetic Operations: Jumble replace arithmetic operations

A fixed table is used

Increments: Increments, decrement are mutated

for example, 0 --> 1

Return Values: Jumble can change return values

for example, i++ becomes i--

Constants: Jumble can change the value of literal constants

+ --> -, * --> /

for example, return X --> return 0

Switch Statements: Jumble can mutate each case of a switch

swapping
49

Jumble Example
public class CalcTest {
Calc c;
public class Calc {
int tot;
Calc(){
tot=0;
}
int somma (int x) {
tot=tot + x;
return tot;
}
int sottrai (int x) {
tot=tot - x;
return tot;
}
int moltiplica (int x) {
tot=tot * x;
return tot;
}
}

@Before
public void setUp(){
c= new Calc();
}
@Test
public void testSomma() {
Assert.assertEquals(c.somma(3), 3);
}
@Test
public void testSottrai() {
Assert.assertEquals(c.sottrai(0), 0);
}
@Test
public void testMoltiplica() {
c.somma(2);
Assert.assertEquals(c.moltiplica(3), 6);
}
}

Mutant non killed

Jumble output

public class Calc {

int tot;
Calc(){
tot=0;
}

Mutating Calc
Tests: CalcTest
Mutation points = 7, unit test
time limit 2.31s
...M FAIL: Calc:15: - -> +
...
Score: 85%

int somma (int x) {

tot=tot + x;
return tot;
}
int sottrai (int x) {
tot=tot + x;
return tot;
}
int moltiplica (int x) {
tot=tot * x;
return tot;
}
}

51

public class CalcTest {

Calc c;
@Before
public void setUp(){
c= new Calc();
}
@Test
public void testSomma() {
Assert.assertEquals(c.somma(3), 3);
}
@Test
public void testSottrai() {
Assert.assertEquals(c.sottrai(3), -3);
}
@Test
public void testMoltiplica() {
c.somma(2);
Assert.assertEquals(c.moltiplica(3), 6);
}
}

Test enhancement
Mutating Calc
Tests: CalcTest
Mutation points = 7, unit test
time limit 2.31s
.......
Score: 100%

Possible exercises at the exam

Given a piece of code:

Build the CFG

Find paths that cover all statements, branches, conditions,
or paths
Choose input data that will result in the selected paths
See exercise Delete rows from array at

Enhance a test set T after having assessed its

adequacy

Given P, T (Junit tests) and a set of Jumble mutations

http://www.site.uottawa.ca/~awilliam/seg3203/Coverage.ppt

Foo:33: negated conditional

Error detection using mutation

Ex. Foo

53

References

(used to prepare these slides)

Slides of the book Foundations of software testing

by Aditya P. Mathur

Slides of Alan Williams, University of Ottawa

http://www.site.uottawa.ca/~awilliam/seg3203/
Glover, Dont be fooled by the Coverage Report,
IBM developer works article

http://www-128.ibm.com/developerworks/java/library/j-cq01316

Slides of Cu Nguyen Duy, Software Analysis and

Testing 2009-2010

http://www.cs.purdue.edu/homes/apm/foundationsBook/
InstructorSlides.html

http://selab.fbk.eu/swat/program.ml?lang=en

Wikipedia
54