!

Lionel Briand 2010

1
Software Verification and Validation
Prof. Lionel Briand
Ph.D., IEEE Fellow
! Lionel Briand 2010
2
White-Box Testing
! Lionel Briand 2010
3
Specification
System
Implementation
Missing functionality:
Cannot be revealed by white-
box techniques
Unexpected functionality:
Cannot be revealed by black-
box techniques
! Lionel Briand 2010
4
Develop an initial
Test suite using BB
techniques
Apply BB coverage
criteria to enhance it
Enhance the Test suite
using WB techniques
Apply WB coverage
criteria to enhance it
Analyze the parts of
the code uncovered
by BB test suite
! Lionel Briand 2010
5
Greatest common divisor (GCD) program
read(x);
read(y);
while x " y loop
if x>y then
x := x y;
else
y := y x;
end if;
end loop;
gcd := x;
x<=y x > y
x = y
x " y
! Lionel Briand 2010
6
Directed graph
Nodes are blocks of sequential statements
Edges are transfers of control
Edges may be labeled with predicate
representing the condition of control
transfer
There are several conventions for flow
graph models with subtle differences
(e.g., hierarchical CFGs, concurrent CFGs)
x<=y x > y
x = y
x " y
! Lionel Briand 2010
7
If-Then-Else While loop Switch
! Lionel Briand 2010
8
[year < 1]
[month in (1,3,5,7,10,12)]
n=31
throw2 n=28
return
throw1
n=29
n=30
[month in (4,6,9,11)]
[month == 2] [leap(year)]
! Lionel Briand 2010
9
Control flow Coverage
Depending on the (adequacy) criteria to cover (traverse)
CFGs, we can have different types of control flow
coverage:
Statement/Node Coverage
Edge Coverage
Condition Coverage
Path Coverage
Discussed in detail next
x<=y x > y
x = y
x " y
! Lionel Briand 2010
10
Statement/Node Coverage
Hypothesis: Faults cannot be discovered if
the statements containing them are not
executed
Statement coverage criteria: Equivalent to
covering all nodes in CFG
Executing a statement is a weak guarantee
of correctness, but easy to achieve
In general, several inputs execute the
same statements
An important question in practice is: how
can we minimize (the number of) test
cases so we can achieve a given statement
coverage ratio?
x<=y x > y
x = y
x " y
! Lionel Briand 2010
11
Incompleteness of Statement/Node
Coverage
Though often used, statement coverage is a weak
guarantee of fault detection
An example:
if x < 0 then
x := -x;
end if
z := x;
A negative x would result in the coverage
of all statements.
But not exercising x >= 0 is an important
case.
Doing nothing for the case x >= 0 may turn
out to be wrong and need to be tested.
! Lionel Briand 2010
12
Based on the program structure, the
control flow graph (CFG)
Edge coverage criterion: Select a test
set T such that, by executing P for each
test case t in T, each edge of Ps control
flow graph is traversed at least once
We need to exercise all conditions that
traverse the control flow of the
program with true and false values
x<=y x > y
x = y
x " y
! Lionel Briand 2010
13
Example: Searching for an element in a table
Derivation of the test set in the next slide
counter:= 0;
found := false;
if number_of_items " 0 then
counter :=1;
while (not found) and counter < number_of_items loop
if table(counter) = desired_element then
found := true;
end if;
counter := counter + 1;
end loop;
end if;
if found then write (the desired element exists in the table);
else write (the desired element does not exists in the table);
end if;
Here is an example of a common mistake in a search algorithm.
If the desired element = last element, we exit the loop before we find
it.
Should have been #
! Lionel Briand 2010
14
Examples Control Flow Graph
while
if
found := true
init
counter++
if
if
else then
! Lionel Briand 2010
15
We choose a test set with two test cases:
1. One table with 0 items and,
2. A table with 3 items, the second element being the desired one
For the second test case, the while loop body is executed twice,
once executing the if-then branch.
The edge coverage criterion is fulfilled and the error is NOT
discovered by the test set
...
while (not found) and counter < number_of_items loop
...
The reason for the above problem?
Not all possible values of the constituents of the condition in the while
loop have been exercised. counter < number_of_items is not
evaluated to False.
! Lionel Briand 2010
16
We need to further strengthen the edge coverage criterion
Condition Coverage (CC) Criterion: Select a test set T such
that, by executing P for each element in T, each edge of Ps
control flow graph is traversed, and all possible values of the
constituents of compound conditions (defined below) are
exercised at least once
Compound conditions: C1 and C2 or C3 where Cis are
relational expressions or Boolean variables (atomic conditions)
Another version: Modified Condition-Decision Coverage (MC
/DC) Criterion: Only combinations of values such that every Ci
drives the overall condition truth value twice (true and false).
Examples next
! Lionel Briand 2010
17
Two equivalent programs: though you would write the left
one
Edge coverage
would not compulsorily cover the hidden edges in the right one
Example: C2 = false might not be covered
Condition coverage would cover C2 = false
if c1 and c2 then
st;
else
sf;
end if;
if c1 then
if c2 then
st;
else
sf;
end if;
else
sf;
end if;
! Lionel Briand 2010
18
Use in industry: The international standard DO-178B for
Airborne Systems Certification (since 1992) requires
testing the airborne software systems with MC/DC
coverage.
Example : A$(B%C), e.g., in a while loop,
Deriving a modified condition /decision
criterion (MC/DC) Test suite:
Take a pair for each constituent:
A: (1,5), or (2,6), or (3,7)
B: (2,4)
C: (3,4)
Two minimal sets to cover the MCC:
(2,3,4,6) or (2,3,4,7)
That is 4 test cases instead of 8 for
all possible combinations.
ABC Results Corresponding negate
Case
1 TTT T A (5)
2 TTF T A (6), B (4)
3 TFT T A (7), C (4)
4 TFF F B (2), C (3)
5 FTT F A (1)
6 FTF F A (2)
7 FFT F A (3)
8 FFF F -
! Lionel Briand 2010
19
A Real Example: MC/DC coverage in Military
Software
Consider the controller SW of a modern tank with three guns
(x, y, z) is the 3D coordinate of a hit target (e.g., from the GPS unit)
The SW has the following requirements:
R1.1: Invoke method Fire1 (firing with gun #1) when (x<y) AND (z*z > y)
AND (previous direction of the gun=East).
R1.2: Invoke Fire2 when (x<y) AND (z*z # y) OR (current direction
=South).
R1.3: Invoke Fire3 when none of the two conditions above is true.
R2: The invocation described above must continue until an input Boolean
variable becomes true (when the stop firing button is pressed).
! Lionel Briand 2010
20
MC/DC coverage: Example (contd.)
! Lionel Briand 2010
21
MC/DC coverage: Example (contd.)
Verify that the following test suite of four tests, executed in
the given order, is adequate with respect to statement, and
decision (edge) coverage criteria but not with respect to the
condition coverage criterion.
Draw a CFG for this program and verify that all statements are
covered.
We assess the decision (edge) and condition coverage here
! Lionel Briand 2010
22
Path Coverage Criterion: Select a test set
T such that, by executing P for each test
case t in T, all paths leading from the
initial to the final node of Ps control flow
graph are traversed
In practice, however, the number of paths
is too large, if not infinite (e.g., when we
have loops)
Some paths are infeasible (e.g., not
practical given the systems business logic)
It may be important to determine critical
paths, leading to more system load,
security intrusions, etc.
x<=y x > y
x = y
x " y
! Lionel Briand 2010
23
Path Coverage - Example
if x " 0 then
y := 5;
else
z := z x;
end if;
if z > 1 then
z := z / x;
else
z := 0;
end if;
Let us compare how the following
two test sets cover this CFG:
T1 (test set) =
{TC11:<x=0, z =1>,
TC12:<x =1, z=3>}
T2 =
{TC21:<x=0, z =3>,
TC22:<x =1, z=1>}
See next
! Lionel Briand 2010
24
Path Coverage Example
T1s coverage
if x " 0 then
y := 5;
else
z := z x;
end if;
if z > 1 then
z := z / x;
else
z := 0;
end if;
T1s coverage:
T1 (test set) =
{TC11:<x=0, z =1>,
TC12:<x =1, z=3>}
T1 executes all edges
but!
Do you see any testing issue
(uncovered paths which can
be sources of failure)?
T1 executes all edges
and all conditions but
does not test risk of
division by 0. (See
the red path)
! Lionel Briand 2010
25
Path Coverage Example
T2s coverage
if x " 0 then
y := 5;
else
z := z x;
end if;
if z > 1 then
z := z / x;
else
z := 0;
end if;
T2s coverage:
T2 =
{TC21:<x=0, z =3>,
TC22:<x =1, z=1>}
T2 would find the problem
(triggering division by 0) by
exercising the remaining
possible flows of control
through the program
fragment.
! Lionel Briand 2010
26
Path Coverage - Example
T1 (test set) =
{TC11:<x=0, z =1>,
TC12:<x =1, z=3>}
T1 executes all edges but do
not show risk of division by 0
T2 = {TC21:<x=0, z =3>,
TC22:<x =1, z=1>}
T2 would find the problem by
exercising the remaining
possible flows of control
through the program fragment
Observation:
T1 & T2 -> all paths
covered
! Lionel Briand 2010
27
In practice, however, the number of paths can be too large, if
not infinite (e.g., when we have loops) ' Impractical
An pragmatic heuristic: Look for conditions that execute
loops
Zero times
A maximum number of times
A average number of times (statistical criterion)
For example, in the array search algorithm
Skipping the loop (the table is empty)
Executing the loop once or twice and then finding the element
Searching the entire table without finding the desired element
! Lionel Briand 2010
28
Path Coverage Dealing with Loops
Another Example: Power Function
Program computing
Z=X^Y
BEGIN
read (X, Y) ;
W = abs(Y) ;
Z = 1 ;
WHILE (W <> 0) DO
Z = Z * X ;
W = W - 1 ;
END
IF (Y < 0) THEN
Z = 1 / Z ;
END
print (Z) ;
END
1
2
3 4
5
6
read(X,Y)
W=abs(Y)
Z=1
Z=Z*X
W=W-1
W!0 W=0
Y<0
Y"0
Z=1/Z
print(Z)
! Lionel Briand 2010
29
Path Coverage comparison with all
branches and all statements
All paths
Infeasible path
1 ' 2 ' 4 ' 5 ' 6, Why infeasible?
The way Y and W relate.
Potentially large number of paths (depends on Y)
As many ways to iterate
2 ' (3 ' 2)* as values of Abs(Y)
All branches
Two test cases are enough
Y<0 : 1 ' 2 ' (3 ' 2)+ ' 4 ' 5 ' 6
Y>0 : 1 ' 2 ' (3 ' 2)* ' 4 ' 6
All statements
One test case is enough
Y<0 : 1 ' 2 ' (3 ' 2)+ ' 4 ' 5 ' 6
1
2
3 4
5
6
read(X,Y)
W=abs(Y)
Z=1
Z=Z*X
W=W-1
W!0
W=0
Y<0
Y"0
Z=1/Z
print(Z)
! Lionel Briand 2010
30
To find test inputs that will execute an arbitrary
statement Q within a program source, the tester must
work backward from Q through the programs flow of
control to an input statement
For simple programs, this amounts to solving a set of
simultaneous inequalities in the input variables of the
program, each inequality describing the proper path
through one conditional
An example next
! Lionel Briand 2010
31
int z;
scanf(%d%d, &x, &y);
if (x > 3) {
z = x+y;
y+= x;
if (2*z == y) {
an example statement
/* e.g., we want this statement to be covered
by a test case */

Inequalities?
. 2(x+y)=x+y
! x = -y
. x> 3
We can have many
solutions, e.g.:
x = 4
y= -4
! Lionel Briand 2010
32
The presence of loops and recursion in the code
makes it very hard (and sometimes impossible)
to write and solve the inequalities in general
Each pass through a loop may alter the values of
variables that figure in a following conditional
and the number of passes cannot be determined
by static analysis in general
! Lionel Briand 2010
33
Control Flow Coverage
Reachability
Not all statements are usually reachable in real-world programs
It is not always possible to decide automatically if a statement is
reachable and the percentage of reachable statements
When one does not reach a 100% coverage, it is therefore difficult
to determine the reason
Tools are needed to support this activity but it cannot be fully
automated
Research focuses on search algorithms to help automate coverage
Control flow testing is, in general, more applicable to testing in the
small
! Lionel Briand 2010
34
! Lionel Briand 2010
35
Definitions and uses
A program written in a procedural language, such as C
and Java, contains variables. Variables are defined by
assigning values to them and are used in expressions.
Statement x = y + z defines variable x and uses
variables y and z
Statement scanf (%d %d, &x, &y) defines
variables x and y
Statement printf (Output: %d \n, x+y)
uses variables x and y
A parameter x passed as call-by-value to a function,
is considered as a use of (or a reference to) x
A parameter x passed as call-by-reference,
can serve as a definition and use of x
! Lionel Briand 2010
36
Definitions and uses: Pointers
Consider the following sequence of statements that use pointers.
The first of the above statements defines a pointer variable z
the second defines y and uses z
the third defines x through the pointer variable z, and
the last defines y and uses x accessed through the pointer
variable z
Variable z is a pointer pointing to variable x and
contains the memory address of variable x.
*z retrieves the value at the memory address pointed
by variable z. Consequently, *z = 25 is to assign 25 to
the memory address pointed by variable z. That is, to
assign 25 to variable x.
y=*z+1 is to define as the sum of 1 and
the value at the memory address pointed
by variable z, i.e., the value of x
! Lionel Briand 2010
37
Definitions and uses: Arrays
Arrays are also tricky. Consider the following declaration
and two statements in C:
The first statement defines variable A.
The second statement defines A and uses i , x, and y.
Alternate: second statement defines A[i] and not the entire
array A.
The choice of whether to consider the entire array A as
defined or the specific element depends upon how stringent is
the requirement for coverage analysis.
! Lionel Briand 2010
38
c-use
Uses of a variable that occurs within an expression as part
of an assignment statement, in an output statement, as a
parameter within a function call, and in subscript
expressions, are classified as c-use, where the c in c-use
stands for computational.
How many c-uses of x can you find in the following
statements?
! Lionel Briand 2010
39
p-use
The occurrence of a variable in an expression used as a
condition in a branch statement such as an if and a while, is
considered as a p-use. The p in p-use stands for
predicate.
How many p-uses of z and x can you find in the following
statements?
! Lionel Briand 2010
40
p-use: possible confusion
Consider the statement:
The use of A is clearly a p-use.
Is the use of x in the subscript, a c-use or a p-use?
! Lionel Briand 2010
41
c-uses within a basic block
While there are two definitions of p in this block, only the
second definition will propagate to the next block. The first
definition of p is considered local to the block while the
second definition is global.
We are concerned with global definitions, and uses.
Consider the basic block
Note that y and z are global uses; their definitions flow into this
block from some other block.
! Lionel Briand 2010
42
Data Flow Analysis - FACTORIAL
Example
1. public int factorial(int n){
2. int i, result = 1;
3. for (i=2; i<=n; i++) {
4. result = result * i;
5. }
6. return result;
7. }
Variable Definition
line
Use line
n 1 3 (Predicate)
result 2 4
(Computation)
result 2 6
result 4 4
result 4 6
i 3 3
i 3 4
! Lionel Briand 2010
43
Data Flow Analysis
Basic Formal Definitions
Definition (defining) node: Node n ( CFG(P) is a defining
node of the variable v ( V, written as DEF(v, n), iff (if
and only if) the value of the variable v is defined in the
statement corresponding to node n
Usage (use) node: Node n ( CFG(P) is a usage node of the
variable v ( V, written as USE(v, n), iff the value of the
variable v is used in the statement corresponding to
node n
Predicate and Computation use: A usage node USE(v, n) is
a predicate use (denoted as P-Use) iff the statement n
is a predicate statement, otherwise USE(v, n) is a
computation use (denoted as C-use)
! Lionel Briand 2010
44
Data Flow Analysis - Basic Definitions
II
PATHS(P): the set of all CFPs in program P
definition-use (du)-path: A definition-use path with
respect to a variable v (denoted du-path) is a path in
PATHS(P) such that, for some v ( V, there are
definition and usage nodes DEF(v, m) and USE(v, n) such
that m and n are initial and final nodes of the path.
definition-clear (dc)-path: A definition-clear path with
respect to a variable v (denoted dc-path) is a definition
-use path in PATH(P) with initial and final nodes DEF(v,
m) and USE(v, n) such that no other node in the path is a
defining node of v.
Examples next
! Lionel Briand 2010
45
Simple Example
if (max>0)
max=len
copy()
Definition of len, max, save[],
including max=0 and len=getLine()
len=getLine()
if (len>=max)
while (len>0)
end
print
definition
of max
definition
of len
p-use
of len
p-use
of max
c-use
of len
6
5
4
2
3
1
8
7
1-2, 1-2-3, 1-2-3-4, 5-2, 5-2-3, 1-2-3-4-5-2 are du-paths wrt variable len.
They are, except for 1-2-3-4-5-2, definition-clear wrt variable len.
! Lionel Briand 2010
46
Data flow graph
A data-flow graph of a program, denoted as def-use
graph, captures the flow of definitions (denotes as defs)
and uses across basic blocks in a program.
It is similar to a control flow graph of a program in that
the nodes, edges, and all paths in the control flow graph
are preserved in the data flow graph.
Attach defs, c-use and p-use to each node in the graph.
Label each edge with the condition which when true causes
the edge to be taken.
We use di(x) to refer to the definition of variable x at
node i. Similarly, ui(x) refers to the use of variable x at
node i.
! Lionel Briand 2010
47
Data flow graph: Example
! Lionel Briand 2010
48
Def-clear path
Any path starting from a node at
which variable x is defined and
ending at a node at which x is used,
without redefining x anywhere else
along the path, is a def-clear path
for x.
Path 2-5 is def-clear for variable z
defined at node 2 and used at node
5. Path 1-2-5 is NOT def-clear for
variable z defined at node 1 and
used at node 5.
Thus definition of z at node 2 is live at
node 5 while that at node 1 is not live
at node 5.
! Lionel Briand 2010
49
Def-use pairs
Def of a variable at line l
1
and its use at line l
2
constitute a
def-use pair. l
1
and l
2
can be the same.
dcu (d
i
(x)) or dcu(x, i) denotes the set of all nodes where
the definition of x at node i is live and used.
dpu (d
i
(x)) or dpu(x,i) denotes the set of all edges (k, l)
such that there is a def-clear path from node i to edge (k,
l) and x is used at node k.
We say that a def-use pair (d
i
(x), u
j
(x)) is covered when a
def-clear path that includes nodes i to node j is executed.
If u
j
(x) is a p-use then all edges of the kind (j, k) must also
be taken during some executions for the def-use pair to be
covered.
! Lionel Briand 2010
50
Def-use pairs (example)
! Lionel Briand 2010
51
Data Flow Analysis Data Flow Coverage Criteria (I)
all-Definitions criterion: The test set T satisfies the all-Definitions
criterion for the program P iff for every variable v ( V, T contains
definition-clear paths from every defining node of v to a use of v.
all-Uses criterion: The test set T satisfies the all-Uses criterion for
the program P iff for every variable v (V, T contains at least one
definition-clear path from every defining node of v to every
reachable use of v.
all-P-Uses/Some C-Uses criterion: The test set T satisfies the all-P
-Uses/Some C-Uses criterion for the program P iff for every
variable v (V, T contains at least one definition clear path from
every defining node of v to every predicate use of v, and if a
definition of v has no P-Uses, there is a definition-clear path to at
least one computation use.
! Lionel Briand 2010
52
Data Flow Analysis Data Flow Coverage Criteria
(II)
all-C-Uses/Some P-Uses criterion: The test set T satisfies the all-C
-Uses/Some P-Uses criterion for the program P iff for every
variable v (V, T contains at least one definition-clear path from
every defining node of v to every computation use of v, and if a
definition of v has no C-Uses, there is a definition-clear path to at
least one predicate use.
all-DU-Paths criterion: The test set T satisfies the all-DU-Paths
criterion for the program P iff for every variable v (V, T contains
all definition-clear paths from every defining node of v to every
reachable use of v, and that these paths are either single loops
traversals, or they are cycle free.
! Lionel Briand 2010
53
POWER Example
1
2
3 4
5
6
read(X,Y)
W=abs(Y)
Z=1
Z=Z*X
W=W-1
W!0 W=0
Y<0
Y"0
Z=1/Z
print(Z)
! Lionel Briand 2010
54
Data Flow Analysis (DFA) Discussion
In DFA, we generate test data according to the way data
is manipulated in the program
DFA can help us define intermediary CFA criteria
between all-nodes testing (possibly too weak) and all
-paths testing (often impossible)
But we need effective tool support for DFA if we want
to use it extensively.
! Lionel Briand 2010
55
The subsumption relationships
All paths
All definition-use paths
All uses
All computational/
some predicate uses
All computational uses
All definitions
All predicate/
some computational uses
All predicate uses
Edge (Branch)
Statement
(possibly too weak)
(possibly too tough. Often impossible)
! Lionel Briand 2010
56
Data Flow Analysis - Measuring Code
Coverage
One advantage of structural criteria is that their
coverage can be measured automatically
To control testing progress
To assess testing completeness in terms of remaining
faults and reliability
High coverage is not a guarantee of fault-free software,
just an element of information to increase our
confidence -> statistical models
! Lionel Briand 2010
57
Analysis of test coverage data
Test Productivity
! Lionel Briand 2010
58
Coverage %
Fault
Detection
score %
0
100
100
Criterion 1
Criterion 2
! Lionel Briand 2010
59
Hutchins, Foster, Goradia, Ostrand, Experiments on the Effectiveness of
Dataflow- and Control flow-Based Test Adequacy Criteria, proceedings of
the International Conference on Software Engineering, 1994
The All-Edges and All-DU coverage criteria were applied to 130 faulty
versions of 7 C programs (141-512 LOC) by seeding realistic faults
The 130 faults were created by 10 different people, mostly without
knowledge of each others work; their goal was to be as realistic as possible.
The test generation procedure was designed to produce a wide range both
of test size and test coverage percentages, i.e., at least 30 test sets for
each 2% coverage interval for each program
They examined the relationship between fault detection and test set
coverage / size
! Lionel Briand 2010
60
! Lionel Briand 2010
61
Analysis of test coverage data
Results of Hutchins et al.s work
Both coverage criteria performed better than random
test selection especially DU-coverage
Significant improvements occurred as coverage increased
from 90% to 100%
100% coverage alone is not a reliable indicator of the
effectiveness of a test set especially edge coverage
Wide variation in test effectiveness for a given coverage
criteria
As expected, on average, achieving all-DU coverage
required significantly larger test sets with all-Edge
coverage
! Lionel Briand 2010
62
A short Background on Eclipse, and
JUnit
Eclipse is a very popular Integrated Development
Environment (IDE) written primarily in and intended for
Java.
But it is now used for developing in many other languages
too, phpUnit, NUnit, .
If you go to the SW industry, you are most probably
going to see/use it!
JUnit is a unit testing framework for Java.
It is extremely popular in the SW industry.
Check the web:
en.wikipedia.org/wiki/List_of_unit_testing_frameworks
www.junit.org
en.wikipedia.org/wiki/JUnit
! Lionel Briand 2010
63
A short Background on CodeCover
CodeCover is an open-source code coverage tool for Java
under Eclipse.
It can be used inside Eclipse.
http://www.codecover.org
There are many other code coverage tool out there, but
this is one of the lightest and powerful ones.
See the list @ http://java-source.net/open-source/code
-coverage
! Lionel Briand 2010
64
Demo of Eclipse and JUnit
T={t1: <x=2, y=3, return=5>, t2: <x=3, y=1, return=3>}
T = {sumTestCase, productTestCase}