Software Testing

1
Organization of this
Lecture

Introduction to Testing.
White-box testing:
statement coverage
path coverage
branch testing
condition coverage
Cyclomatic complexity
Summary
2
How do you test a
system?
Input test data to the
system.
Observe the output:
Check if the system behaved
as expected.

3
How do you test a
system?
m
u
p
ne

S t
t

It

s
y

O
u
p

u
t

4
How do you test a
system?
If the program does
not behave as
expected:
note the conditions
under which it failed.
later debug and correct.
5
Errors and Failures
A failure is a manifestation
of an error (aka defect or
bug).
mere presence of an error
may not lead to a failure.

6
Test cases and Test
suite
Test a software using a set of
carefully designed test cases:
the set of all test cases is
called the test suite

7
Test cases and Test
suite
A test case is a triplet [I,S,O]:
I is the data to be input to the
system,
S is the state of the system at
which the data is input,
O is the expected output from the
system.

8
Verification versus
Validation
Verification is the process of
determining:
whether output of one phase of development
conforms to its previous phase.
Validation is the process of determining
whether a fully developed system
conforms to its SRS document.

9
Verification versus
Validation

Aim of Verification:
phase containment of errors
Aim of validation:
final product is error free.

10
Verification versus
Validation
Verification:
are we doing right?
Validation:
have we done right?

11
Design of Test Cases
Exhaustive testing of any non-
trivial system is impractical:
input data domain is extremely large.
Design an optimal test suite:
of reasonable size
to uncover as many errors as
possible.

12
Design of Test Cases
If test cases are selected randomly:
many test cases do not contribute to the
significance of the test suite,
do not detect errors not already detected
by other test cases in the suite.
The number of test cases in a randomly
selected test suite:
not an indication of the effectiveness of the
testing.

13
Design of Test Cases

Testing a system using a large number

of randomly selected test cases:
does not mean that many errors in the
system will be uncovered.
Consider an example:
finding the maximum of two integers x
and y.

14
Design of Test Cases

If (x>y) max = x;
else max = x;
The code has a simple error:
test suite {(x=3,y=2);(x=2,y=3)}
can detect the error,
a larger test suite
{(x=3,y=2);(x=4,y=3); (x=5,y=1)}
does not detect the error.
15
Design of Test Cases

Systematic approaches are

required to design an optimal
test suite:
each test case in the suite
should detect different errors.

16
Design of Test Cases
Two main approaches to
design test cases:
Black-box approach
White-box (or glass-box)
approach

17
Black-box Testing
Test cases are designed using only
functional specification of the
software:
without any knowledge of the internal
structure of the software.
For this reason, black-box testing is
also known as functional testing.

18
White-box Testing
Designing white-box test
cases:
requires knowledge about the
internal structure of software.
white-box testing is also
called structural testing.

19
Black-box Testing
Two main approaches to
design black box test cases:
Equivalence class partitioning
Boundary value analysis

20
White-Box Testing
There exist several popular white-box
testing methodologies:
Statement coverage
branch coverage
path coverage
condition coverage
mutation testing
data flow-based testing
21
Statement Coverage
Statement coverage
methodology:
design test cases so that
every statement in a
program is executed at least
once.

22
Statement Coverage

The principal idea:

unless a statement is executed,
we have no way of knowing if
an error exists in that
statement.

23
Statement coverage
criterion

Based on the observation:

an error in a program can
not be discovered:
unless the part of the
program containing the error
is executed.
24
Statement coverage
criterion

Observing that a statement

behaves properly for one input
value:
no guarantee that it will behave
correctly for all input values.

25
Example
int f1(int x, int y){
1 while (x != y){
2 if (x>y) then
3 Euclid's
x=x-y; GCD Algorithm
4 else y=y-x;
5}
6 return x; }

26
Euclid's GCD computation
algorithm

By choosing the test set

{(x=3,y=3),(x=4,y=3),
(x=3,y=4)}
all statements are executed at
least once.

27
Branch Coverage

Test cases are designed

such that:
different branch conditions
given true and false values in
turn.

28
Branch Coverage

Branch testing guarantees

statement coverage:
a stronger testing compared
to the statement coverage-
based testing.

29
Stronger testing
Test cases are a superset of a
weaker testing:
discovers at least as many errors
as a weaker testing
contains at least as many
significant test cases as a weaker
test.

30
Example
int f1(int x,int y){
1 while (x != y){
2 if (x>y) then
3 x=x-y;
4 else y=y-x;
5}
6 return x; }

31
Example
Test cases for branch coverage
can be:
{(x=3,y=3),(x=3,y=2),
(x=4,y=3), (x=3,y=4)}

32
Condition Coverage

Test cases are designed such

that:
each component of a composite
conditional expression
given both true and false values.

33
Example
Consider the conditional
expression
((c1.and.c2).or.c3):
Each of c1, c2, and c3 are
exercised at least once,
i.e. given true and false values.
34
Branch testing
Branch testing is the
simplest condition testing
strategy:
compound conditions
appearing in different branch
statements
are given true and false values.
35
Branch testing
Condition testing
stronger testing than branch
testing:
Branch testing
stronger than statement
coverage testing.

36
Condition coverage

Consider a boolean
expression having n
components:
for condition coverage we
n
require 2 test cases.
37
Condition coverage

Condition coverage-based
testing technique:
practical only if n (the
number of component
conditions) is small.

38
Example

if ((A || B) && C)
{
<< Few Statements >>
}
else
{
<< Few Statements >>
}

39
So, in our example, the 3 following tests would be
sufficient for 100% Condition coverage testing.
A = true | B = not eval | C = false
A = false | B = true | C = true
A = false | B = false | C = not eval

40
Write Test cases using condition
coverage

if {(X or Y) and Z} then

41
 To satisfy condition coverage, each Boolean
expression X,Y and Z in above statement should be
evaluated to TRUE and FALSE at least one time.
The TEST CASES for condition coverage will be:
TEST CASE1: X=TRUE, Y=TRUE, Z=TRUE
TEST CASE2: X=FALSE, Y=FALSE, Z=FALSE

42
Multiple Condition Decision Coverage

Multiple Condition Decision

Coverage(MCDC) is also known as
Modified Condition Decision Coverage.

In MCDC each condition should be

evaluated at least once which affects the
decision outcome independently.

43
To satisfy the decision coverage we need to
ensure that the IF statement evaluates to
TRUE and FALSE at least once. So the test set
will be:
TEST CASE1: X=TRUE, Y=TRUE,
Z=TRUE
TEST CASE2: X=FALSE, Y=FALSE,
Z=FALSE
44
Path Coverage

Design test cases such that:

all linearly independent
paths in the program are
executed at least once.

45
Linearly independent
paths
Defined in terms of
control flow graph (CFG) of
a program.

46
Path coverage-based
testing
To understand the path
coverage-based testing:
we need to learn how to draw
control flow graph of a
program.

47
Control flow graph
(CFG)
A control flow graph (CFG)
describes:
the sequence in which different
instructions of a program get
executed.
the way control flows through
the program.
48
How to draw Control
flow graph?

Number all the statements of

a program.
Numbered statements:
represent nodes of the control
flow graph.

49
How to draw Control
flow graph?
An edge from one node to
another node exists:
if execution of the statement
representing the first node
can result in transfer of control
to the other node.

50
Example
int f1(int x,int y){
1 while (x != y){
2 if (x>y) then
3 x=x-y;
4 else y=y-x;
5}
6 return x; }

51
Example Control Flow
Graph
1
2
3 4
5
6

52
How to draw Control
flow graph?

Sequence: 1
1 a=5;
2
2 b=a*b-1;

53
How to draw Control
flow graph?

Selection: 1
1 if(a>b) then 2 3
2 c=3; 4
3 else c=5;
4 c=c*c;

54
How to draw Control
flow graph?

Iteration:
1

1 while(a>b){ 2

2 b=b*a; 3
3 b=b-1;} 4
4 c=b+d;

55
Path
A path through a program:
a node and edge sequence
from the starting node to a
terminal node of the control
flow graph.
There may be several terminal
nodes for program. 56
Independent path
Any path through the
program:
introducing at least one new
node:
that is not included in any other
independent paths.
57
Independent path
It is straight forward:
to identify linearly independent
paths of simple programs.
For complicated programs:
it is not so easy to determine the
number of independent paths.

58
McCabe's cyclomatic
metric
An upper bound:
for the number of linearly
independent paths of a program
Provides a practical way of
determining:
the maximum number of linearly
independent paths in a program.
59
McCabe's cyclomatic
metric
Given a control flow graph G,
cyclomatic complexity V(G):
V(G)= E-N+2
N is the number of nodes in G
E is the number of edges in G

60
Example Control Flow
Graph
1
2
3 4
5
6

61
Example
Cyclomatic complexity =
7-6+2 = 3.

62
Cyclomatic complexity

Another way of computing cyclomatic

complexity:
inspect control flow graph
determine number of bounded areas in
the graph
V(G) = Total number of bounded
areas + 1
63
Bounded area
Any region enclosed by a
nodes and edge sequence.

64
Example Control Flow
Graph
1
2
3 4
5
6

65
Example
From a visual examination of
the CFG:
the number of bounded areas is
2.
cyclomatic complexity =
2+1=3.
66
Cyclomatic complexity

McCabe's metric provides:

a quantitative measure of testing
difficulty and the ultimate reliability
Intuitively,
number of bounded areas increases
with the number of decision nodes
and loops.

67
Cyclomatic complexity

The first method of computing

V(G) is amenable to automation:
you can write a program which
determines the number of nodes
and edges of a graph
applies the formula to find V(G).

68
Cyclomatic complexity

The cyclomatic complexity of a

program provides:
a lower bound on the number of
test cases to be designed
to guarantee coverage of all
linearly independent paths.

69
Cyclomatic complexity

Defines the number of

independent paths in a
program.
Provides a lower bound:
for the number of test cases for
path coverage.
70
Cyclomatic complexity

Knowing the number of test cases

required:
does not make it any easier to
derive the test cases,
only gives an indication of the
minimum number of test cases
required.
71
Path testing
The tester proposes:
an initial set of test data using
his experience and judgement.

72
Path testing
A dynamic program analyzer is
used:
to indicate which parts of the
program have been tested
the output of the dynamic analysis
used to guide the tester in selecting
additional test cases.
73
Derivation of Test
Cases

Let us discuss the steps:

to derive path coverage-
based test cases of a
program.

74
Derivation of Test
Cases
Draw control flow graph.
Determine V(G).
Determine the set of linearly
independent paths.
Prepare test cases:
to force execution along each path.
75
Example
int f1(int x,int y){
1 while (x != y){
2 if (x>y) then
3 x=x-y;
4 else y=y-x;
5}
6 return x; }

76
Example Control Flow
Diagram
1
2
3 4
5
6

77
Derivation of Test Cases

Number of independent paths:

3
1,6 test case (x=1, y=1)
1,2,3,5,1,6 test case(x=1, y=2)
1,2,4,5,1,6 test case(x=2,
y=1)
78
An interesting application
of cyclomatic complexity

Relationship exists between:

McCabe's metric
the number of errors existing in
the code,
the time required to find and
correct the errors.
79
Cyclomatic complexity

Cyclomatic complexity of a
program:
also indicates the psychological
complexity of a program.
difficulty level of understanding
the program.
80
Cyclomatic complexity

From maintenance perspective,

limit cyclomatic complexity
of modules to some reasonable value.
Good software development
organizations:
restrict cyclomatic complexity of
functions to a maximum of ten or so.

81
Summary
Exhaustive testing of non-
trivial systems is impractical:
we need to design an optimal
set of test cases
should expose as many errors as
possible.
82
Summary
If we select test cases
randomly:
many of the selected test cases
do not add to the significance
of the test set.

83
Summary
There are two approaches to
testing:
black-box testing and
white-box testing.

84
Summary
Designing test cases for black box
testing:
does not require any knowledge of
how the functions have been
designed and implemented.
Test cases can be designed by
examining only SRS document.
85
Summary
White box testing:
requires knowledge about
internals of the software.
Design and code is required.

86
Summary
We have discussed a few white-
box test strategies.
Statement coverage
branch coverage
condition coverage
path coverage
87
Summary
A stronger testing strategy:
provides more number of
significant test cases than a
weaker one.
Condition coverage is strongest
among strategies we discussed.
88
Summary
We discussed McCabe’s
Cyclomatic complexity metric:
provides an upper bound for
linearly independent paths
correlates with understanding,
testing, and debugging difficulty of
a program.
89