Software Testing

1
How do you test a
system?
 Input test data to the
system.
 Observe the output:
 Check if the system behaved
as expected.

2
How do you test a
system?

3
How do you test a
system?
 If the program does not
behave as expected:
 note the conditions under
which it failed.
 later debug and correct.

4
Errors and Failures
 A failure is a manifestation
of an error (aka defect or
bug).
 presence of an error may not
lead to a failure.

5
Test cases and Test
suite
 Test a software using a set of
carefully designed test cases:
 the set of all test cases is
called the test suite

6
Test cases and Test
suite
 A test case is a triplet [I,S,O]:
 I is the data to be input to the
system,
 S is the state of the system at
which the data is input,
 O is the expected output from the
system.

7
Verification versus
Validation
 Verification is the process of
determining:
 whether output of one phase of development
conforms to its previous phase.
 Validation is the process of determining
 whether a fully developed system
conforms to its SRS document.

8
Verification versus
Validation
 Aim of Verification:
 phase containment of errors
 Aim of validation:
 final product is error free.

9
Verification versus
Validation
 Verification:
 are we doing right?
 Validation:
 have we done right?

10
Design of Test Cases
 Exhaustive testing of any
system is impractical:
 input data domain is extremely large.
 Design an optimal test suite:
 of reasonable size
 to uncover as many errors as
possible.

11
Design of Test Cases
 If test cases are selected randomly:
 many test cases do not contribute to the
significance of the test suite,
 do not detect errors not already detected
by other test cases in the suite.
 The number of test cases in a randomly
selected test suite:
 not an indication of the effectiveness of the
testing.

12
Design of Test Cases
 Testing a system using a large number
of randomly selected test cases:
 does not mean that many errors in the
system will be uncovered.
 Consider an example:
 finding the maximum of two integers x
and y.

13
Design of Test Cases
 If (x>y) max = x;
else max = x;
 The code has a simple error:
 test suite {(x=3,y=2);(x=2,y=3)} can
detect the error,
 a larger test suite {(x=3,y=2);
(x=4,y=3); (x=5,y=1)} does not
detect the error.
14
Design of Test Cases

 Systematic approaches are

required to design an optimal
test suite:
 each test case in the suite
should detect different errors.

15
Design of Test Cases
 Two main approaches to
design test cases:
 Black-box approach
 White-box (or glass-box)
approach

16
Black-box Testing
 Test cases are designed using only
functional specification of the
software:
 without any knowledge of the internal
structure of the software.
 For this reason, black-box testing is
also known as functional testing.

17
White-box Testing
 Designing white-box test
cases:
 requires knowledge about the
internal structure of software.
 white-box testing is also
called structural testing.

18
Black-box Testing
 Two main approaches to
design black box test cases:
 Equivalence class partitioning
 Boundary value analysis

19
White-Box Testing
 There exist several popular white-box
testing methodologies:
 Statement coverage
 branch coverage
 path coverage
 condition coverage
 mutation testing-not in syllabus
 data flow-based testing-not in syllabus
20
Statement Coverage
 Statement coverage
methodology:
 design test cases so that
 every statement in a
program is executed at least
once.

21
Statement Coverage
 The principal idea:
 unless a statement is executed,
 we have no way of knowing if
an error exists in that
statement.

22
Example
 int f1(int x, int y){
 1 while (x != y){
 2 if (x>y) then
 3 x=x-y;
 4 else y=y-x;
 5}
 6 return x; }

Test suit is: (3,3), (4,3) and (3,4) which will

do 100% statement coverage 23
Branch Coverage
 Test cases are designed
such that:
 different branch conditions
 given true and false values in
turn.

24
Branch Coverage
 Branch testing guarantees
statement coverage:
 a stronger testing compared
to the statement coverage-
based testing.

25
Example
 int f1(int x,int y){
 1 while (x != y){
 2 if (x>y) then
 3 x=x-y;
 4 else y=y-x;
 5}
 6 return x; }

26
Example
 Test cases for branch
coverage can be:
 {(x=3,y=3),(x=3,y=2),
(x=4,y=3), (x=3,y=4)}

27
Condition Coverage
 Test cases are designed such
that:
 each component of a composite
conditional expression
 given both true and false values.

28
Example
 Consider the conditional
expression
 ((c1.and.c2).or.c3):
 Each of c1, c2, and c3 are
exercised at least once,
 i.e. given true and false values.

29
Branch testing
 Condition testing
 stronger testing than branch
testing:
 Branch testing
 stronger than statement
coverage testing.

30
Condition coverage

 Consider a boolean
expression having n
components:
 for condition coverage we
n
require 2 test cases. (n is no.
of condtions)
31
Path Coverage
 Design test cases such that:
 all linearly independent
paths in the program are
executed at least once.

32
Linearly independent
paths
 Defined in terms of
 control flow graph (CFG) of a
program.

33
Path coverage-based
testing
 To understand the path
coverage-based testing:
 we need to learn how to draw
control flow graph of a
program.

34
Control flow graph
(CFG)
 A control flow graph (CFG)
describes:
 the sequence in which different
instructions of a program get
executed.
 the way control flows through
the program.
35
How to draw Control
flow graph?
 Number all the statements of
a program.
 Numbered statements:
 represent nodes of the control
flow graph.

36
How to draw Control
flow graph?

 Sequence: 1
 1 a=5;
2
 2 b=a*b-1;

37
How to draw Control
flow graph?

 Selection:
 1 if(a>b) then
2 c=3;
 3 else c=5;
 4 c=c*c;

38
39
How to draw Control
flow graph?

 Iteration:
 1 while(a>b){
2 b=b*a;
3 b=b-1;}
 4 c=b+d;

40
41
Example
 int f1(int x,int y){
 1 while (x != y){
 2 if (x>y) then
 3 x=x-y;
 4 else y=y-x;
 5}
 6 return x; }

42
Example Control Flow
Graph
1
2
3 4
5
6

43
Path
 A path through a program:
 a node and edge sequence from
the starting node to a terminal
node of the control flow graph.
 There may be several terminal
nodes for program.
44
Independent path
 It is straight forward:
 to identify linearly independent
paths of simple programs.
 For complicated programs:
 it is not so easy to determine the
number of independent paths.

45
McCabe's cyclomatic
metric
 Given a control flow graph G,
cyclomatic complexity V(G):
 V(G)= E-N+2
 N is the number of nodes in G
 E is the number of edges in G

46
Example Control Flow
Graph
1
2
3 4
5
6

47
Example
 Cyclomatic complexity =
7-6+2 = 3.

48
Cyclomatic complexity
 Another way of computing cyclomatic
complexity:
 inspect control flow graph
 determine number of bounded areas in
the graph
 V(G) = Total number of bounded
areas + 1
49
Bounded area
 Any region enclosed by a
nodes and edge sequence.

50
Example Control Flow
Graph
1
2
3 4
5
6

51
Example
 From a visual examination of
the CFG:
 the number of bounded areas is
2.
 cyclomatic complexity =
2+1=3.
52
Cyclomatic complexity
 McCabe's metric provides:
 a quantitative measure of testing
difficulty and the ultimate reliability
 Intuitively,
 number of bounded areas increases
with the number of decision nodes
and loops.

53
Cyclomatic complexity

 The first method of computing

V(G) is amenable to automation:
 you can write a program which
determines the number of nodes
and edges of a graph
 applies the formula to find V(G).

54
Cyclomatic complexity

 The cyclomatic complexity of a

program provides:
 a lower bound on the number of
test cases to be designed
 to guarantee coverage of all
linearly independent paths.

55
Cyclomatic complexity

 Defines the number of

independent paths in a
program.
 Provides a lower bound:
 for the number of test cases for
path coverage.
56
Cyclomatic complexity

 Knowing the number of test cases

required:
 does not make it any easier to
derive the test cases,
 only gives an indication of the
minimum number of test cases
required.
57
Derivation of Test
Cases
 Draw control flow graph.
 Determine V(G).
 Determine the set of linearly
independent paths.
 Prepare test cases:
 to force execution along each path.
58
59
60
Example
 int f1(int x,int y){
 1 while (x != y){
 2 if (x>y) then
 3 x=x-y;
 4 else y=y-x;
 5}
 6 return x; }

61
Example Control Flow
Diagram
1
2
3 4
5
6

62
Derivation of Test Cases

 Number of independent paths:

3
 1,6 test case (x=1, y=1)
 1,2,3,5,1,6 test case(x=2, y=1)
 1,2,4,5,1,6 test case(x=1, y=2)

63
An interesting application
of cyclomatic complexity

 Relationship exists between:

 McCabe's metric
 the number of errors existing in
the code,
 the time required to find and
correct the errors.
64
Cyclomatic complexity

 Cyclomatic complexity of a
program:
 also indicates the psychological
complexity of a program.
 difficulty level of understanding
the program.
65
Cyclomatic complexity
 From maintenance perspective,
 limit cyclomatic complexity
 of modules to some reasonable value.
 Good software development
organizations:
 restrict cyclomatic complexity of
functions to a maximum of ten or so.

66
Software Reliability
 It is truth-worthiness or dependability.
 Defined as probability of the product working
correctly over a given period of time
 It also depends not only number of errors but
also on the exact location of errors.
 It also depends on execution profile.
 It keep changing as errors are detected an fixed

67
Reliability Metrics

1. Rate of occurrence of failure(ROCOF):

frequency of occurrence of failure
2. Mean time to failure(MTTF)
3. Mean time to repair(MTTR)
4. Mean time between failure (MTBF)
5. Probability of failure on demand (POFOD)
6. Availability

68
Types of failure

1. Transient: only for certain input values while

invoking a function
2. Permanent: for all inputs values while invoking
a function
3. Recoverable: when it occurs, system can
recover without having to shutdown the
system
4. Unrecoverable: need to restart the system

69