Coverage Criteria

Dr. Hager Hussein

1
 Coverage Criteria for Testing
Coverage criterion: A collection of rules and
a process that define test requirements
̶ Cover every statement
̶ Cover every functional requirement
Coverage criteria give structured, practical
ways to search the input space
Search the input space thoroughly
Not much overlap in the tests

2
 Characteristics of a Good
Coverage Criterion
1. Supports in covering test requirements.
2. Supports in the difficulty of generating
tests.
3. Shows how well the tests reveal faults.

3
 Advantages of Coverage
Criteria
Make regression testing easier
Gives testers a “stopping rule” … when
testing is finished
Can be well supported with powerful
tools

4
 Types of Test Activities
Testing can be broken up into four general types of
activities
1. Test Design
2. Test Automation
3. Test Execution
4. Test Evaluation
Each type of activity requires different skills,
background knowledge, education and training
No reasonable software development organization uses
the same people for requirements, design,
implementation, integration and configuration control
Why don’t test organizations still use the same people
for all four test activities?? This clearly wastes
resources.

5
 1.Test Design—(a) Criteria-
Based
Design test values to satisfy coverage criteria
or other engineering goal
This is the most technical job in software
testing
Requires knowledge of :
Discrete math
Programming
Testing
Requires much of a traditional CS degree
Using people who are not qualified to design
tests is a sure way to get ineffective tests

6
 1.Test Design—(b) Human-
Based
Design test values based on domain knowledge
of the program and human knowledge of
testing
This is much harder than it may seem to
developers
Requires knowledge of :
Domain, testing, and user interfaces
Requires almost no traditional CS
A background in the domain of the software is
essential
An empirical background is very helpful
(biology, psychology, …)
A logic background is very helpful (law,
philosophy, math, …)
7
 2. Test Automation
Embed test values into executable scripts
This is slightly less technical
Requires knowledge of programming
Requires very little theory
Can be boring for test designers
Who is responsible for determining and
embedding the expected outputs ?
Test designers may not always know the
expected outputs
Test evaluators need to get involved early to
help with this

8
 3. Test Execution
Run tests on the software and record the results
This is easy – and trivial if the tests are well
automated
Requires basic computer skills

9
 4. Test Evaluation
Evaluate results of testing, report to developers
This is much harder than it may seem
Requires knowledge of : Domain, Testing, User
interfaces, and psychology.
Usually requires almost no traditional CS
A background in the domain of the software is
essential
An empirical background is very helpful
(biology, psychology, …)
A logic background is very helpful (law,
philosophy, math, …)
10
 Other Activities
 Test management : Sets policy, organizes team,
interfaces with development, chooses criteria, decides
how much automation is needed, …
 Test maintenance : Save tests for reuse as software
evolves
Requires cooperation of test designers and automators
Deciding when to trim the test suite is partly policy and
partly technical – and in general, very hard !
 Test documentation : All parties participate
Each test must document “why” – criterion and test
requirement satisfied or a rationale for human-designed
tests
Ensure traceability throughout the process
Keep documentation in the automated tests

11
 Test Requirement
Test requirement (TR): Specific things that
must be satisfied or covered during testing
Each statement is a test requirement
If the goal is to cover all decisions in the
program (branch coverage), then each
decision leads to two test requirements,
one for the decision to evaluate to false,
and one for the decision to evaluate to
true.
If every method must be called at least
once (call coverage), each method leads to
one test requirement.
12
 Testing researchers have defined dozens of
criteria, but they are all really just a few
criteria on four types of structures …
1. Input domains
2. Graphs
3. Logic expressions
4. Syntax descriptions

13
 Coverage
 TR={flavor=Lemon, flavor=Pistachio, flavor=Cantaloupe,
flavor=Pear, flavor=Tangerine, flavor=Apricot}.
 Coverage: Given a set of test requirements TR for a
coverage criterion C, a test set T satisfies C if and only if
for every test requirement tr in TR, at least one test t in T
exists such that t satisfies tr.
 For the following reasons, we try to achieve a certain
coverage level.
1. It is sometimes expensive to satisfy a coverage criterion.
2. Some requirements that cannot be satisfied that is called
“infeasible”.
 Coverage level: given a set of test requirements TR and a
test set T, the coverage level is simply the ratio of the
number of test requirements satisfied by T to the size of
TR.
14
 A generator: is a procedure that
automatically generates values to satisfy a
criterion.
A recognizer: is a procedure that decides
whether a given set of test case values
satisfies a criterion.
In practical terms of commercial automated
test tools, a generator corresponds to a tool
that automatically creates test case values.
A recognizer is a coverage analysis tool.

15
 Criteria Subsumption
 Criteria Subsumption: A coverage criterion C1
subsumes C2 if and only if every test set that satisfies
criterion C1 also satisfies C2.
 Generally, a criterion C1 can subsume another C2 in
one of two ways. The simpler way is if the test
requirements for C1 always form a superset of the
requirements for C2.
 For example, given a requirement for the flavor
criterion: {Lemon, Pistachio, Cantaloupe, Pear,
Tangerine, Apricot}. If we choose a requirement of a
flavor that begins with the letter ‘C’, that would result in
the test requirements {Cantaloupe}, which is a subset
of the requirements for the flavor criterion. Thus, the
flavor criterion subsumes the “starts-with-C” criterion.

16
 Subsumption Example
If a test set has covered every branch in a
program (satisfied branch coverage), then
the test set is guaranteed to have covered
every statement as well.
Thus, the branch coverage criterion
subsumes the statement coverage
criterion.

17
 Older Software Testing
Terminology
We present testing as proceeding from
abstract models of the software such as
graphs, which can as easily be derived from
a black-box view or a white-box view.
Black box testing: Deriving tests from
external descriptions of the software,
including specifications, requirements, and
design.
White box testing: Deriving tests from the
source code internals of the software,
specifically including branches, individual
conditions, and statements.
18
 The following pair of definitions assumes
that software can be viewed as a tree of
software procedures, where the edges
represent calls and the root of the tree is
the main procedure.
Top-Down Testing: Test the main procedure,
then go down through procedures it calls,
and so on.
Bottom-Up Testing: Test the leaves in the
tree (procedures that make no calls), and
move up to the root. Each procedure is
tested only if all of its children have been
19
tested.
 Dynamic and Static Testing
 OO software leads to a more general problem. The
relationships among classes can be formulated as
general graphs with cycles, requiring test engineers to
make the difficult choice of what order to test the
classes in.
 Some parts of the literature separate static and
dynamic testing as follows:
 Static testing: testing without executing the program.
This includes some forms of analysis.
 Dynamic testing: testing by executing the program
with real inputs.
 Most of the literature currently uses “testing” to refer
to dynamic testing and “static testing” is called
“verification activities.” .
20
 CHAPTER 2
GRAPH COVERAGE
2.1 AND 2.2

21
 Introduction
 Directed graphs form the foundation for many
coverage criteria.
 Given an artifact under test, the idea is to obtain
a graph abstraction of that artifact.
 Basic notion: A graph G formally is:
a set N of nodes
a set N0 of initial nodes, where N 0 ⊆ N
a set Nf of final nodes, where N f ⊆ N
a set E of edges, where E is a subset of N × N
 A subgraph of a graph is also a graph and is
defined by a subset of N, along with the
corresponding subsets of N 0, Nf , and E.
22
 if Nsub is a subset of N, then for the
subgraph defined by Nsub, the set of initial
nodes is N0sub ∩ N0, the set of final nodes is
Nfsub ∩ Nf , and the set of edges is (N sub ×
Nsub) ∩ E.

23
  Edges are considered to be from one node and to
another and written as (Ni , Nj).
 The edge’s initial node Ni is sometimes called the
predecessor and Nj is called the successor.
 We always identify final nodes, and there must be at
least one final node. The reason is that every test must
start in some initial node and end in some final node.
 Graph theory texts sometimes call a node as a vertex,
and testing texts typically identify a node with the
structure it represents, often a statement or a basic
block. Similarly, graph theory texts sometimes call an
edge an arc, and testing texts typically identify an
edge with the structure it represents, often a branch.

24
25
 Test path: A path p, possibly of length zero,
that starts at some node in N0 and ends at
some node in Nf .
A path is a sequence [n 1, n2,..., nM] of
nodes, where each pair of adjacent nodes,
(ni, ni+1), 1 ≤ i < M, is in the set E of
edges.
The length of a path is defined as the
number of edges it contains.
We sometimes consider paths and
subpaths of length zero.
26
A subpath of a path p is a subsequence of p
 Tests and Test Paths
test 1 many-to-one
Test
test 2
Path
test 3
Deterministic software – a test always executes the same test path

many-to-many
test 1 Test Path 1

test 2 Test Path 2

test 3 Test Path 3

Non-deterministic software – a test can execute different test paths
27
28
 Structural Coverage Criteria
We specify graph coverage criteria by
specifying a set of test requirements, TR.
We will start by defining criteria to visit
every node and then every edge in a
graph.
The first criterion is probably familiar and is
based on the old notion of executing every
statement in a program. This concept has
variously been called “statement
coverage,” “block coverage,” “state
coverage,” and “node coverage.” We use
the general graph term “node coverage.”
29
 Node Coverage (NC) (Standard Definition):
Test set T satisfies node coverage on graph
G if and only if for every syntactically
reachable node n in N, there is some path p
in path(T) such that p visits n.
Edge Coverage (EC): TR contains each
reachable path of length up to 1, inclusive,
in G.
Edge-Pair Coverage (EPC): TR contains
each reachable path of length up to 2,
inclusive, in G.
30
 Node Coverage
 Node Coverage (NC): T R contains each reachable node
in G.
 This simplification allows us to simplify the
writing of the test requirements for Figure 2.3
to only contain the nodes:
TR = {n0, n1, n2, n3, n4, n5, n6}.
Test path p1 = [n0, n1, n3, n4, n6] meets the first,
second, fourth, fifth, and seventh test requirements,
and test path p2 = [n0, n2, n3, n5, n6] meets the
first, third, fourth, sixth, and seventh. Therefore,
if a test set T contains {t1, t2}, where path(t1) = p1
and path(t2) = p2, then T satisfies node coverage on G.

31
 Example
Node Coverage
TR = { 0, 1, 2, 3, 4, 5, 6 }
Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 5, 4, 6 ]
0
Edge Coverage
TR = { (0,1), (0,2), (1,2), (2,3), (2,4), (3,6), (4,5), (4,6),
1 (5,4) }
Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 2, 4, 5, 4, 6 ]
2 Edge-Pair Coverage
TR = { [0,1,2], [0,2,3], [0,2,4], [1,2,3], [1,2,4], [2,3,6],
[2,4,5], [2,4,6], [4,5,4], [5,4,5], [5,4,6] }
3 4 Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 6 ] [ 0, 2, 3, 6 ]
[ 0, 2, 4, 5, 4, 5, 4, 6 ]
5
6 Complete Path Coverage
Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 6 ] [ 0, 1, 2, 4, 5, 4, 6 ]
[ 0, 1, 2, 4, 5, 4, 5, 4, 6 ] [ 0, 1, 2, 4, 5, 4, 5, 4, 5, 4, 6 ] …

32
33
 Simple Paths and Prime Paths
Simple Path : A path from node ni to nj is simple if
no node appears more than once, except possibly
the first and last nodes are the same
 No internal loops
 A loop is a simple path

Prime Path : A simple path that does not appear

Simple Paths : [ 0, 1, 3, 0 ], [ 0, 2, 3, 0], [ 1, 3, 0, 1 ],
as a proper subpath
[ 2, 3, 0, 2of
], [ any
3, 0, 1,other
3 ], [ 3, 0,simple
2, 3 ], [ 1, path
3, 0, 2 ],
[ 2, 3, 0, 1 ], [ 0, 1, 3 ], [ 0, 2, 3 ], [ 1, 3, 0 ], [ 2, 3, 0 ],
[ 3, 0, 1 ], [3, 0, 2 ], [ 0, 1], [ 0, 2 ], [ 1, 3 ], [ 2, 3 ], [ 3, 0 ],
0 [0], [1], [2], [3]
1 2
Prime Paths : [ 0, 1, 3, 0 ], [ 0, 2, 3, 0], [ 1, 3, 0, 1 ],
3 [ 2, 3, 0, 2 ], [ 3, 0, 1, 3 ], [ 3, 0, 2, 3 ], [ 1, 3, 0, 2 ],
[ 2, 3, 0, 1 ]

34
 Prime Path Coverage
A simple, elegant and finite criterion that requires
loops to be executed as well as skipped

Prime Path Coverage (PPC) : TR contains each prime

path in G.

• Will tour all paths of length 0, 1, …

• That is, it subsumes node and edge coverage
• Note : The book has a mistake, PPC does NOT subsume EPC
• If a node n has an edge to itself, EPC will require [n, n, m]
• [n, n, m] is not prime

35
 Round Trips
Round-Trip Path : A prime path that starts and
ends at the same node

Simple Round Trip Coverage (SRTC) : TR contains at

least one round-trip path for each reachable node in
G that begins and ends a round-trip path.

Complete Round Trip Coverage (CRTC) : TR contains

all round-trip paths for each reachable node in G.

• These criteria omit nodes and edges that are not in round trips
• That is, they do not subsume edge-pair, edge, or node coverage

36
 Prime Path Example
The previous example has 38 simple paths
Only nine prime paths

0
Prime Paths
1 [ 0, 1, 2, 3, 6 ]
[ 0, 1, 2, 4, 5 ] Execute
[ 0, 1, 2, 4, 6 ] loop 0 times
2 [ 0, 2, 3, 6 ]
[ 0, 2, 4, 5] Execute
3 4 [ 0, 2, 4, 6 ] loop once
[ 5, 4, 6 ]
5 [ 4, 5, 4 ] Execute loop
6 [ 5, 4, 5 ] more than once

37
 Touring, Sidetrips and
Detours
Prime paths do not have internal loops … test paths might

• Tour : A test path p tours subpath q if q is a subpath of p

• Tour With Sidetrips : A test path p tours subpath q with sidetrips

iff every edge in q is also in p in the same order
• The tour can include a sidetrip, as long as it comes back to the same node

• Tour With Detours : A test path p tours subpath q with detours iff
every node in q is also in p in the same order
• The tour can include a detour from node ni, as long as it comes back to
the prime path at a successor of ni

38
 Sidetrips and Detours
Example
1 2 3 4
0 1 2 4 5

Touring the prime path

[1, 2, 3, 4, 5] without 3
sidetrips or detours
1 2 5 6
0 1 2 4 5
3 4
Touring with a
sidetrip 3

1 2 5
0 1 2 4 5
3
Touring with a 4
3
detour

39
 Prime Path
Prime Path Coverage (PPC): T R contains
each prime path in G.

40
41
 An Example for Prime Test
Paths

42
 There are 8 prime paths:

43
44
 Data Flow Criteria
 The next few testing criteria are based on the
assumption that to test a program adequately, we should
focus on the flows of data values.
 We should try to ensure that the values created at one
point in the program are created and used correctly.
 This is done by focusing on definitions and uses of
values.
 A definition (def) is a location where a value for a
variable is stored into memory (assignment, input, etc.).
 A use is a location where a variable’s value is accessed.
 Data flow testing criteria use the fact that values are
carried from defs to uses.
 We call these du-pairs (or definition-use, def-use, and
du).
45
 DU paths

46
 Thank You

47