Path Coverage

● Other criteria focus on single elements.

○ However, all tests execute a sequence of elements -
a path through the program.
○ Combination of elements matters - interaction
sequences are the root of many faults.
● Path coverage requires that all paths
through the CFG are covered.
● Coverage = Number of Paths Covered
Number of Total Paths
Gregory Gay CSCE 747 - Spring 2016 3
 Path Coverage
i=0
int flipSome(int A[], int N, int X)
{
int i=0;
while (i<N and A[i] <X) i<N and A[i]
{ <X True
if (A[i]<0) False
A[i] = - A[i]; A[i]<0
True
i++;
False
}
A[i] = - A[i];
return(1);
} return(1)

i++

In theory, path coverage is the ultimate coverage metric.

In practice, it is impractical.
● How many paths does this program have?

Gregory Gay CSCE 747 - Spring 2016 4

 Path Testing
How many cases
for
Statement
Branch
Path

loop <= 20

Gregory Gay CSCE 747 - Spring 2016 5

 Number of Tests

Path coverage for that loop bound requires:

3,656,158,440,062,976 test cases

If you run 1000 tests per second, this will

take 116,000 years.

However, there are ways to get some of the

benefits of path coverage without the cost...
Gregory Gay CSCE 747 - Spring 2016 6
 Path Coverage
● Theoretically, the strongest coverage metric.
○ Many faults emerge through sequences of
interactions.
● But… Generally impossible to achieve.
○ Loops result in an infinite number of path variations.
○ Even bounding number of loop executions leaves an
infeasible number of tests.

Gregory Gay CSCE 747 - Spring 2016 7

 Boundary Interior Coverage
● Need to partition the infinite set of paths into
a finite number of classes.
● Boundary Interior Coverage groups paths
that differ only in the subpath they follow
when repeating the body of a loop.
○ Executing a loop 20 times is a different path than
executing it twice, but the same subsequences of
statements repeat over and over.

Gregory Gay CSCE 747 - Spring 2016 8

 Boundary Interior Coverage
A
A B -> M
B
B

B -> C -> E -> L -> B

M C
M C

D E B -> C -> D -> DF -> L ->E B

L
F G
F G
B -> C -> D -> G -> H ->
B L -> B
L H I
H I

B L L
L
B -> C -> D -> G -> I -> L -> B
B B

Gregory Gay CSCE 747 - Spring 2016 9

 Number of Paths
● Boundary Interior Coverage if (a) S1;
removes the problem of
if (b) S2;
infinite loop-based paths.
● However, the number of if (c) S3;
paths through this code can …
still be exponential. if (x) SN;
○ N non-loop branches results
in 2N paths.
● Additional limitations may
need to be imposed on the
paths tested.

Gregory Gay CSCE 747 - Spring 2016 10

 Loop Boundary Coverage
● Focus on problems related to loops.
● Cover scenarios representative of how loops might
be executed.
● For simple loops, write tests that:
○ Skip the loop entirely.
○ Take exactly one pass through the loop.
○ Take two or more passes through the loop.
○ (optional) Choose an upper bound N, and:
■ M passes, where 2 < M < N
■ (N-1), N, and (N+1) passes

Gregory Gay CSCE 747 - Spring 2016 11

 Nested Loops
● Often, loops are nested within other loops.
● For each level, you should execute similar
strategies to simple loops.
● In addition:
○ Test innermost loop first with outer loops
executed minimum number of times.
○ Move one loops out, keep the inner loop at
“typical” iteration numbers, and test this
layer as you did the previous layer.
○ Continue until the outermost loop tested.

Gregory Gay CSCE 747 - Spring 2016 12

 Concatenated Loops
● One loop executes. The next line of code
starts a new loop.
● These are generally independent.
○ Most of the time...
● If not, follow a similar strategy to nested
loops.
○ Start with bottom loop, hold higher loops
at minimal iteration numbers.
○ Work up towards the top, holding lower
loops at “typical” iteration numbers.

Gregory Gay CSCE 747 - Spring 2016 13

 Why These Loop Strategies?
Why do these loop values make sense?
● In proving formal correctness of a loop, we would establish
preconditions, postconditions, and invariants that are true on
each execution of the loop, then prove that these hold.
○ The loop executes zero times when the postconditions
are true in advance.
○ The loop invariant is true on loop entry (one), then each
loop iteration maintains the invariant (many).
■ (invariant and !(loop condition) implies postconditions)
● Loop testing strategies echo these cases.

Gregory Gay CSCE 747 - Spring 2016 14

 Linear Code Sequences and Jumps
collapseNewlines(String
● Often, we want to reason about the B1 argSt)

subpaths that execution can take. char last = argStr.charAt(0);

StringBuffer argBuf = new
● A subpath from one branch of control B2 StringBuffer();
int cldx = 0;

to another is called a LCSAJ.

cldx < B3
● The LCSAJs for this example: argStr.
length()
F J1 ;
From To Sequence of Basic Blocks
J3
T B4
char ch = argStr.charAt
entry j1 b1, b2, b3 (cldx);

entry j2 b1, b2, b3, b4, b5

B5 (ch != ‘\n’
entry j3 b1, b2, b3, b4, b5, b6, b7
|| last !=
J2
‘\n’)
j1 return b8 F
T
j2 j3 b7 argBuf.append(ch);
cldx++;
last = ch;
j3 j2 b3, b4, b5
B6
B7
return argBuf.toString();
j3 j3 b3, b4, b5, b6, b7

B8

Gregory Gay CSCE 747 - Spring 2016 15

 LCSAJ Coverage
● We can require coverage of all sequences of LCSAJs of
length N.
○ We can string subpaths into paths that connect N subpaths.
○ LCSAJ Coverage (N=1) is equivalent to statement coverage.
○ LCSAJ Coverage (N=2) is equivalent to branch coverage
● Higher values of N achieve stronger levels of path
coverage.
● Can define a threshold that offers stronger tests while
remaining affordable.

Gregory Gay CSCE 747 - Spring 2016 16

 Procedure Call Testing
● Metrics covered to this point all look at code
within a procedure.
● Good for testing individual units of code, but
not well-suited for integration testing.
○ i.e., subsystem or system testing, where we bring
together units of code and test their combination.
● Should also cover connections between
procedures:
○ calls and returns.

Gregory Gay CSCE 747 - Spring 2016 17

 Entry and Exit Testing
● A single procedure may int status (String str){
if(str.equals(”panic”))
have several entry and
return 0;
exit points. else if(str.contains(“+”))
○ In languages with goto return 1;
statements, labels allow else if(str.contains(“-”))
multiple entry points. return 2;
○ Multiple returns mean else
multiple exit points. return 3;

● Write tests to ensure }

these entry/exit points

are entered and exited ● Finds interface errors
in the context they are that statement coverage
intended to be used. would not find.

Gregory Gay CSCE 747 - Spring 2016 18

 Call Coverage
● A procedure might be void orderPizza (String str){
if(str.contains(”pepperoni”))
called from multiple
addTopping(“pepperoni”);
locations. if(str.contains(“onions”))
● Call coverage requires addTopping(“onions”);
that a test suite if(str.contains(“mushroom”))
addTopping(“mushroom”)
executes all possible }
method calls.
● Also finds interface ● Challenging for OO
errors that systems, where a
statement/branch method call might be
coverage would not find. bound to different
objects at runtime.

Gregory Gay CSCE 747 - Spring 2016 19

 Activity:
Writing Loop-Covering Tests
For the binary-search code:
1. Draw the control-flow graph for the method.
2. Identify the subpaths through the loop and
draw the unfolded CFG for boundary interior
testing.
3. Develop a test suite that achieves loop
boundary coverage.

Gregory Gay CSCE 747 - Spring 2016 20

 CFG
int bott, top, mid;
bott=0; top=size-1;
L = 0;
bott<=to F
p && ! EXIT
found

T
T[L] mid=round
== (top+bott/2);
key

T F
T[mid]
== key F
found=true; found=false; T T
bott=mid+1;
T[mid]
< key
found=true;
top=mid-1;
L= mid; F

Gregory Gay CSCE 747 - Spring 2016 21

 CFG

A
E -> EXIT

E -> F -> G -> H -> E

B
E -> F -> G -> I -> J -> E
T F
E -> F -> G -> I -> K -> E

C D
F

EXIT E
F T
T J
I F
F G K

H
T

Gregory Gay CSCE 747 - Spring 2016 22

 CFG

A
E -> EXIT

E -> F -> G -> H -> E

B
E -> F -> G -> I -> J -> E
T F
E -> F -> G -> I -> K -> E

C D
F

EXIT E
F T
T J E
I F
F G K E

H
T E

Gregory Gay CSCE 747 - Spring 2016 23

 CFG
Tests that execute the loop:
A key = 1, T = [1], size = 1
● 0 times
● 1 time key = 2, T = [1, 2], size = 2
● 2+ times key = 3, T = [1, 2, 3], size = 3
B

T F

C D
F

EXIT E
F T
T J
I F
F G K

H
T

Gregory Gay CSCE 747 - Spring 2016 24

 Cyclomatic Testing
● Generally, there are many options for the set
of basis subpaths.
● When testing, count the number of
independent paths that have already been
covered, and add any new subpaths covered
by the new test.
○ You can identify allpaths with a set of independent
subpaths of size = the cyclomatic complexity.

Gregory Gay CSCE 747 - Spring 2016 15

 Uses of Cyclomatic Complexity
● A way to guess “how much testing is enough”.
○ Upper bound on number of tests for branch coverage.
○ Lower bound on number of tests for path coverage.
● Used to refactor code.
○ Components with a complexity > some threshold
should be split into smaller modules.
○ Based on the belief that more complex code is more
fault-prone.

Gregory Gay CSCE 747 - Spring 2016 16