Lecture 9: Test Adequacy and

Structural Testing
Gregory Gay
DIT635 - February 16, 2022
We Will Cover
• Test Adequacy Criteria
• Structural Testing:
• Use structural coverage to judge tests, create new tests.
• Statement, Branch, Condition, Path Coverage

2
Every developer must answer:
Are our tests are any good?

More importantly… Are they good

enough to stop writing new tests?

3
Have We Done a Good Job?
What we want:
• We’ve found all the faults.
What we (usually) get:
• We compiled and it worked.
• We run out of time or budget.
• Inadequate.

4
Test Adequacy Criteria
Can we compromise between
the impossible and the inadequate?

• Measure “good testing”

• Test adequacy criteria “score” tests by measuring
completion of test obligations.
• Checklists of properties that must be met by test cases.

5
(In)Adequacy Criteria
• Criteria identify inadequacies in the tests.
• If no test reaches a statement, test suite is inadequate for
finding faults in that statement.
• If we plant a fake fault and no test exposes it, our tests
are inadequate at detecting that fault.
• Tests may still miss faults, but maximizing criteria
shows that tests at least meet certain goals.

6
Adequacy Criteria
• Adequacy Criteria based on coverage of factors
correlated to finding faults (hopefully).
• Exercising elements of source code (structural testing).
• Detection of planted faults (fault-based testing)
• Widely used in industry - easy to understand, cheap
to calculate, offer a checklist.
• Enable tracking of “testing completion”
• Can be measured in IntelliJ, Eclipse, etc.

7
Use of Criteria Test Inputs

• Measure adequacy of existing tests

• Create additional tests to cover missed
obligations.
• Create tests directly
• Choose specific obligations, create
tests to cover those. Uncovered
Goals
• Targets for automated test generation.
New Test
Inputs

8
Structural Testing

9
Structural Testing
• The structure of software is int[] flipSome(int[] A, int N, int X)
{
valuable information. int i=0;
while (i<N and A[i] <X)
• Prescribe how code elements {
if (A[i] < 0)
should be executed, and measure A[i] = - A[i];
coverage of execution. i++;
}
• If-statements, Boolean expressions, return A;
}
loops, switches, paths between
statements...

10
The basic idea:
You can’t find all of the
faults without exercising all
of the code.
11
Structural Testing - Motivation
• Requirements-based tests should execute most
code, but will rarely execute all of it.
• Helper functions.
• Error-handling code.
• Requirements missing outcomes.
• Structural testing compliments functional testing by
covering gaps in the source code.

12
Structural Does Not Replace Functional
• Should not be the basis for all test cases
• Harder to make verification argument.
• May not map directly to requirements.
• Does not expose missing functionality.
• Useful for supplementing functional tests.
• Functional tests good at exposing conceptual faults.
• Structural tests good at exposing coding mistakes.

13
Control and Data Flow
• We need to understand how system executes.
• Conditional statements result in branches in execution,
jumping between blocks of code.
• Control flow: how control passes through code.
• Which code is executed, and when.
• Data flow: how data passes through code.
• How variables are used in different expressions.

14
Control-Flow Graphs
• Directed graph representing
flow of control. i=0
• Nodes represent blocks of
sequential statements. i<N
• Edges connect nodes in the True
False A[i]<0
sequence they are executed. True
• Multiple edges indicate False A[i] = - A[i];
conditional statements. return(1)
i++

15
Control Flow: If-then-else
1 if (1==x) {
2 y=45; 1==x

3 } T F

4 else {
y=45; y=23456;
5 y=23456;
6 }
7 /* continue */ /* continue */

16
Loop
1 while (1<x) {
2 x--; 1<x
3 } T F
4 /* continue */
/* continue */
x--;

17
Case

test
1 switch (test) {
2 case 1 : ...
3 case 2 : ... case 1... case 2... case 3...
4 case 3 : ...
5 }
6 /* continue */ /* continue */

18
Basic Blocks
for(int i=0; i < 10; i++){
• Nodes are basic blocks. sum += i;
• Set of sequential }
instructions with single
int i = 0;
entry and exit point.
• Typically adjacent i < 10
statements, but one T F
sum += i;
statement might be broken i++;
up to model control flow in
the statement.
19
 Control Flow Graph Example
1. public static String collapseNewlines(String argSt){ 1-3
2. char last = argStr.charAt(0);
3. StringBuffer argBuf = new StringBuffer();
4. int cldx = 0;
5. for(int cldx = 0; cldx < argStr.length(); cldx++){
6. char ch = argStr.charAt(cldx);
cldx <
7. if (ch != ‘\n’ || last != ‘\n’){ argStr.length();
8. argBuf.append(ch);
9. last = ch; F T
10. { 13 6
11. }
12.
13. return argBuf.toString();
7
F
14. } T
8-9 cldx++;

20
Structural Coverage Criteria
• Criteria based on exercising:
• Statements (nodes of CFG)
• Branches (edges of CFG)
• Conditions
• Paths
• … and many more
• Measurements used as adequacy criteria

21
Statement Coverage
• Most intuitive criteria. Did we execute every
statement at least once?
• Cover each node of the CFG.
• The idea: a fault in a statement cannot be revealed
unless we execute the statement.
• Coverage = Number of Statements Covered
Number of Total Statements

22
Statement Coverage
int[] flipSome(int[] A, int N, int X) i=0
{
int i=0;
while (i<N and A[i] <X) i<N and A[i] <X
{ True
if (A[i] < 0) False A[i]<0
A[i] = - A[i]; True
i++; False A[i] = - A[i];
}
return A; return A
} i++

Can cover in one test: [-1], 1, 10

23
A Note on Test Suite Size
• Coverage not correlated to test suite size.
• Coverage depends on whether obligations are met.
• Some tests might not cover new code.
• However, larger suites often find more faults.
• They exercise the code more thoroughly.
• How code is executed often more important than
whether it was executed.

24
Test Suite Size
• Favor large number of targeted tests over small
number of tests that cover many statements.
• If a test targets a small number of obligations, it is easier
to tell where a fault is.
• If a test covers a large number of obligations, we get
higher coverage, but at the cost of being able to identify
and fix faults.
• The exception - cost to execute each test is high.

25
Branch Coverage
• Do we have tests that take all of the control
branches at some point?
• Cover each edge of the CFG.
• Helps identify faults in decision statements.
• Coverage = Number of Branches Covered
Number of Total Branches

26
Subsumption
• Criterion A subsumes Criterion B if, for every
program P, every test suite satisfying A also
satisfies B on P.
• If we satisfy A, there is no point in measuring B.
• Branch coverage subsumes statement coverage.
• Covering all edges in CFG requires covering all nodes.

27
Subsumption
• Shouldn’t we always choose the stronger metric?
• Not always…
• Typically require more obligations.
• (so, you have to come up with more tests)
• Or, at least, tougher obligations.
• (making it harder to come up with the test cases).
• May end up with unsatisfiable obligations.

28
Branch Coverage
int[] flipSome(int[] A, int N, int X) i=0
{
int i=0;
while (i<N and A[i] <X) i<N and A[i] <X
{ True
if (A[i] < 0) False A[i]<0
A[i] = - A[i]; True
i++; False A[i] = - A[i];
}
return A; return A
} i++

● ([-1], 1, 10) leaves one edge uncovered.

● ([-1, 1], 2, 10) achieves Branch Coverage.
29
Let’s take a break.

30
Decisions and Conditions
• A decision is a complex Boolean expression.
• Often cause control-flow branching:
• if ((a && b) || !c) { ...
• But not always:
• Boolean x = ((a && b) || !c);

31
Decisions and Conditions
• A decision is a complex Boolean expression.
• Made up of conditions
• Connected with Boolean operators (and, or, xor, not):
• Boolean variables: Boolean b = false;
• Subexpressions that evaluate to true/false involving (<, >, <=, >=,
==, and !=): Boolean x = (y < 12);

32
Decision Coverage
• Branch Coverage deals with a subset of decisions.
• Branching decisions that decide how control is routed
through the program.
• Decision coverage requires that all boolean
decisions evaluate to true and false.
• Coverage = Number of Decisions Covered
Number of Total Decisions

33
Basic Condition Coverage
• Several coverage metrics examine the individual
conditions that make up a decision.
• Identify faults in decision statements.
(a == 1 || b == -1) instead of (a == -1 || b == -1)

• Most basic form: make each condition T/F.

• Coverage = Number of Truth Values for All Conditions
2x Number of Conditions
34
Basic Condition Coverage
• Make each condition both True and False
Test Case A B
(A and B) 1 True False
2 False True

• Does not require hitting both branches.

• Does not subsume branch coverage.
• In this case, false branch is taken for both tests

35
Basic Condition Coverage
int[] flipSome(int[] A, int N, int X) i=0
{
int i=0;
while (i<N and A[i] <X) i<N and A[i] <X
{ True
if (A[i] < 0) False A[i]<0
A[i] = - A[i]; True
i++; False A[i] = - A[i];
}
return A; return A
} i++
● ([-1, 1], 2, 10)
○ Negative value in array
○ Positive value (but < X)
● ([11], 1, 10)
○ Positive, but > X
● Both eventually cause i < N to be false.
36
Compound Condition Coverage
• Evaluate every combination of the conditions
Test Case A B

1 True True

(A and B) 2 True False

3 False True
4 False False

• Subsumes branch coverage.

• All outcomes are now tried.
• Can be expensive in practice.
37
Compound Condition Coverage
• Requires many test cases.
Test Case A B C D
1 True True True True
2 True True True False
3 True True False True

(A and 4
5
True
True
True
False
False
True
False
True
6 True False True False

(B and 7
8
True
True
False
False
False
False
True
False
9 False True True True

(C and 10
11
False
False
True
True
True
False
False
True
12 False True False False

D)))) 13
14
False
False
False
False
True
True
True
False
15 False False False True
16 False False False False

38
Short-Circuit Evaluation
• In many languages, if the first condition determines
the result of the entire decision, then fewer tests are
required.
• If A is false, B is never evaluated.
Test Case A B

1 True True
(A and B) 2 True False
3 False -

39
Modified Condition/Decision Coverage(MC/DC)
• Requires:
• Each condition evaluates to true/false
• Each decision evaluates to true/false
• Each condition shown to independently affect outcome
of each decision it appears in.
Test Case A B (A and B)
1 True True True
2 True False False
3 False True False
4 False False False

40
Activity https://bit.ly/34OYvjq

Draw the CFG and write tests that provide statement, branch,
and basic condition coverage over the following code:
public int search(String[] A, String what){
int index = 0;
if ((A.length == 1) && (A[0] == what)){
return 0;
} else if (A.length == 0){
return -1;
} else if (A.length > 1){
while(index < A.length){
if (A[index] == what){
return index;
} else
index++;
}
}
}
return -1;
}
41
Activity - Control Flow Graph
index=0

False False
A.length
False A.length
(A.length ==1)
==0 >1 return -1;
&& (A[0] = what)
True False
True index <
True
A.length
return 0; return -1; True
A[index] False
== what
index++;

True return index;

42
 Activity - Possible Solution
index=0

(N==1) && False False False

(A[0] = what)
N==0 N>1 return -1;
True False
True index
True
<N
return 0; return -1; True
A[index] False
== what
index++;
1: A[“Bob”, “Jane”], 2, “Jane”
2: A[“Bob”, “Jane”], 2, “Spot” return index;
True
3: A[], 0, “Bob”
4. A[“Bob”], 1, “Bob”
5. A[“Bob”], 1, “Spot”

43
Loop Boundary Coverage
• Focus on problems related to loops.
• Cover scenarios representative of how loops might be
executed.
• For each loop, write tests that:
• Skip the loop entirely.
• Take exactly one pass through the loop.
• Take two or more passes through the loop.

44
Nested Loops
• Often, loops are nested within other loops.
• For each level, execute 0, 1, 2+ times
• In addition:
• Test innermost loop first with outer loops executed
minimum number of times.
• Move one loops out, keep the inner loop at “typical”
iteration numbers, and test this layer as you did the
previous layer.
• Continue until the outermost loop tested.

45
Concatenated Loops
• One loop executes. Next line of code starts a new
loop. These are generally independent.
• If not, follow a similar strategy to nested loops.
• Start with bottom loop, hold higher loops at minimal iteration
numbers.
• Work up towards the top, holding lower loops at “typical”
iteration numbers.

46
Why These Loop Strategies?
• If proving correctness, we establish preconditions,
postconditions, and invariants that are true on each
execution of loop.
• The loop executes zero times when the postconditions
are true in advance.
• The loop invariant is true on loop entry (one), then each
loop iteration maintains the invariant (many).
• (invariant and !(loop condition) implies postconditions are met)

• Loop testing strategies echo these cases.

47
Activity: Binary Search https://bit.ly/34OYvjq

For the binary-search code:

1. Draw the control-flow graph for the method.
2. Develop a test suite that achieves loop boundary
coverage (executes while loop 0, 1, 2+ times.

48
Activity: Binary Search
1-5 Tests that execute the loop:
● 0 times key = 1, T = [1]
● 1 time key = 2, T = [1, 2]
7 ● 2+ times key = 3, T = [1, 2, 3]

T F

8 9
F
25 13
F T 19
T
18
14 15 F
21

16
T
49
The Infeasibility Problem
Sometimes, no test can satisfy an obligation.
• Impossible combinations of conditions.
• Unreachable statements as part of defensive
programming.
• Error-handling code for conditions that can’t occur.
• Dead code in legacy applications.

50
The Infeasibility Problem
• Stronger criteria call for potentially infeasible
combinations of elements.

(a > 0 && a < 10)

• It is not possible for both conditions to be false.

• A is negative and greater than 10
• Loop boundary coverage - loop can’t be skipped.

51
The Infeasibility Problem
• Adequacy “scores” based on coverage.
• 95% branch coverage, 80% MC/DC coverage, etc.
• Stop once a threshold is reached.
• Unsatisfactory solution - elements are not equally
important for fault-finding.
• Manual justification for omitting each impossible
test obligation.
• Helps refine code and testing efforts.
• … but very time-consuming.
52
Which Coverage Criterion Should I Use?

Loop Boundary Testing MC/DC Coverage

Branch and Condition

Coverage

Basic Condition
Branch Coverage
Coverage
Power,
Cost Statement Coverage

53
We Have Learned
• Test adequacy metrics “measure” how good our
tests are.
• Prescribe test obligations that remove inadequacies from
test suites.
• Code structure is used in many adequacy criteria.
• Criteria, based on statements, branches,
conditions, loops, etc.

54
Next Time
• Next class: Structural coverage and data-flow
• Optional Reading - Pezze and Young, Chapters 6 and 13
• Friday Exercise Session: Structural Coverage
• Using Meeting Planner code

• Homework - Assignment 2 due Feb 27

55