Structural Testing

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 1

 Learning objectives
• Understand rationale for structural testing
– How structural (code-based or glass-box) testing
complements functional (black-box) testing
• Recognize and distinguish basic terms
– Adequacy, coverage
• Recognize and distinguish characteristics of
common structural criteria
• Understand practical uses and limitations of
structural testing

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 2

 “Structural” testing
• Judging test suite thoroughness based on the
structure of the program itself
– Also known as “white-box”, “glass-box”, or “code-
based” testing
– To distinguish from functional (requirements-based,
“black-box” testing)
– “Structural” testing is still testing product functionality
against its specification. Only the measure of thoroughness
has changed.

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 3

 Why structural (code-based) testing?
• One way of answering the question “What is
missing in our test suite?”
– If part of a program is not executed by any test case
in the suite, faults in that part cannot be exposed
– But what’s a “part”?
• Typically, a control flow element or combination:
• Statements (or CFG nodes), Branches (or CFG edges)
• Fragments and combinations: Conditions, paths
• Complements functional testing: Another way
to recognize cases that are treated differently
– Recall fundamental rationale: Prefer test cases that
are treated differently over cases treated the same

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 4

 No guarantees
• Executing all control flow elements does not
guarantee finding all faults
– Execution of a faulty statement may not always
result in a failure
• The state may not be corrupted when the statement is
executed with some data values
• Corrupt state may not propagate through execution to
eventually lead to failure
• What is the value of structural coverage?
– Increases confidence in thoroughness of testing
• Removes some obvious inadequacies

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 5

 Structural testing complements
functional testing
• Control flow testing includes cases that may not
be identified from specifications alone
– Typical case: implementation of a single item of the
specification by multiple parts of the program
– Example: hash table collision (invisible in interface
spec)
• Test suites that satisfy control flow adequacy
criteria could fail in revealing faults that can be
caught with functional criteria
– Typical case: missing path faults

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 6

 Structural testing in practice
• Create functional test suite first, then measure
structural coverage to identify & see what is missing
• Interpret unexecuted elements
– may be due to natural differences between specification and
implementation
– or may reveal flaws of the software or its development process
• inadequacy of specifications that do not include cases present in
the implementation
• coding practice that radically diverges from the specification
• inadequate functional test suites

• Attractive because automated

– coverage measurements are convenient progress indicators
– sometimes used as a criterion of completion
• use with caution: does not ensure effective test suites

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 7

 Statement testing
• Adequacy criterion: each statement (or node in
the CFG) must be executed at least once
• Coverage:
# executed statements
# statements
• Rationale: a fault in a statement can only be
revealed by executing the faulty statement

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 8

 Statements or blocks?
• Nodes in a control flow graph often represent
basic blocks of multiple statements
– Some standards refer to basic block coverage or
node coverage
– Difference in granularity, not in concept
• No essential difference
– 100% node coverage <-> 100% statement coverage
• but levels will differ below 100%
– A test case that improves one will improve the other
• though not by the same amount, in general

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 9

 Example
int cgi_decode(char *encoded, char *decoded)

{char *eptr = encoded; A

char *dptr = decoded;
int ok = 0;
T0 =
while (*eptr) { B
{“”, “test”,
False True
“test+case%1Dadequacy”} char c; C
17/18 = 94% Stmt Cov. c = *eptr;
if (c == '+') {
False True
T1 = D E
elseif (c == '%') { *dptr = ' ';
{“adequate+test }

%0Dexecution%7U”} False True

18/18 = 100% Stmt Cov. else F int digit_high = Hex_Values[*(++eptr)]; G

*dptr = *eptr; int digit_low = Hex_Values[*(++eptr)];
} if (digit_high == -1 || digit_low == -1) {
T2 = False True

{“%3D”, “%A”, “a+b”, else { H ok = 1; I

*dptr = 16 * digit_high + }
“test”} digit_low;
18/18 = 100% Stmt Cov. }

*dptr = '\0'; M ++dptr; L

return ok; ++eptr;
} }

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 10

 Coverage is not size
• Coverage does not depend on the number of
test cases
– T0 , T1 : T1 >coverage T0 T1 <cardinality T0
– T1 , T2 : T2 =coverage T1 T2 >cardinality T1
• Minimizing test suite size is seldom the goal
– small test cases make failure diagnosis easier
– a failing test case in T2 gives more information for
fault localization than a failing test case in T1

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 11

 “All statements” can miss some cases
int cgi_decode(char *encoded, char *decoded)
• Complete statement
{char *eptr = encoded; A
coverage may not imply char *dptr = decoded;
executing all branches in int ok = 0;

a program while (*eptr) { B

• Example: False True

char c; C
– Suppose block F were c = *eptr;
if (c == '+') {
missing False True
– Statement adequacy elseif (c == '%') { D *dptr = ' '; E
}
would not require false False True
branch from D to L else { F int digit_high = Hex_Values[*(++eptr)]; G
*dptr = *eptr; int digit_low = Hex_Values[*(++eptr)];
T3 = } if (digit_high == -1 || digit_low == -1) {
{“”, “+%0D+%4J”} False True
100% Stmt Cov. else { H ok = 1; I
*dptr = 16 * digit_high + }
No false branch from D digit_low;
}

*dptr = '\0'; M ++dptr; L

return ok; ++eptr;
} }

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 12

 Branch testing
• Adequacy criterion: each branch (edge in the
CFG) must be executed at least once
• Coverage:
# executed branches
# branches

T3 = {“”, “+%0D+%4J”}
100% Stmt Cov. 88% Branch Cov. (7/8 branches)

T2 = {“%3D”, “%A”, “a+b”, “test”}

100% Stmt Cov. 100% Branch Cov. (8/8 branches)
(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 13
 Statements vs branches
• Traversing all edges of a graph causes all nodes
to be visited
– So test suites that satisfy the branch adequacy
criterion for a program P also satisfy the statement
adequacy criterion for the same program
• The converse is not true (see T3)
– A statement-adequate (or node-adequate) test suite
may not be branch-adequate (edge-adequate)

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 14

 “All branches” can still miss conditions
• Sample fault: missing operator (negation)
digit_high == 1 || digit_low == -1
• Branch adequacy criterion can be satisfied by
varying only digit_low
– The faulty sub-expression might never determine the
result
– We might never really test the faulty condition, even
though we tested both outcomes of the branch

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 15

 Condition testing
• Branch coverage exposes faults in how a
computation has been decomposed into cases
– intuitively attractive: check the programmer’s case
analysis
– but only roughly: groups cases with the same
outcome
• Condition coverage considers case analysis in
more detail
– also individual conditions in a compound Boolean
expression
• e.g., both parts of digit_high == 1 || digit_low == -1

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 16

 Basic condition testing
• Adequacy criterion: each basic condition must be
executed at least once
• Coverage:

# truth values taken by all basic conditions

2 * # basic conditions

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 17

 Basic conditions vs branches
• Basic condition adequacy criterion can be
satisfied without satisfying branch coverage

T4 = {“first+test%9Ktest%K9”}
satisfies basic condition adequacy
does not satisfy branch condition adequacy

Branch and basic condition are not comparable

(neither implies the other)

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 18

 Covering branches and conditions
• Branch and condition adequacy:
– cover all conditions and all decisions
• Compound condition adequacy:
– Cover all possible evaluations of compound conditions
– Cover all branches of a decision tree
digit_high == -1

false true

digit_low == 1 FALSE

true false

TRUE FALSE

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 19

 Compound conditions:
Exponential complexity
(((a || b) && c) || d) && e

Test a b c d e
Case
(1) T — T — T
(2) F T T — T
(3) T — F T T
(4) F T F T T
(5) F F — T T
(6) T — T — F
(7) F T T — F
(8) T — F T F
(9) F T F T F
(10) F F — T F
(11) T — F F —
(12) F T F F —
(13) F F — F —

•short-circuit evaluation often reduces this to a more manageable

number, but not always

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 20

Modified condition/decision (MC/DC)
• Motivation: Effectively test important
combinations of conditions, without exponential
blowup in test suite size
– “Important” combinations means: Each basic
condition shown to independently affect the
outcome of each decision
• Requires:
– For each basic condition C, two test cases,
– values of all evaluated conditions except C are the
same
– compound condition as a whole evaluates to true for
one and false for the other

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 21

 MC/DC: linear complexity
• N+1 test cases for N basic conditions

(((a || b) && c) || d) && e

Test a b c d e outcome
Case
(1) true -- true -- true true
(2) false true true -- true true
(3) true -- false true true true
(6) true -- true -- false false
(11) true -- false false -- false
(13) false false -- false -- false

• Underlined values independently affect the output of the decision

• Required by the RTCA/DO-178B standard

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 22

 Comments on MC/DC
• MC/DC is
– basic condition coverage (C)
– branch coverage (DC)
– plus one additional condition (M):
every condition must independently affect the
decision’s output
• It is subsumed by compound conditions and
subsumes all other criteria discussed so far
– stronger than statement and branch coverage
• A good balance of thoroughness and test size
(and therefore widely used)

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 23

Paths? (Beyond individual branches)
• Should we explore
sequences of branches
(paths) in the control
flow?
• Many more paths than
QuickTime™ and a
TIFF (LZW) decompressor branches
are needed to see this picture.
– A pragmatic compromise
will be needed

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 24

 Path adequacy
• Decision and condition adequacy criteria
consider individual program decisions
• Path testing focuses consider combinations of
decisions along paths
• Adequacy criterion: each path must be
executed at least once
• Coverage:
# executed paths
# paths

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 25

 Practical path coverage criteria
• The number of paths in a program with loops is
unbounded
– the simple criterion is usually impossible to satisfy
• For a feasible criterion: Partition infinite set of
paths into a finite number of classes
• Useful criteria can be obtained by limiting
– the number of traversals of loops
– the length of the paths to be traversed
– the dependencies among selected paths

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 26

 Boundary interior path testing
• Group together paths that differ only in the
subpath they follow when repeating the body of
a loop
– Follow each path in the control flow graph up to the
first repeated node
– The set of paths from the root of the tree to each
leaf is the required set of subpaths for
boundary/interior coverage

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 27

Boundary interior adequacy for cgi-decode

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 28

 Limitations of boundary interior adequacy
• The number of paths can still grow exponentially

if (a) { • The subpaths through this control

S1; flow can include or exclude each of
} the statements Si, so that in total N
if (b) {
branches result in 2N paths that must
S2;
} be traversed
if (c) { • Choosing input data to force
S3; execution of one particular path
}
may be very difficult, or even
...
if (x) { impossible if the conditions are not
Sn; independent
}
(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 29
 Loop boundary adequacy
• Variant of the boundary/interior criterion that treats
loop boundaries similarly but is less stringent with
respect to other differences among paths
• Criterion: A test suite satisfies the loop boundary
adequacy criterion iff for every loop:
– In at least one test case, the loop body is iterated zero times
– In at least one test case, the loop body is iterated once
– In at least one test case, the loop body is iterated more than
once
• Corresponds to the cases that would be considered in a
formal correctness proof for the loop

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 30

 LCSAJ adequacy
• Linear Code Sequence And Jumps:
sequential subpath in the CFG starting and
ending in a branch
– TER1 = statement coverage
– TER2 = branch coverage
– TERn+2 = coverage of n consecutive LCSAJs

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 31

 Cyclomatic adequacy
• Cyclomatic number:
number of independent paths in the CFG
– A path is representable as a bit vector, where each component
of the vector represents an edge
– “Dependence” is ordinary linear dependence between (bit)
vectors
• If e = #edges, n = #nodes, c = #connected components
of a graph, basis set is:
– e - n + c for an arbitrary graph
– e - n + 2 for a CFG
• Cyclomatic coverage counts the number of independent
paths that have been exercised, relative to cyclomatic
complexity
(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 32
 Towards procedure call testing
• The criteria considered to this point measure
coverage of control flow within individual
procedures.
– not well suited to integration or system testing
• Choose a coverage granularity commensurate
with the granularity of testing
– if unit testing has been effective, then faults that
remain to be found in integration testing will be
primarily interface faults, and testing effort should
focus on interfaces between units rather than their
internal details

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 33

 Procedure call testing
• Procedure entry and exit testing
– procedure may have multiple entry points (e.g.,
Fortran) and multiple exit points
• Call coverage
– The same entry point may be called from many
points

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 34

 Subsumption relation

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 35

 Satisfying structural criteria
• Sometimes criteria may not be satisfiable
– The criterion requires execution of
• statements that cannot be executed as a result of
– defensive programming
– code reuse (reusing code that is more general than strictly
required for the application)
• conditions that cannot be satisfied as a result of
– interdependent conditions
• paths that cannot be executed as a result of
– interdependent decisions

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 36

 Satisfying structural criteria
• Large amounts of fossil code may indicate
serious maintainability problems
– But some unreachable code is common even in well-
designed, well-maintained systems
• Solutions:
– make allowances by setting a coverage goal less than
100%
– require justification of elements left uncovered
• RTCA-DO-178B and EUROCAE ED-12B for modified MC/DC

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 37

 Summary
• We defined a number of adequacy criteria
– NOT test design techniques!
• Different criteria address different classes of errors
• Full coverage is usually unattainable
– Remember that attainability is an undecidable problem!
• …and when attainable, “inversion” is usually hard
– How do I find program inputs allowing to cover something buried
deeply in the CFG?
– Automated support (e.g., symbolic execution) may be necessary
• Therefore, rather than requiring full adequacy, the
“degree of adequacy” of a test suite is estimated by
coverage measures
– May drive test improvement

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 38