Code Coverage Testing & Tool Support

W. Eric Wong
Department of Computer Science
The University of Texas at Dallas
[email protected]
http://www.utdallas.edu/~ewong

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 1
Speaker Biographical Sketch
Professor & Director of International Outreach
Department of Computer Science
University of Texas at Dallas
Guest Researcher
Computer Security Division
National Institute of Standards and Technology (NIST)
Vice President, IEEE Reliability Society
Secretary, ACM SIGAPP (Special Interest Group on Applied Computing)
Principal Investigator, NSF TUES (Transforming Undergraduate Education in
Science, Technology, Engineering and Mathematics) Project
– Incorporating Software Testing into Multiple Computer Science and Software
Engineering Undergraduate Courses
Founder & Steering Committee co-Chair for the SERE conference
(IEEE International Conference on Software Security and Reliability)
(http://paris.utdallas.edu/sere13)

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 2
Our Focus
We focus on testing programs
– subsystems or complete systems
– written in a formal language
– a large collection of techniques and tools

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 3
Testing for Correctness?
Identify the input domain of P
– Input domain of a program P is the set of all valid inputs that P can expect
– The size of an input domain is the number of elements in it
– An input domain could be finite or infinite
– Finite input domains might still be very large!
Execute P against each element of the input domain
For each execution of P, check if P generates the correct output as per its
specification S
– This form of testing is also known as exhaustive testing
as we execute P on all elements of the input domain.

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 4
Testing for Correctness? Sorry!
For most programs exhaustive testing is not feasible
– It will take several light years to execute a program
on all inputs on the most powerful computers of today!
What is the alternative?

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 5
Confidence in Your Program
Confidence is a measure of one’s belief in the correctness of the program.
It is not measured in binary terms: a correct or an incorrect program.
Instead, it is measured as the probability of correct operation of a
program when used in various scenarios.
It can be measured, for example, by test completeness
– The extent to which a program has been tested and errors found have been
removed.

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 6
How and why does testing improve our
confidence in program correctness ?

7
Example: Increase in Confidence
We consider a non-programming example to illustrate what is meant by
“increase in confidence.”
Example: A rectangular field has been prepared with respect to certain
specifications.
– One item in the specifications is
“There should be no stones remaining in the field.”

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 8
Rectangular Field
Search for stones inside a rectangular field

(0,W)

(0,0) (L,0)
x

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 9
Testing the Rectangular Field
The field has been prepared and our task is to test it to make sure that it
has no stones.

How should we organize our search?

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 10
Partitioning the Field
We divide the entire field into smaller search rectangles.

The length and breadth of each search rectangle is one half that of the
smallest stone.

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 11
Partitioning into Search Rectangles
Stone
8
7
width 6
5
y 4
3
2
1
1 2 3 4 5 6 7
length
x

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 12
Input Domain
Input domain is the set of all possible inputs to the search process.

In our example this is the set of all points in the field.

Thus, the input domain is infinite!

To reduce the size of the input domain we partition the field into
finite size rectangles.

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 13
Rectangle Size
The length and breadth of each search rectangle is one half that of the
smallest stone.

This ensures that each stone covers at least one rectangle.

Is this always true?

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 14
Constraints
Testing must be completed in less than H hours
Any stone found during testing is removed
Upon completion of testing the probability of finding a stone must be less
than P

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 15
Number of Search Rectangles
Let
L: length of the field
W: width of the field
α: length of the smallest stone
β: width of the smallest stone
Size of each rectangle: (α/2) * (β/2)
Number of rectangles: N = (L/α)*(W/β)*4
Assume that L/α and W/β are integers.

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 16
Time to Test
Let t be the time to look inside one rectangle.
Assume that no rectangle is examined more than once.
Let o be the overhead in moving from one rectangle to another.
Total time to search T =N * t + (N − 1)* o
Testing with N rectangles is feasible only if T < H

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 17
Partitioning the Input Domain
This set consists of all rectangles (N).
Number of partitions of the input domain is finite (N).
However, if T > H then the number of partitions is too large and
scanning each rectangle once is infeasible.
What should we do in such a situation?

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 18
Option 1: Do a Limited Search
Of the N rectangles we examine only n where n is such that
(t * n + o* (n − 1)) < H.
This limited search will satisfy the time constraint.
Will it satisfy the probability constraint?

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 19
Distribution of Stones
To satisfy the probability constraint we must scan enough rectangles so
that the probability of finding a stone, after testing, is less than P.
Let us assume that
– there are si stones remaining after i test cycles.
– There are Ni rectangles remaining after i test cycles.
– Stones are distributed uniformly over the field
– An estimate of the probability of finding a stone in a randomly selected
remaining search rectangle is pi = si / Ni

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 20
Probability Constraint
We will stop looking into rectangles if pi ≤ P
Can we really apply this test method in practice?

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 21
Why Not
Number of stones in the field is not known in advance.
Hence we cannot compute the probability of finding a stone after a
certain number of rectangles have been examined.
The best we can do is to scan as many rectangles as we can and remove
the stones found.

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 22
Coverage
After a rectangle has been scanned for a stone, we say that the rectangle
has been covered.
Suppose that n rectangles have been scanned from a total of N. Then we
say that the coverage is n / N.

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 23
Coverage and Confidence
What happens when coverage increases?
– As coverage increases so does our confidence in a “stone-free” field
In this particular example, when the coverage reaches 100%, all stones
have been found and removed.
Can you think of a situation when this might not be true?

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 24
Option 2: Reduce Number of Partitions
If the number of rectangles to scan is too large,
we can increase the size of a rectangle.
– This reduces the number of rectangles.
Increasing the size of a rectangle also implies that
there might be more than one stone within a rectangle.
– Is it good for a tester?
– It also implies . . . . . . . . . . .

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 25
Rectangle Size
As a stone may now be smaller than a rectangle, detecting a stone inside
a rectangle (by examining only one point) is not guaranteed.
Despite this fact our confidence in a “stone-free” field
still increases with coverage.
However, when the coverage reaches 100%
we cannot guarantee a “stone-free” field.

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 26
Coverage versus Confidence

Does not imply that the field

is “stone-free”.
Confidence

0 Coverage 1(=100%)

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 27
Rectangle Size

t, p p = Probability of detecting a stone inside a

rectangle, given that the stone is there
t = time to complete the testing

Rectangle size
small large

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 28
Analogy
Field Program
Stone Error
Scan a rectangle Test program on one input
Remove stone Remove error
Partition Subset of input domain
Size of stone Size of an error
Rectangle size Size of a partition (wrt “Program”)

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 29
Confidence and Probability
Increase in coverage increases our confidence in a “stone-free” field.
It might not increase the probability that the field is “stone-free.”

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 30
Review Questions
What is the effect of reducing the partition size on probability
of finding errors?
How does coverage affect our confidence in program correctness?
Does 100% coverage imply that a program is fault-free?
Indicate whether the following statements are true or false
– The objective of software testing is to prove the correctness of the program
being tested
– The reliability of a program will always increase as your confidence of the
program being correct increases

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 31
 What is coverage
and
what role does it play in testing?

32
Coverage Principle
The basic idea of coverage testing is that testing is complete
when a well-defined set of tests is complete.
– Example
Pilots use pre-flight check lists
Shoppers use grocery lists
to assure the correct completion of their tasks
– In the same way testers can count the completed elements of a test plan
Example
Requirements
Functionalities
Blocks, Decisions (control-flow based)
C-uses, P-uses and All-Uses (dataflow-based)

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 33
The Role of Coverage in Testing
It provides a way of monitoring and measuring the progress of testing
against explicit quantitative completion criteria
– Gives a clear measure of the completion of the testing task
– Example, for requirements testing
How many of the requirements have been tested?
How many tests have run per requirement?

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 34
Topics
Code Coverage testing and code inspection
Code Coverage testing and functional testing
Controlflow-based testing
Dataflow-based testing

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 35
What is Code Coverage Testing
• It is “White Box Testing”
• Takes into account the structure of the software being tested
• Measures how thoroughly the code has been tested with respect to certain
metrics

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 36
Code Coverage Testing versus Code Inspection
• Code inspection is a technique whereby the source code is inspected for
possible errors

• Code coverage testing is a dynamic method, whereas code inspection is

a static method

• Code coverage testing is a form of code inspection

– Code that is executed successfully is disregarded for visual inspection
– Code that is not executed is inspected
– One is not likely to replace testing by code inspection

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 37
Code Coverage Testing versus Functional Testing
When test inputs are generated using program specifications,
we say that we are doing functional testing
– Functional testing tests how well a program meets the
functionality requirements
These two types of testing are complementary
– Basic functionalities should always be tested
– The set of tests generated from functional testing provides
a good basis for code coverage testing

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 38
History of Code Coverage Testing
• Using profiling tools to assess the amount of code coverage during testing
(1960’s)
• Using tcov to give statement coverage data for C and Fortran programs
(1970’s)
• Two groups of test criteria
– Controlflow-based testing (block & decision)
– Dataflow-based testing (c-use, p-use and all-uses)

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 39
Basic Block
• A basic block is a sequence of consecutive statements or expressions,
containing no branches except at the end, such that if one element of the
sequence is executed all are.

A program, its control flowgraph, basic blocks, and decision

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 40
Decision
• A decision is a boolean predicate with two possible values, true and false

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 41
C-use & P-
P-use

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 42
Importance of Code Coverage Testing
• In general, a piece of code must be executed before a fault in it can be
exposed
• Helps early fault detection
– Are system testers finding faults that should have been found and fixed by
developers?
– Relative cost of fixing a software fault

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 43
State of Practice
• A published study (ICSE’92)
– Coverage above 60-70% in system testing is very difficult

• Don Knuth’s system testing of TeX (23,000 LOC)

– 85% block and 72% decision coverage (1992)

• Brian Kernigan’s testing of AWK

– 70% block and 59% decision coverage (1991)

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 44
Efficient Coverage Testing (1)
• How much code is currently tested?
What is missing?
– Which statements were exercised?
– Which paths were traversed?
– Which def-use associations were exercised?
– Which functions got invoked from where?

• Need help in creating tests?

– Which statement should I try to cover next?

Analyzing the controlflow graph of the program to find the

dominant blocks, decisions, and def-use pairs.

For example, when a test covers highly dominant blocks it

will cover many other blocks.

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 45
Efficient Coverage Testing (2)

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 46
Efficient Coverage Testing (3)
Use prioritization and visualization to provide hot spots that give the most value in coverage.
Each color represents a different weight determined by a control flow analysis using the
concept of superblocks and dominators.

Code in white has already been

covered by a test case and covering
it again will not add new coverage

Covering this red block

guarantees the execution of
at least 8 additional blocks.

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 47
Efficient Coverage Testing (4)

Covering either true or

false branch guarantees
the execution of at least
another 8 branches.

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 48
Dominator & Super Block (1)
• A super block consists of one or more basic blocks that if one block in the
super block is executed all are
– If any statement in a super block is executed, then all statements in it must be
executed, provided the execution terminates on that input
– A super block needs not be contiguous
• Block u dominates block v if every path from entry to end, via v, contains u
– u dominates v if covering v implies the coverage of u
– Test execution cannot reach v without going through u
• Given a program, identify a subset of super blocks whose coverage implies
that of all super blocks and, in turn, that of all basic blocks

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 49
Dominator & Super Block (2)

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 50
Dominator & Super Block (3)

• Quiz: Does node 4 or node 12 • Quiz: Does node 9 postdominate node

predominate node 13? Why? 8? Why?

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 51
Dominator & Super Block (4)

• A strongly connected component of a

basic block dominator graph has the
property that every node in the
component dominates all
other nodes in that component

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 52
Dominator & Super Block (5)

• Obtained by removing the composite

edges in the right Figure on the
previous slide • Only need to create test cases that cover
• An edge e from a node u to a node v basic blocks 4, 7, 9, and 10 – one from
is said to be a composite edge if v is each leaf node in the super block
also reachable from u without going dominator graph
through e
Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 53
Dominator & Super Block (6)
• At most four test cases need to be developed to cover all 14 basic blocks

• An alternative order is 10, 7, 4, and 9

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 54
Dominator & Super Block (7)
• Experimental results

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 55
Weight Re-
Re-Computation (1)
The weight of a given node is the number of nodes that have not been
covered but will be if that node is covered

• To arrive at node 18 requires the

execution also go through nodes 1, 2, 4,
7, 12 and 13
• Node 18 is dominated by nodes 1, 2, 4,
7,12 and 13
• These nodes will be covered (if they
haven’t been) by a test execution if that
execution covers node 18
• Assuming none of the nodes is covered
so far, we say that node 18 has a weight
of 7 because covering it will increase the
coverage by at least 7 additional nodes.

Why is it important to take a “conservative” approach?

– Will node 6 be covered by covering node 18?

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 56
Weight Re-
Re-Computation (2)

• Arriving at node 6 requires the execution only goes through nodes 1, and 2
•Assuming none of the nodes is covered so far, we say that node 6 has a weight of 3

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 57
Weight Re-
Re-Computation (3)
• The execution of certain tests may change the weights of nodes that
are not covered by these tests.

• After a test is executed to cover node 18, the weight (in terms of
increasing the coverage) of node 6 is reduced from 3 to 1.

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 58
The χSuds Tool Suite
Telcordia Technologies (formerly Bellcore or Bell Communications
Research)
– χSuds (Software understanding and diagnosis systems): a set of
software testing, analysis, and understanding tools for C and C++
programs
χATAC
χSlice
χRegress
χVue
χProf

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 59
χSuds Home Page

http://xsuds.argreenhouse.com
Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 60
χATAC Demo: Coverage Testing of C Code

Source display after

Compile code with χATAC Initial display χSuds User’s Manual executing wordcount.1

Source display after 100 % block coverage after Source display after 100 % block & decision
executing wordcount.2 executing wordcount.5 executing wordcount.6 coverage after executing
wordcount.9

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 61
Coverage Testing Tools for Java Code
eXVantage (eXtreme Visual-aid novel testing and generation)
– A tool suite for code coverage prioritization, test generation, test execution,
debugging, and performance profiling of Java, C, and C++ programs
– Based on the JBT (Java Bytecode Testing) tool suite developed at UTD since
2002
Clover
Cobertura
etc.

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 62
 The End

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 63
eXVantage Home Page

W. Eric Wong and J. Jenny Li, “An Integrated Solution for Testing and Analyzing Java
Applications in an Industrial Setting,” in Proceedings of The 12th IEEE Asia-Pacific
Software Engineering Conference (APSEC), pp. 576-583, Taipei, Taiwan, December 2005

Code Coverage Testing & Tool Support (© 2012 Professor W. Eric Wong, The University of Texas at Dallas) 72