See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/221181402

A Multipurpose Code Coverage Tool for Java

Conference Paper · January 2007

DOI: 10.1109/HICSS.2007.24 · Source: DBLP

CITATIONS READS

32 522

3 authors:

Raghu Lingampally Atul Gupta

1 PUBLICATION 32 CITATIONS
PDPM Indian Institute of Information Technology, Design and Manufacturing Jaba…
59 PUBLICATIONS 1,305 CITATIONS
SEE PROFILE
SEE PROFILE

Pankaj Jalote
Indraprastha Institute of Information Technology
136 PUBLICATIONS 3,405 CITATIONS

SEE PROFILE

All content following this page was uploaded by Atul Gupta on 31 January 2014.

The user has requested enhancement of the downloaded file.

 Proceedings of the 40th Hawaii International Conference on System Sciences - 2007

A Multipurpose Code Coverage Tool for Java

Raghu Lingampally, Atul Gupta, and Pankaj Jalote

Indian Institute of Technology Kanpur, India
[email protected], {atulkg,jalote}@cse.iitk.ac.in

Abstract ing technique for regression testing on C programs and re-

ported huge savings in time and size of the resulting re-
Most test coverage analyzers help in evaluating the ef- gression test suites. Test-suite augmentation problem is to
fectiveness of testing by providing data on statement and determine which new test cases must be developed to test
branch coverage achieved during testing. If made avail- added/uncovered functionality of the modified code [22].
able, the coverage information can be very useful for many For more on regression testing and test suite augmentation,
other related activities, like, regression testing, test case please refer to [32, 21, 22].
prioritization, test-suite augmentation, test-suite minimiza- Test case prioritization is an approach to ordering the
tion, etc. In this paper, we present a Java-based tool test cases in terms of some effectiveness criteria which can
JavaCodeCoverage for test coverage reporting. It sup- be used further for other activities like enhancing regres-
ports testing and related activities by recording the test cov- sion testing [19, 28, 24], test suite minimization, etc. There
erage for various code-elements and updating the coverage are various approaches for test case prioritization, many of
information when the code being tested is modified. The which make use of coverage information. For example,
tool maintains the test coverage information for a set of test method/block coverage in [28]and branch/MCDC cover-
cases on individual as well as test suite basis and provides age in [24]. Efficient prioritization-algorithms are designed
effective visualization for the same. Open source database which make use the information of uncovered blocks at bi-
support of the tool makes it very useful for software testing nary code level [28].
research. A study done by Wong et al. [32] suggested that test
suite minimization may reduce test suite size remarkably,
at little cost to the fault-detection effectiveness of those test
suites, but later, Rothermal et al.[27] in another empirical
1 Introduction
study, found that the fault detection capabilities of test
suites can be severely compromised by minimization. Such
Software testing is an essential activity for program val- contradictory results from experimental research are not
idation. Given that the exhaustive testing of a program gen- uncommon and suggest further explorations of the facts to
erally not possible, next best thing to do is adequate testing increase our understanding of these approaches, for which
which is typically performed by obtaining a test model of availability of appropriate tools and other infrastructural
the code under test (CUT), may be at different levels of ab- support is absolutely essential[18]. The work described
straction, and covering some specific aspect of that model. in this paper can be considered as one such effort in that
Control flow graph (CFG) based coverage testing for struc- direction.
tural entities (e.g. statements in the code) are among the
most popular testing techniques used in practice. The main contributions of this paper are:
Bug fixes, code re-factoring, software-upgrades are most
frequent activities in software development because of 1. The tool that we introduce here is a Java bytecode cov-
which code keeps on changing. The purpose of regression erage analyzer which provides test coverage informa-
testing is to ensure that these changes in the code do not tion at method, block, branch and predicate level.
adversely affect the correct functionality inherited from its 2. The tool supports change management by identifying
previous version. Agrawal et al. [16] demonstrated an ef- changed (added, deleted or modified) code-elements,
fective approach for incremental regression testing of C pro- and correspondingly updating test code coverage for
grams by not selecting those tests which do not cover any the changed code.
of the changed statements of the code using a tool ATAC 3. An important aspect of the tool is that it perform all the
[26]. Vokolos and Frankl [30] applied textual differenc- analysis at individual test case level, thereby facilitat-

Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)
0-7695-2755-8/07 $20.00 © 2007 1530-1605/07 $20.00 © 2007 IEEE 1
 Proceedings of the 40th Hawaii International Conference on System Sciences - 2007

ing the detailed analysis and support for activities like and therefore more effective than the other two. It can pro-
test suite minimization, test case prioritization, regres- vide integral coverage for single statement at source code
sion testing, experimental validations, etc. level (a source code line may contain multiple code state-
4. We demonstrate the usefulness of our tool by using two ments) and can be easily integrated to run on the fly.
simple examples.
5. The tool has simple open source database interface 2.2 Existing Coverage Tools for Java
for MySQL. The database can be access directly or
through the tool to facilitate further exploration of the Table 1 presents a comparison between existing code
facts. We illustrate this point by discussing some of the coverage tools for Java including the one we introduced
studies done using the tool in this regard. here.
Most of the tools (Table 1) provide statement, branch,
The paper is organized as follows: First we discuss cov- and/or method level coverage information and do not have
erage based testing and present a brief survey on some support for change management. Many tools do not provide
of the existing Java coverage tools in the Section 2. The detailed coverage information and present a coverage sum-
tool JavaCodeCoverage organization and underlying ap- mary only. Our tool provides detailed coverage information
proach is presented in Section 3, and its usefulness is for all these program elements including predicates. The
demonstrated using two simple examples in Section 4. In open-source database support with detailed test coverage
Section 5, we highlight further prospective usage of the tool. information including coverage updates on code changes
We present the summary and future work in Section 6. makes it a very useful tool for software testing research.

2 Coverage Based Testing 3 JavaCodeCoverage Tool

Code coverage analysis is the process of finding areas JavaCodeCoverage is a bytecode analyzer tool for test
of a program not exercised by a set of test cases, thereby coverage analysis for Java software which neither requires
resulting a quantitative measure of code coverage, which is the language grammar nor the source code. An important
an indirect measure of code quality [23]. At the same time aspect of JavaCodeCoverage is that it stores the coverage
it can help in creating additional test cases to increase the information for individual test case thereby facilitating de-
coverage and may identify unreachable potions of the code. tailed coverage analysis. Another important aspect of Java-
Additionally, it can identify redundant test cases that do not CodeCoverage is that it records all vital code-elements and
increase coverage, as well as help in testing changes made test coverage information in a open source database soft-
to the code during regression testing. A proper coverage ware MySQL [11]. This allows easy accessibility to stored
tool is highly desirable to meet all these objectives. information which increases the utility of the tool for many
other purposes like those used in [20, 29]. Besides storing
2.1 Java Coverage Analyzers the information in the database, it also provides a friendly
graphical user interface (GUI) to visualize typical code and
Coverage measures based on various code-elements such test coverage information. Some other features of the tool
as methods, statements, blocks, branches, predicates are are:
most widely employed for coverage based testing. The cov-
erage analysis tools are language dependent. Coverage an- • It records information regarding source code structure
alyzers work by adding probe instructions in the program of the software including classes and their methods.
which increment counters. These instructions are collec- • For each method, it records information about number
tively called as instrumentation. Java coverage analyzers of blocks, branches, predicates (conditions) including
can be categorized according to the instrumentation tech- their location in source as well as in bytecode.
niques used, as:
• It records the changes made in the code compared to
• Source code instrumentation its previous version and accordingly update the test
• Bytecode instrumentation coverage information of the test suite executed on its
previous version. This helps in performing regression
• Those that run the code in a modified JVM.
testing effectively.
Bytecode instrumentation adds probe instructions di- • It also helps in identifying heavily tested coverage ele-
rectly to the bytecode. It is very useful in situations where ments as well as lightly or un-tested coverage elements
the source code is not available. It neither requires a modi- by recording their frequency of execution in the test
fied JVM nor requires recompiling all the source code twice case/test suite.

Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)
0-7695-2755-8/07 $20.00 © 2007 2
 Proceedings of the 40th Hawaii International Conference on System Sciences - 2007

Coverage Tool Type Types of Cov- Instrumentation type GUI sup- Coverage De- Support Direct Requirements
erage port tailing (Test for Change Data-
case/ Test Management base
Suite) Access
Hansel [1] Open Statement ClassLoader N Test suite N N BCEL 5.0,
Source JDK1.4
Quilt [2] Open Block, branch ClassLoader N Test suite N N JDK 1.4
Source
Clover [3] Proprietary Statement, Source code Y Test suite N N Jakarta pack-
branch, ages, Antlr
method
Emma [4] Open Line, block, instruments Byte- Y Test suite N N No external li-
Source method, class code off line or brary dependen-
ClassLoader cies
JavaCodeCoverage Open Method, block, Bytecode Y Test case/Test Y Y BCEL 5.1,
[5] Source branch, predi- Suite MySQL, GNU
cate diff utility
JCover [6] Proprietary Statement, Bytecode Y Test suite N Y Java Debugger
branch, Interface
method
Jcoverage [7] both op- Line Bytecode N Test suite N N NA
tions
TCAT [8] Proprietary Branch, Bytecode Y Test suite Y N NA
method-pair
Java Test Cover- Proprietary Branch Source code Y Test case/Test N N NA
age [9] suite
InsectJ [10] Open Statement, Bytecode Y Test case/Test Yes N BCEL, JDK
Source branch, suite
method call
coverage

Table 1. Coverage Tool Comparison

• The tool can be augmented with Java testing frame-

works like JUnit [12].
• It can be used as a command line or, alternatively, via
its GUI browser.

In this section, we describe the overall organization of

the tool JavaCodeCoverage and underline its instrumen-
tation and change analysis approaches.

3.1 JavaCodeCoverage Architecture

Functionally, the overall system has four components as

shown in Figure 1.

• Instrumentor
• Database
• Code Change Analyzer Figure 1. JavaCodeCoverage Components
• GUI
so as to effectively identify the portions of the changed pro-
Code under test (CUT) is given to the instrumentation
gram to be covered by further testing. This may be done
engine, i.e., the Instrumentor, which analyzes the bytecode
by selecting relevant test cases from the original test suite
using BCEL APIs [13] and inserts probes at appropriate lo-
and/or applying new test cases.
cations depending on the type of coverage desired. The in-
strumented code is then run with a test case or a test suite.
During a test-execution, when a probe is invoked, the rele- 3.2 The Instrumentor
vant data is recorded in the database.
When the program is changed, the changes are tracked The bytecode of a class typically represents a unit of
during change analysis and decoded. The Code Change An- code for the Java-software. We analyze the whole software
alyzer is responsible for updating the coverage information by doing the class-level bytecode analysis. For each class,

Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)
0-7695-2755-8/07 $20.00 © 2007 3
 Proceedings of the 40th Hawaii International Conference on System Sciences - 2007

its bytecode is instrumented in two passes as shown in Fig- in the programs is handled separately. Branch probe
ure 2 In the first pass, CFG for each method of the class consists of class-id, method-id, and branch-id.
is generated and appropriate probe locations for all types • Predicate coverage: In the first pass, all the predi-
of coverage are identified. In the second pass, the class is cates are identified. In second pass, a probe instruc-
instrumented using probes calculated in the first pass. The tion is inserted to track the operands for predicate.
class is instrumented for four types of coverage, namely, A probe instruction is inserted before every simple
method, block, branch, and predicate coverage. During in- predicate and branches are set appropriately. A pred-
strumentation phase, probes are inserted which call static icate probe consists of information about predicate-
methods of execution monitors to store the coverage infor- id, method-id, class-id, and instructions to duplicate
mation in the database. predicate-operands on the stack.

The process for code instrumentation and obtaining cov-

erage information is outlined in Figure 2.

3.3 DataBase Organization

A MySQL database [11] is used to record the code-

elements and test coverage information. Database organi-
zation can be categorized into three kinds of tables. (See
Table 2)

1. Information Tables: These tables capture various

code-elements information such as methods, blocks,
branches, and predicates present in the program. They
also maintain the code-change information about the
instrumented classes from their previous versions, in-
cluding source-line number, offset in the bytecode and
Figure 2. Code Instrumentation for Coverage indices which are used by the other tables.
2. Coverage Tables: These tables record over-all cover-
Execution monitors consist of static methods which up- age information for different types of coverage which
date coverage information while instrumented program is is updated dynamically with test-executions.
running. Whenever a call to static method of execution 3. Test case Tables: These tables consist of test case
monitor is made, the information about the call is buffered information and execution details. Test case defini-
and call is returned immediately. Subsequently, the buffered tions consists of date of test case creation, tester, com-
information is transferred to database using database access mand used, input details and comments provided by
layer. the tester. Each execution or run of the program is con-
Database access layer is a set of classes which stores and sidered as a test case. Coverage for each test case and
retrieves the information from database persistently. The overall coverage can be obtained separately. Coverage
way in which probes are inserted into the program depends information and execution trace for each test case is
on the type of coverage. Here is a brief description of the also stored in these tables.
places at which probes are inserted for each coverage type.

• Method coverage: At the beginning of each method,

3.4 Code Change Analyzer
a probe is inserted to track coverage information for
method coverage. Here the probe consists of class-id A change in the code, typically represented by code addi-
and method-id. tions, deletions, and/or modifications in the code, results in
a new version of that code. In this section we describe how
• Block coverage: A probe is inserted just before the code changes are analyzed and reflected in the database. We
first instruction of every block to track coverage infor- assume that the whole code is organized into a set pack-
mation. This probe consists of class-id, method-id, and ages arranged in a tree-structured hierarchy. We describe
block-id. the approach for identifying changes at a package scope, as
• Branch coverage: Probes are inserted just before a the complete change identification then would be analyz-
branch instruction and before the two branch targets to ing all such packages which constitute the whole software.
track branch coverage. Branch coverage of exceptions The Code Change Analyzer consists of a Change Decoder, a

Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)
0-7695-2755-8/07 $20.00 © 2007 4
 Proceedings of the 40th Hawaii International Conference on System Sciences - 2007

S. MySQL Tables Description

No.
1 Information PACKAGES package information of the CUT
2 Tables CLASSES classes information of the CUT
3 METHODS method information the CUT (method id,class id, etc)
4 BLOCKS block information of the CUT (block id, method id,class id, etc)
5 BRANCHS branch information of the CUT (branch id, method id,class id, etc)
6 PREDICATES simple predicate information of the CUT (predicate id,method id,class id, etc)
7 Coverage METHOD COVER method coverage information and # of times a block is executed by test cases
8 Tables BLOCK COVER block coverage information and # of times a block is executed by test cases
9 BRANCH COVER branch coverage information and # of times a block is executed by test cases
10 TRUEPREDICATE COVER predicate coverage information if it has been exercised as TRUE or 1(long values)
11 FALSEPREDICATE COVER predicate coverage information if it has been exercised as FALSE or 0 (long values)
12 MINUSPREDICATE COVER predicate coverage information if it has been exercised as -1(long values)
13 Test METHOD EXECUTION sequential method execution information for a set of test cases
14 Execution BLOCK EXECUTION sequential block execution information for a set of test cases
15 Tables BRANCH EXECUTION sequential branch execution information for a set of test cases
16 PREDICATE EXECUTION sequential predicate execution information for a set of test cases with their result

Table 2. Important Database Tables and their Descriptions for JavaCodeCoverage

Change Analyzer and an Impact Analyzer as shown in Fig- deleted, and/or modified from the source code us-
ure 1. Change Decoder finds the changes made to the code ing GNU diff utility [14]. We mapped this infor-
at class level. The effects of these changes are consolidated mation to the byte-code block positions of the older
by the Change Analyzer by propagating them to include version of the method to determine the information
other sections of the code. The impact of these changes on of added/deleted/modified/unchanged blocks in the
test coverage data is analyzed through the Impact Analyzer. newer version.

3.4.1 Change Decoder Please note that the default marking for the code-
elements in the database prior to performing change analy-
Change decoder finds the changes made to the code. We sis was ’unchanged’. So if an element is not marked
follow a combination of top-down and bottom-up approach during change analysis then it remained as ‘unchanged’.
for change identification in the two version of the code. Ini- For added and modified blocks in the newer version,
tially, we begin to look for the set of the classes in a package we compare branches and predicates. As every block
that has been changed (i.e. added, deleted, or modified) in ends with a branch condition, so any branch pointing to
the newer version and then analyze the methods of a class added/deleted/modified block is also marked as ‘modified’.
for the change identification in the similar way. Most of the For deleted blocks in the older version, we mark predicates
changes are worked out at byte-code level except the case and branches in that block as ‘deleted’.
with the changes in a method-body, which uses a mixed ap-
proach at source and byte-code levels. Overall, the steps in
3.4.2 Change Analyzer
identifying the changes in the two versions of the code are:
Changes made to the CUT are further analyzed and cat-
• A class in the newer version, if not existed in the older egorized as shown in Table 3. Whenever a class is
version, marked1 as ‘added’. For a given class which changed, the class and the package containing it are con-
existed in older version and not in the newer version, sidered as changed. Whenever a method is changed, the
we marked it as ‘deleted’. method, the corresponding class, and the package contain-
• If a class existed in old and new version then we ing that class are also considered as changed. Whenever
mapped the methods in the older version and new ver- any block/branch/predicate is changed the corresponding
sion by considering their signature so as to identify method, class and package are considered as changed.
’added’ or ’deleted’ methods. For other methods (i.e. Bytecode of two versions of the classes are parsed and
common to both versions) of the class, we compared compared to find the differences at class, method, block,
byte-codes of each method in the two versions. If they branch, and predicate level. For example, method level dif-
are found to be different at any stage, the method is ferences are computed by comparing all the methods of the
marked as ‘modified’. new version of the CUT with the methods available in old
version. If part of the code of a method in the new ver-
• For a modified method, we found the source line sion is not matched with old version then it is considered as
numbers (in the new version) which are added, modified. For all such modified methods (i.e. with added,
1 please note that the word ‘mark’ stands for updating the status for the deleted, and/or modified code), code differencing at block,
relevant code-element in the database branch and predicate level is computed. If a method in new

Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)
0-7695-2755-8/07 $20.00 © 2007 5
 Proceedings of the 40th Hawaii International Conference on System Sciences - 2007

version is not found in old version, it is considered as an eling a class method. They identified the ‘dangerous’ enti-
added method. If a method in old version is missing in the ties as those branches of the CFG branch targets which con-
new version, it is marked as deleted. Likewise, bytecode tained any changed statements and behave differently from
differences are computed for other levels. This changed in- the previous version of the code. Effectively all kinds of
formation is used by the impact analyzer for resetting the code changes are mapped on a set of affected branches in
test code coverage information. some method. Then they generate a regression test suite by
selecting the test cases, from the test suite of the previous
version of the code, which exercised these dangerous edges.
3.4.3 Impact Analyzer Here we provide more flexibility by recording not only the
It analyzes the impact of changes made to the code and re- affected branches but blocks and predicates as well, which
sets the test coverage appropriately. It also marks the ex- may help in performing more sophisticated change analysis.
ecution details for test cases as un-covered for those code- Moreover, our tool also supports for test suite augmentation
elements which get affected by these changes. problem [22] by providing the information of newly added
Impact on Coverage : After a change in the code, the code elements.
test coverage for the new version is accordingly updated. Herrold’s approach considered a method call as a branch
The coverage of the changed blocks is reset, i.e., initialized in the CFG and therefore was able to perform inter-
to zero. The coverage information for unchanged part of procedural change analysis. As of now inter-procedural
the code is retained. For simplicity, we reset the coverage change analysis is not incorporated in our tool. This we
for the changed blocks only though more involved cover- consider as one of the future work to enhance the capability
age re-setting is also possible in which control flow for the of our tool.
method is considered and coverage information for the ele-
ments which are successors of changed elements in the con-
trol flow will be reset.
Method coverage for the method is reset when part of the
code of a method is changed. When a method is changed
the impact analyzer analyzes changes made to the method
at block, branch, and predicates level. The approach for
identification of change is a more conservative one in which
source line numbers of the old version of code are identified
for change management. The bytecode blocks of the old
version corresponding to changed source line numbers are
marked as modified, if they are also present in the new ver-
sion of the code. Coverage is retained for the blocks which
are unmodified and the coverage for the modified and added
blocks is reset. Branch coverage for the branches which
lead to the changed blocks (i.e. added, modified, or deleted)
is reset and these branches are marked as modified. Cov-
erage for all the predicates which are mapped on changed
source line number is reset to zero and they are marked as
modified. Accordingly, the coverage is updated for the re- Figure 3. Change and impact analysis
organized block structures of the modified methods in the
new version of the code.
Impact on test cases : Test cases which covered the modi-
fied elements of the code in the older version, are the desired 3.5 GUI
candidates for this analysis. Test case coverage information
for the new version of the CUT is computed and reflected
in database. This is done by re-setting the coverage for the The tool has a friendly GUI implemented using JAVA
affected code elements as obtained in impact analysis. This Swing. It retrieves code-elements, test coverage, and test-
forms the basis for regression testing and test case prioriti- execution information from the database. The information
zation as the desired information can be obtained simply by for all the coverage measures is displayed at package, class,
querying the database. and test case levels. Updated test coverage information can
A similar approach for identifying the changes in Java also be displayed for the changed versions of the CUT. A
programs, Herrold et. al. [22] have used a CFG for mod- snapshot of the tool’s GUI is shown in Figure 4.

Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)
0-7695-2755-8/07 $20.00 © 2007 6
 Proceedings of the 40th Hawaii International Conference on System Sciences - 2007

Change Package Class Method Block Branch Predicate

Added A new package is added A new class is added New method A new block is added New branch condition is New pred-
is added to to the control flow of added in the control flow icate is
class method added
Deleted A package is deleted A Class is deleted Existing An existing block in the Existing branch condi- Existing
method is method is deleted tion is deleted predicate is
deleted deleted
Modified A class in package is A class is modified or Part of the Instructions in the block Conditions relating to the Predicate or
modified or some classes base class for the class or method is modified added or branch or branch target its operands
added or deleted or mod- inheritance changed changed deleted changed changed
ified

Table 3. Code Change Categorization

Test cases and Classes

Instrumented executed by the
classes test case #6

Block coverage
highlighted for
test case #6

Coverage
summary for
the test case
#6 for class
GameMaster

Figure 4. A snapshot of the GUI

4 JavaCodeCoverage: Illustrative Examples S.

No
Mutation Operator (Bug Type) Abbr. # Bugs

1 Unreachable code URC 1

In this section, we demonstrate the usefulness of our tool 2 Initialization operator ITO 2
3 Literal Change Operator LCO 3
using two simple examples. Some other prospective usage 4 Language Operator Replacement LOR 3
of the tool is given in Section 5. Please note that the use 5 Argument Change Order ACO 1
6 Method Name Replacement MNR 1
of JavaCodeCoverage is not limited to the applications 7 Variable Replacement Operator VRO 3
discussed in this paper. 8 Statement Swap Operator SSO 1
9 Control Flow Disruption CFD 3

4.1 Coverage Based Testing: An Illustra-

Table 4. Bug Statistics for HotelBooking
tive Example for bug identification

In this experiment, we show how effective are the

three coverage measures, namely, block, branch and pred- ence. Table 4 shows the information about the bugs seeded
icate coverage, at detecting faults during testing. For in the program. Bugs revealed by the random test suites for
this, we have taken simple hotel management program, each coverage type is given in Table 5. The results suggest
HotelBooking, having six classes. Twenty bugs were in- that predicate coverage was the best in terms of number of
serted in the program according to a set of applicable muta- faults identified but worst in the efforts requirements (taken
tion operators [25]. Using JavaCodeCoverage with 90% 23 test cases) whereas block coverage performed the other
coverage for the respected measures as stopping criteria, the way round. Branch coverage appeared as a better compro-
program was tested by three testers having similar experi- mise in this case.

Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)
0-7695-2755-8/07 $20.00 © 2007 7
 Proceedings of the 40th Hawaii International Conference on System Sciences - 2007

Coverage Type Bug Type

URC ITO LCO LOR ACO MNR VRO SSO CFD Total Test Suite Size
Total-Bugs 1 2 3 3 1 1 3 1 3 18
Block Coverage 0 0 2 2 0 1 3 1 2 11 13
Branch Coverage 1 1 3 2 0 1 3 1 2 14 16
Predicate Coverage 1 1 3 3 0 1 3 1 3 16 23

Table 5. Bug detected by Coverage type

Package Jakarta RegExp

Number of Classes 11
program comprehension, mutation testing, etc. We present
Number of Methods 179 the case for such analysis in the following sub-sections.
Number of Blocks 727
Number of Branches 699
Number of simple predicates 277 5.1 Test Suite Minimization
Time taken for instrumentation 5582 ms

Table 6. RegExp Package Details From the test case coverage information for various code
elements, we can reduce the size of the test suite by identi-
Coverage type No. of No. of Coverage % fying and removing redundant test cases from it. Test suite
units Covered
units minimization can be effectively used for comparing vari-
Class 11 11 100 ous coverage criteria. Gupta and Jalote [20] have compared
Method 79 53 53
Block 727 451 67 the fault detecting effectiveness and efforts requirements of
Branch 699 420 60 the three control coverage criteria, namely, block, branch
Predicate 277 165 40
and predicate coverage by randomly generating coverage
adequate minimal test suites for these criteria. They also
Table 7. Coverage for RegExp V 1.0
reported that block test suites’ performance index was the
highest but with greatest variability whereas performance
4.2 Testing Changes Made to the Code: index for predicate test suites was lowest with least varia-
Regression Testing tion. This diagnosis was possible as they were able to com-
pute minimal test suites using the test coverage information
In this experiment four versions of Jakarta’s RegExp supplied by JavaCodeCoverage.
package [15] consisting of 143 test cases for versions 1.0,
1.1 and 1.2 and 215 test cases for version 1.3 are analyzed 5.2 Strengthen Static Analysis
for version changes at block coverage.
After code change, some test cases become invalid and Static analysis is a complementary approach to dynamic
some test cases need to be re-executed. Regression testing testing for bug identification. The code is scanned and dis-
was done so as to execute maximum portion of changed crepancies like null pointer referencing are identified. Ex-
code and achieve maximum coverage. haustive static analysis of the program through all paths is
Table 6 shows the RegExp package details and time not always possible, especially for a large software causing
taken to instrument RegExp version 1.0. Table 7 - 10 show incompleteness in the analysis. The complexity of the static
code coverage updates after each version change. For every analysis can be reduced by effective program partitioning.
next version, change impact on coverage is computed and Vipindeep and Jalote [29] described a technique which uses
recorded. Then the test cases from the older version are pri- coverage data from testing to remove the tested paths and
oritized according to maximum block change coverage and then statically analyzes the remaining code. This pruning
executed on the newer version. Results show that coverage of tested paths allowed a static analyzer to perform a more
stabilizes after executing around 30 percent of prioritized predictive and thorough analysis of the reduced code, which
test cases. led to considerable improvements in its effectiveness.

5 More on JavaCodeCoverage 5.3 Extending Coverage Measures

Besides providing coverage information for various From existing coverage information, some other related
code-elements, or supporting change management, the coverage measures like Condition/Decision, Relational-
tool can also be used for many other useful purpose like operator, and MCDC, can be measured without re-
strengthen static analysis, computing code coverage infor- instrumenting or parsing the class file. The desired informa-
mation for other coverage criteria, efficient de-bugging, tion can simply be obtained by making suitable queries to

Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)
0-7695-2755-8/07 $20.00 © 2007 8
 Proceedings of the 40th Hawaii International Conference on System Sciences - 2007

Coverage Type No.of Modified Added Coverage after ver- Coverage after priori-
units units units sion change tized test case execution
Class 14 11 3 79 100
Method 108 3 29 49 (53/108) 55 (60/108)
Block 801 130 74 40 (321/801) 58 (467/801)
Branch 764 143 65 36 (277/764) 55 (426/764)
Predicate 298 68 44 25 (74/298) 39 (116/298)

Table 8. Coverage after version change RegExp V 1.0 to RegExp V 1.1

Type No.of Modified Added Coverage after ver- Coverage after priori-
units units units sion change tized test case execution
Class 14 13 0 100 100
Method 108 5 3 49 (52/108) 55 (60/108)
Block 806 71 5 49 (396/806) 52 (423/806)
Branch 764 77 1 45 (349/765) 56 (433/765)
Predicate 298 3 0 25 (74/298) 39 (116/298)

Table 9. Coverage after version change RegExp V 1.1 to RegExp V 1.2

the database. Here is a brief description of how to compute 6 Summary and Future Work
the new coverage measures with the existing framework.
Condition/Decision coverage : Condition/Decision cover- Bug fixes, code re-factoring, software-upgrades are most
age requires every simple condition (predicate) to be evalu- frequent activities in software development because of
ated both as true and false as well as both the outcomes for which code keeps on changing. Due to frequent code
the conditional expression are to be exercised. This infor- changes, and also in a resource constraints environment,
mation can easily be obtained from the coverage tables in one needs to optimize the overall testing efforts which typ-
the database. ically include effective techniques for regression testing,
Relational operator coverage : An information table for test case prioritization, test case selection, test suite aug-
predicates contains opcode for every simple predicate. To mentation, and test suite minimization. These techniques
measure relational-operator coverage, only simple predi- mostly rely on code coverage information for various code-
cates with relational-operator opcodes are to be considered elements.
in the database queries. In this work, we presented a Java based coverage tool
MC/DC coverage: The Modified Condition Decision Cov- JavaCodeCoverage, which records class, method, block,
erage (MC/DC) reports coverage of occurrences in which branch and predicate coverage measures for the tests ap-
triggering individual simple condition should also trigger plied on a CUT. The tool performs the analysis at bytecode
(i.e. flip) the overall decision. The above information can level which is more effective than other source code instru-
be inferred from execution and information tables for pred- mentation tools. Additionally, the tool performs the code
icates. change analysis at the bytecode augmented with source
code change information and updates the test code cover-
5.4 Efficient De-bugging age information for the newer version of the code. This
information can be very useful for performing other testing-
The tool can effectively used for bug-place identification related activities. The tool uses a MySQL database to store
during testing. The GUI of the tool displays the code cover- code elements and test coverage information at the granu-
age for each test case exercised manually or in batch mode. larity of individual test case.
From the test code coverage information, one can quickly Using a set of experiments, we have demonstrated the
get to the potential buggy places. use of JavaCodeCoverage for Coverage testing, Code
change tracking and test case prioritization. Several other
5.5 Program Comprehension studies done using JavaCodeCoverage clearly highlight
the utility of our tool in other fields of software testing.
Open source database support of the tool makes it a very
The test cases execution traces can be used for generat-
useful for software testing research.
ing the behavior models like state diagrams for objects of
We further plan to strengthen the capabilities of
the CUT. This helps in program comprehension and testing JavaCodeCoverage by incorporating other coverage mea-
activities. Recently, Dallmeier et al. [17] used execution sures like MC/DC, simple path coverage and data flow cov-
traces from a set of test cases and constructed objects’ state erage. We are also considering more sophisticated code
models with pre-defined abstract states. Similar work is also change analysis such as control flow based inter-procedural
reported in [31]. analysis for coverage updates. The tool can be downloaded

Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)
0-7695-2755-8/07 $20.00 © 2007 9
 Proceedings of the 40th Hawaii International Conference on System Sciences - 2007

Type No.of Modified Added Coverage after ver- Coverage after priori- Coverage after additional
units units units sion change tized test case execution test cases execution
Class 14 13 0 100 100 100
Method 110 23 2 34(37/110) 55 (61/110) 58 (64/110)
Block 754 210 0 21 (161/754) 52 (393/754) 64 (487/754)
Branch 862 151 0 21(184/862) 37 (323/862) 42 (364/862)
Predicate 346 68 44 12 (41/346) 37 (128/346) 40 (139/346)

Table 10. Coverage after version change RegExp V 1.2 to RegExp V 1.3

from [5] with other information. [22] M. J. Harrold, J. A. Jones, T. Li, D. Liang, A. Orso, M. Pen-
nings, S. Sinha, S. A. Spoon, and A. Gujarathi. Regression
test selection for java software. In OOPSLA ’01: Proceed-
References ings of the 16th ACM SIGPLAN conference on Object Ori-
ented Programming, Systems, Languages, and Applications,
[1] Hansel. http://hansel.sourceforge.net/. pages 312–326, New York, NY, USA, 2001. ACM Press.
[2] Quilt. http://quilt.sourceforge.net/. [23] P. Jalote. An Integrated Approach to Software Engineering.
[3] Clover. http://www.cenqua.com/clover/. Addison-Wesley, 2002.
[4] EMMA. http://emma.sourceforge.net/. [24] J. A. Jones and M. J. Harrold. Test-suite reduction and pri-
[5] JavaCodeCoverage. http://www.cse.iitk.ac.in/
oritization for modified condition/decision coverage. IEEE
users/atulkg/.
[6] JCover. http://www.codework.com/JCover/product. Trans. Softw. Eng., 29(3):195–209, 2003.
[25] S. Kim, J. Clark, and J. McDermid. The rigorous generation
htm.
[7] Jcoverage. http://jcoverage.com/products/
of java mutation using hazop. In ICSSEA ’99 : Proceedings
jcoverage-commercial.html.
of the 12th International Conference on Software & Systems
[8] TCAT for Java. http://www.soft.com/TestWorks/. Engineering and their Applications, pages 9–10, 1999.
[9] Java Test Coverage. http://www.semdesigns.com/ [26] M. Lyu, J. Horgan, and S. London. A coverage analysis tool
Products/TestCoverage/JavaTestCoverage.html. for the effectiveness of software testing. IEEE Trans. on Re-
[10] InsectJ. http://insectj.sourceforge.net/. liability, 43(4):527–535, 1994.
[11] MySQL. http://www.mysql.com/. [27] G. Rothermel, M. J. Harrold, J. Ostrin, and C. Hong. An em-
[12] JUnit Home Page. http://www.junit.org . pirical study of the effects of minimization on the fault de-
[13] BCEL. http://jakarta.apache.org/bcel/. tection capabilities of test suites. In ICSM ’98: Proceedings
[14] GNU diff utility. http://www.gnu.org/software/
of the International Conference on Software Maintenance,
diffutils/.
[15] Jakarta RegExp Package. http://jakarta.apache.org/ pages 34–43, Washington, DC, USA, 1998. IEEE Computer
regexp/. Society.
[16] H. Agrawal, J. R. Horgan, E. W. Krauser, and S. London. [28] A. Srivastava and J. Thiagarajan. Effectively prioritizing
Incremental regression testing. In ICSM ’93: Proceedings tests in development environment. In ISSTA ’02: Proceed-
of the International Conference on Software Maintenance, ings of the 2002 ACM SIGSOFT International Symposium
pages 348–357, Washington, DC, USA, 1993. IEEE Com- on Software Testing and Analysis, pages 97–106, New York,
puter Society. NY, USA, 2002. ACM Press.
[17] V. Dallmeier, C. Lindig, A. Wasylkowski, and A. Zeller. [29] V. Vipindeep and P. Jalote. Efficient static analysis with path
Mining object behavior with adabu. In WODA ’06: Pro- pruning using coverage data. In WODA ’05: Proceedings
ceedings of the 4th Workshop on Dynamic Analysis, pages of the 3rd Workshop on Dynamic analysis, pages 1–6, New
17–24, New York, NY, USA, 2006. ACM Press. York, NY, USA, 2005. ACM Press.
[18] H. Do, S. Elbaum, and G. Rothermel. Infrastructure support [30] F. I. Vokolos and P. G. Frankl. Empirical evaluation of the
for controlled experimentation with software testing and re- textual differencing regression testing technique. In ICSM
gression testing techniques. In ISESE ’04: Proceedings of ’98: Proceedings of the International Conference on Soft-
the 2004 International Symposium on Empirical Software ware Maintenance, pages 44–53, Washington, DC, USA,
Engineering, pages 60–70, Washington, DC, USA, 2004. 1998. IEEE Computer Society.
[31] J. Whaley, M. C. Martin, and M. S. Lam. Automatic extrac-
IEEE Computer Society.
[19] S. Elbaum, A. G. Malishevsky, and G. Rothermel. Test case tion of object-oriented component interfaces. In ISSTA ’02:
prioritization: A family of empirical studies. IEEE Trans. Proceedings of the 2002 ACM SIGSOFT International Sym-
Softw. Eng., 28(2):159–182, 2002. posium on Software Testing and Analysis, pages 218–228,
[20] A. Gupta and P. Jalote. An experimental comparison of New York, NY, USA, 2002. ACM Press.
control flow based coverage criteria on seeded faults. In [32] W. E. Wong, J. R. Horgan, S. London, and A. P. Mathur. Ef-
TACAS ’06: Proceedings of the 12th. International Con- fect of test set minimization on fault detection effectiveness.
ference on Tools and Algorithms for the Construction and In ICSE ’95: Proceedings of the 17th International Confer-
Analysis of Systems, pages 365–368, Heidelberg, Germany, ence on Software Engineering, pages 41–50, New York, NY,
2006. Springer. USA, 1995. ACM Press.
[21] R. Gupta, M. J. Harrold, and M. L. Soffa. Program slicing-
based regression testing techniques. Software Testing, Verifi-
cation, and Reliability, 6(2):83–111, 1996.

Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)
0-7695-2755-8/07 $20.00 © 2007 10
View publication stats