Toward Optimal MC/DC Test Case Generation

Sangharatna GODBOLEY† , Joxan JAFFAR∗ ,

Rasool MAGHAREH? , Arpita DUTTA∗

∗ National University of Singapore, Singapore

{joxan,arpita}@comp.nus.edu.sg
? Huawei Canada Research Centre, Canada

[email protected]
† National Institute of Technology Warangal, India

[email protected]

KLEE Workshop - September 2022

Toward Optimal MC/DC Test Case Generation 1/28

Accepted contributions

Accepted Contribution: Technical Track

Sangharatna Godboley, Joxan Jaffar, Rasool Maghareh & Arpita Dutta. “Toward optimal MC/DC test case generation.” In Proceedings of the 30th
ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), pp. 505-516, 2021, Aarhus, Denmark.
https://dl.acm.org/doi/10.1145/3460319.3464841

Accepted Contribution: Poster Track

Sangharatna Godboley, Joxan Jaffar, Rasool Maghareh & Arpita Dutta. “Toward optimal MC/DC test case generation.” In 30th ACM SIGSOFT
International Symposium on Software Testing and Analysis (ISSTA), 2021, Aarhus, Denmark.

Artifact Available
Badges obtained: Available, Functional, and Reusable
Sangharatna Godboley, Joxan Jaffar, Rasool Maghareh, & Arpita Dutta. CUSTOM-Interpolation: ISSTA artifact evaluation. In 30th ACM SIGSOFT
International Symposium on Software Testing and Analysis (ISSTA), Aarhus, Denmark. Zenodo: http://doi.org/10.5281/zenodo.4771439

Website: https://tracer-x.github.io/
Github: https://github.com/tracer-x/

Toward Optimal MC/DC Test Case Generation 2/28

Outline

1 Introduction

2 Survey

3 Proposed Idea

4 Experimental Evaluation

5 Conclusion

Toward Optimal MC/DC Test Case Generation 3/28

What is MC/DC?

Modified Condition/Decision Coverage (MC/DC)

MC/DC is the second strongest coverage criterion for unit testing. It requires linear
number of test cases wrt. the number of atomic conditions present in the program.
MC/DC requires all the following requirements [1]:
Each entry and exit point should get invoked.
Each predicate takes both possible truth values.
Each atomic condition(AC) in a predicate takes both possible truth values.
Each AC in a predicate shown as independent.
Why MC/DC?
According to RTCA standard of DO-178B/C[1], it is mandatory to achieve MC/DC
for Level A certificate of safety critical application.

Toward Optimal MC/DC Test Case Generation 4/28

Outline

1 Introduction

2 Survey

3 Proposed Idea

4 Experimental Evaluation

5 Conclusion

Toward Optimal MC/DC Test Case Generation 5/28

A Survey with Industrial Practitioners

Figure: Which Domain? Figure: Which automatic tools?

Toward Optimal MC/DC Test Case Generation 6/28

A Survey with Industrial Practitioners

Figure: Automatic vs
Manual! Figure: Which strategy?

Toward Optimal MC/DC Test Case Generation 7/28

Outline

1 Introduction

2 Survey

3 Proposed Idea

4 Experimental Evaluation

5 Conclusion

Toward Optimal MC/DC Test Case Generation 8/28

Proposed Idea

Figure: The Overall Architecture of our Framework

Toward Optimal MC/DC Test Case Generation 9/28

Sequence Generator (SG)

An Example Predicate: if ((b<0 || c<0) && d < 0 ) { ... }

SG generates five short-circuited sequences{S1 : 101}, {S2 : 102}, {S3 : 211},

{S4 : 212}, and {S5 : 220} where 1, 2, 0 represent True, False, and Don’t Care for
the atomic conditions.

Table: Short-circuit Truth Table for Predicate

B C D Output
S1 T X T T
S2 T X F F
S3 F T T T
S4 F T F F
S5 F F X F

Toward Optimal MC/DC Test Case Generation 10/28

Resource Annotator

k=0

<1> b≥0
b<0 k = k * 10 + 2
k = k * 10 + 1 <2>
k = k * 10 c<0 Table: Paths from sequences
k = k * 10 + 1 c≥0
k = k * 10 + 2 Id Seq Path
<3> d≥0 k = k * 10
d<0 k = k * 10 + 2
S1 101 1→3→T
k = k * 10 +1 S2 102 1→3→F
<F>
<T> S3 211 1→2→3→T
S4 212 1→2→3→F
<postDom>
S5 220 1→2→F
assert( k ∉ {101, 102,
211, 212, 220 } )

Figure: Annotated CFG for the Predicate

Toward Optimal MC/DC Test Case Generation 11/28

Employing DSE

Algorithm 1 Test Case Generation Using DSE

Input: Upd_LLVM_IR,
Output: Test_Suite
1: Test_Suite ← ∅
2: errorPaths ← Run_DSE (Upd_LLVM_IR)
3: for each errorPath in errorPaths do
4: Input_Values ← extractInputValues(errorPath)
5: Test_Suite ← Test_Suite + Input_Values
6: end for

Toward Optimal MC/DC Test Case Generation 12/28

Taming the Path Explosion

Figure: Exploration of Symbolic Execution Tree in Non-pruning DSE vs. Pruning DSE

Toward Optimal MC/DC Test Case Generation 13/28

Example for Standard Interpolation

x=0
x ≠ 1 ∧ x ≠16 ∧
x ≠ 13 ∧ x ≠ 28 <1a>

Consider the program: x ≠ 1 ∧ x ≠16 <2a> <3a> x ≠ 13 ∧ x ≠ 28

x += 12 skip
x = 0; x≠13 ∧ x ≠28 <4a> <4b>
if (b1) x += 12;
if (b2) x += 15; x ≠ 13 <5a> <6a> x ≠ 28
assert(x != 28); x += 15 skip

<7a> <7b>
assert( x ≠ 28)

Figure: SET of the program

Toward Optimal MC/DC Test Case Generation 14/28

Custom Interpolation on top of TracerX [5, 7]

Algorithm 2 Custom Interpolation

1: function P RE(Annotation , ChildInt)
2: if BASE _ BUG THEN RETURN (κ 6∈ SeqVals − {Valκ })
3: END IF
4: IF BASE _ NO _ BUG THEN RETURN (κ 6∈ SeqVals)
5: END IF
6: C HILD I NT ≡ (κ 6∈ Set )
7: ParentSet ← {}
8: FOR EACH s IN Set DO
9: ParentSet ← ParentSet + PRE C OND (s , A NNOTATION )
10: END FOR
11: ParentSet ← R EMOVE N ON I NTEGRALS (ParentSet )
12: RETURN (κ 6∈ ParentSet )
13: END FUNCTION
14: FUNCTION J OIN (PATH I NT _1 , PATH I NT _2)
15: PATH I NT _1 ≡ (κ 6∈ Set _1)
16: PATH I NT _2 ≡ (κ 6∈ Set _2)
17: RETURN (κ 6∈ Set _1 ∪ Set _2)
18: END FUNCTION

Toward Optimal MC/DC Test Case Generation 15/28

Custom Interpolation

Example
if (a < 0)b = 3;

if ((b < 0||c < 0)&&d < 0) {...}

Figure: The Main Example Program

Toward Optimal MC/DC Test Case Generation 16/28

Custom Interpolation

Figure: The Execution Tree of the Main Example Program

Toward Optimal MC/DC Test Case Generation 17/28

Outline

1 Introduction

2 Survey

3 Proposed Idea

4 Experimental Evaluation

5 Conclusion

Toward Optimal MC/DC Test Case Generation 18/28

Experimental Setup
Used Setup:
We experimented on Intel Core i7-6700 3.40 GHz Linux Box with 32GB RAM,
and a timeout of 3600 seconds.
The raw experimental results can be accessed at [28].
Experimental Evaluation:
1 Main Experiment
Our Method (CUSTOM) v/s CBMC
2 Supplementary Experiment
No Interpolation (KLEE) and Standard Interpolation v/s CUSTOM Interpolation

Used Data set:

Table: Programs Experimented.
Type psyco RERS(12-20) RERS(19-Industry) zodiac Total
Numbers 14 181 14 1 210

Toward Optimal MC/DC Test Case Generation 19/28

KLEE v/s Standard Interpolation v/s CUSTOM Interpolation
No Interpolation (KLEE) v/s Standard Interpolation (TracerX)
Forward Symbolic Execution to find feasible paths (Similar to KLEE).
Intermediate execution states preserved (Unlike KLEE).
Half interpolants are generated by backward tracking and Full interpolants
generated by merging half interpolants.
Full interpolants used for subsumption at similar program points.

Standard Interpolation (TracerX) v/s CUSTOM Interpolation (Paper’s Contribution)

CUSTOM is designed to discover the MC/DC sequences and generate test
cases for those sequences unlike TracerX which is used only in case of safety.
Symbolic execution typically stops the path on witnessing a bug. In contrast,
CUSTOM modifies the interpolant and continue the path.
In CUSTOM, we generate a weakest precondition (WP) interpolant on the
ghost variable (κ) alongside the standard interpolant on the rest of the
variables.
Toward Optimal MC/DC Test Case Generation 20/28
Main Experiment Results
Groups
1 Both CUSTOM and CBMC terminate.
2 CUSTOM terminates, but CBMC does not
3 CBMC terminates, but CUSTOM does not
4 Neither of the tools terminate

Table: Experimented Programs

Groups Group1 Group2 Group3 Group4 #Total
#Programs 91 71 5 43 210

Table: Main Results (Total 710.4K sequences)

Tool Proved Sequences UnProven Optimal
(Feasible + Infeasible) Sequences Programs
CBMC 104.7K 605.7K 96/210 (45.71%)
CUSTOM 531.2K 179.2K 162/210 (77.14%)
Toward Optimal MC/DC Test Case Generation 21/28
Main Experiment Results

CBMC CUSTOM
CBMC CUSTOM
10000
10000

7500

MC/DC UnProven Sequences

7500
MC/DC Proved Sequences

5000 5000

2500 2500

0 0
0 50 100 150 200 0 50 100 150 200

Program Program

Figure: Scatter chart for MC/DC Proved Figure: Scatter chart for MC/DC
Sequences UnProven Sequences

Toward Optimal MC/DC Test Case Generation 22/28

Supplementary Experimental Results

Figure: Comparison of Execution Time in No Interpolation (KLEE) [2], Standard

Interpolation [3] vs. CUSTOM Interpolation

Toward Optimal MC/DC Test Case Generation 23/28

Major Takeaway

Symbolic Execution(SE) is designed to perform either of these two:

1 Bug Finding
2 Program Verification
In contrast, we used it to discover MC/DC sequences and generate MC/DC
specific test cases.
Our CUSTOM interpolation technique is clever enough to prune the sub trees
which contain already discovered MC/DC sequences.
Our algorithm, if it terminates, generates an optimal set of MC/DC test cases.

Toward Optimal MC/DC Test Case Generation 24/28

Outline

1 Introduction

2 Survey

3 Proposed Idea

4 Experimental Evaluation

5 Conclusion

Toward Optimal MC/DC Test Case Generation 25/28

Conclusion

We have surveyed and found that in industrial practice, automatic MC/DC test
generation is woefully inadequate and most practitioners rely on manual effort.
Our algorithm, if terminates, generates an optimal set of MC/DC test cases.
We compared CUSTOM against CBMC, the only practical method available
which address large programs.
A comprehensive experimental evaluation shows our implementation to
perform at a higher level.

Toward Optimal MC/DC Test Case Generation 26/28

References ≫ I

1 Kelly J., Hayhurst and Dan S., Veerhusen and John J., Chilenski and Leanna K., Rierson. A Practical Tutorial on Modified Condition/Decision
Coverage. NASA Langley Technical Report Server (2001).

2 Cadar, C., Dunbar, D. and Engler, D.R., 2008, December. KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems
Programs. In OSDI (Vol. 8, pp. 209-224).

3 Jaffar J., Murali V., Navas J.A., Santosa A.E. (2012) TRACER: A Symbolic Execution Tool for Verification. In: Madhusudan P., Seshia S.A. (eds)
Computer Aided Verification. CAV 2012. Lecture Notes in Computer Science, vol 7358. Springer, Berlin, Heidelberg

4 Kroening D., Tautschnig M. (2014) CBMC-C Bounded Model Checker. In: Abraham E., Havelund K. (eds) Tools and Algorithms for the Construction
and Analysis of Systems. TACAS 2014. Lecture Notes in Computer Science, vol 8413. Springer, Berlin, Heidelberg

5 Jaffar J., Maghareh R., Godboley S., Ha XL. (2020) TracerX: Dynamic Symbolic Execution with Interpolation (Competition Contribution). In: Wehrheim
H., Cabot J. (eds) Fundamental Approaches to Software Engineering. FASE 2020. Lecture Notes in Computer Science, vol 12076. Springer, Cham

Toward Optimal MC/DC Test Case Generation 27/28

References ≫ I

6 Jaffar J., Godboley S., and Maghareh R. (2019). Optimal MC/DC test case generation. In Proceedings of the 41st International Conference on
Software Engineering: Companion Proceedings (ICSE’19). IEEE Press, 288-289. DOI:https://doi.org/10.1109/ICSE-Companion.2019.00118

7 Jaffar J., Maghareh R., Godboley S., Ha XL. (2020) TracerX: Dynamic Symbolic Execution with Interpolation. KLEE 2020 (2nd International KLEE
Workshop on Symbolic Execution) Imperial College London, South Kensington Campus

8 SV-COMP Benchmarks: Verification Tasks, https://github.com/sosy-lab/sv-benchmarks/tree/master/c/psyco, Dec 2017

9 RERS: http://rers-challenge.org/, Jun, 2018

10 Artifact Workbook for CUSTOM-Interpolation, https://doi.org/10.6084/m9.figshare.13650242.v1

Toward Optimal MC/DC Test Case Generation 28/28