Symbolic Testing

Meenakshi D’Souza

International Institute of Information Technology

Bangalore.

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 1/1
 Program proving: Formal Methods

Program proving involves proving programs correct.

This is different from testing, where a program is tested only against
a set of test cases.
Test cases are not exhaustive, exhaustive testing is not possible.
Working of the program on other inputs is not tested.
Program proving involves formal methods.

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 2/1
 Formal Methods

Formal methods involve mathematical techniques that are based on

deriving formal models from a program or system under test.
Formal models have mathematically sound semantics.
Three techniques:
Model checking
Theorem proving
Pogram analysis

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 3/1
 Program testing vs. proving

Program testing and program proving can be considered as extreme

alternatives.
In testing, one can be assured that sample test runs work correctly by
carefully checking the results.
Correct execution of program for inputs not in the sample is still in
doubt.
In program proving, one formally proves that program meets its
specification (of correct behavior) without executing the program at
all.
Not all proofs can be automated and most proofs need to be done by
hand.

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 4/1
 Symbolic Execution and Program Testing

Symbolic execution is a practical approach between the two extremes

of program testing and program proving.
It is basically a testing technique that executes a program
symbolically for a set of inputs, instead of a sample input.
Each symbolic execution is equivalent to a large number of normal
test cases.

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 5/1
 Example 1

Consider the following program fragment:

1 Sum(a,b,c)
2 x = a+b ;
3 y = b+c;
4 z = x+y-b;
5 return(z);
6 end

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 6/1
 Normal execution of Sum on inputs

Normal execution of program Sum on inputs 1, 3, 5 is given in the table

below:

After stmt. x y z a b c
1 ? ? ? 1 3 5
2 4 – – – – –
3 – 8 – – – –
4 – – 9 – – –
5 returns 9

In the above table, – represents unchanged values and ? represents

undefined (uninitialized) values.

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 7/1
 Symbolic execution of Sum

After stmt. x y z a b c PC
1 ? ? ? α1 α2 α3 true
2 α1 + α2 – – – – – –
3 – α2 + α3 – – – – –
4 – – α1 + α2 + α3 – – – –
5 Returns α1 + α2 + α3

The above table represents symbolic execution of Sum(α1 , α2 , α3 ). Path

Condition/Constraint is abbreviated as PC.

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 8/1
 Example 2
Consider the following program:
1 int twice (int v ) {
2 return 2 ∗ v ;
3 }
4 void testme (int x, int y ) {
5 z = twice(y );
6 if(z == x) {
7 if(x > y + 10)
8 ERROR;
9 }
10 }
11 int main() {
12 x = sym input();
13 y = sym input();
14 testme(x, y );
15 return(0);
16 }
Meenakshi D’Souza (International Institute of Information Technology
Symbolic Testing
Bangalore.) 9/1
 Symbolic Execution Tree of testme

2 ∗ y == x
false true

x =0
y =1 x > y + 10
false true

x =2 x = 30
y =1 y = 15

ERROR

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 10 / 1
 Symbolic Execution

Symbolic execution uses symbolic values instead of concrete data

values as input.
Program variables are represented as symbolic expressions over the
symbolic input values.
Hence, output values computed by a program are expressed as a
function of symbolic input values.
Symbolic state σ maps variables to symbolic expressions.
Symbolic path constraint, PC, is a quantifier-free, first-order formula
over symbolic expressions.

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 11 / 1
 Symbolic execution in software testing

In software testing, symbolic execution is used to generate a test

input for each execution path of a program.
An execution path is a sequence of true and false, where true
(respectively, false) at the ith position in the sequence denotes that
the ith conditional statement encountered along the execution path
took the “then” (respectively, “else”) branch.
All the execution paths of a program can be represented using a tree,
called the execution tree.

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 12 / 1
 Symbolic Execution: Symbolic state σ

At the beginning of symbolic execution: σ is initialized to an empty

map.
At a read statement: symbolic execution adds the mapping assigning
the variable being read to its new symbolic variable, to σ.
At every assignment v = e: symbolic execution updates σ by
mapping v to σ(e), the symbolic expression obtained by evaluating e
in the current symbolic state.

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 13 / 1
 Symbolic execution: Path constraint PC

At the beginning of symbolic execution: PC is initialized to true.

At a conditional statement if(e) then S1 else S2 : PC is updated to
PC ∧σ(e) (“then” branch) and a fresh path constraint PC’ is created
and initialized to PC ∧¬σ(e) (“else” branch).
If PC (PC’) is satisfiable for some assignment of concrete to symbolic
values, then symbolic execution continues along the “then” (“else”)
branch with σ and PC (PC’).
Note: Unlike concrete execution, both the branches can be taken
resulting in two execution paths.
If any of PC or PC’ is not satisfiable, symbolic execution terminates
along the corresponding path.

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 14 / 1
 Symbolic execution

At the end of symbolic execution along an execution path of the

program, PC is solved using a constraint solver to generate concrete
input values.
Symbolic execution can also be terminated if the program hits an exit
statement or an error.
If the program is executed on these concrete input values, it will take
exactly the same path as the symbolic execution and terminate in the
same way.
These concrete inputs ensure that coverage is achieved in testing, by
taking different execution paths in the program.

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 15 / 1
 Example 2, re-cap
Consider the program in example 2 again:
1 int twice (int v ) {
2 return 2 ∗ v ;
3 }
4 void testme (int x, int y ) {
5 z = twice(y );
6 if(z == x) {
7 if(x > y + 10)
8 ERROR;
9 }
10 }
11 int main() {
12 x = sym input();
13 y = sym input();
14 testme(x, y );
15 return(0);
16 }
Meenakshi D’Souza (International Institute of Information Technology
Symbolic Testing
Bangalore.) 16 / 1
 Symbolic Execution: Example 2

After executing statements 12 and 13: σ = {x 7→ x0 , y 7→ y0 }, where

x0 and y0 are two initially unconstrained symbolic values.
After executing line 5: σ = {x 7→ x0 , y 7→ y0 , z 7→ 2y0 }.
After line 6: two instances of symbolic execution are created with
path constraints x0 = 2y0 and x0 6= 2y0 .
Similarly after line 7: two instances of symbolic execution are created
with (x0 = 2y0 ) ∧ (x0 > y0 + 10) and (x0 = 2y0 ) ∧ (x0 ≤ y0 + 10).
Solving the path constraints, we get three instances of symbolic
executions resulting in concrete test inputs {x = 0, y = 1},
{x = 2, y = 1} and {x = 30, y = 15}.

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 17 / 1
 Example with loops: Example 3

1 void testme inf() {

2 int sum = 0;
3 int N = sym input();
4 while (N > 0) {
5 sum = sum +N;
6 N = sym input();
7 }
8 }

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 18 / 1
 Symbolic execution with loops

The loop in example 3 has an infinite number of execution paths,

where each execution path is either a sequence of arbitrary number of
true-s followed by a false or a sequence of infinite number of true-s.
PC for the loop with a sequence of n true-s followed by a false is

(∧i∈[1,n] Ni > 0) ∧ (Nn+1 ≤ 0)

where each Ni is a fresh symbolic value.

σ at the end of the execution is {N 7→ Nn+1 , sum 7→ Σi∈[1,n] Ni }.
In general, symbolic execution of code containing loops or recursion
may result in an infinite number of paths if the termination condition
for the loop or recursion is symbolic.
In practice, one needs to put a limit on the search, e.g. a timeout, or
a limit on the number of paths, loop iterations or exploration depth.

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 19 / 1
 Satisfiability problem for PC

Symbolic testing is dependent on a constraint solver that takes a path

constraint and gives a set of input values to be used as test cases.
These input values are solutions to the path constraint, popularly
known as the satisfiability problem.
Satisfiability problem for constraints written in propositional logic is
NP-complete and is undecidable for first order logic.
Consequently, if the PC generated during a symbolic execution
contains formulas that cannot be (efficiently) solved by a constraint
solver, symbolic execution cannot generate test inputs.
This is considered as one of the key disadvantages of symbolic testing.

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 20 / 1
 Symbolic testing: Disadvantages

PC generated during symbolic execution cannot be solved by any

constraint solver.
The underlying program uses functions whose code is not available, in
which case the corresponding PC cannot be solved.
Many real-life programs have huge number of program paths,
resulting in several path constraints. Exploring all paths might be
impossible or infeasible.
Several techniques are available to alleviate this problem and generate test
inputs using symbolic testing.

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 21 / 1
 Concolic Testing

Directed Automated Random Testing (DART) or Concolic testing

performs symbolic execution dynamically, while the program is
executed on some concrete inputs.
Concolic: Concrete + symbolic.
Concolic testing maintains a concrete state and a symbolic state.
Concrete state is the normal notion of state of a program: values of
all variables in the program.

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 22 / 1
 Concolic execution: Example 2

Concolic execution generates a random input: {x = 22, y = 7} and

executes the program both concretely and symbolically.
Concrete execution takes the “else” branch; symbolic execution
generates the PC x0 6= 2y0 .
Concolic testing negates PC, solves x0 = 2y0 to get test input
{x = 2, y = 1}, forcing the program along a different execution path.
Next concolic execution generates PC (x0 = 2y0 ) ∧ (x0 ≤ y0 + 1).
This PC is negated, constraint (x0 = 2y0 ) ∧ (x0 > y0 + 10) is solved
to get test input {x = 30, y = 15}, error state is reached.
After three executions, all program paths have been explored.

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 23 / 1
 Concolic testing

Generate random input, execute program on random input.

Collect symbolic path constraints encountered along this execution.
Steer program execution along different execution paths:
Use constraint solver to infer variants of symbolic path constraints.
Generate inputs that drive program along different execution paths.
Repeat till all execution paths are explored/user-defined coverage
criteria is met or time budget expires.

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 24 / 1
 Example 4

Consider the function h in the program segment below:

int f(int x)
{ return 2 ∗ x; }
int h(int x, int y) {
if (x! = y )
if (f(x) == x+10)
abort(x);
return 0;
}

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 25 / 1
 Concolic testing on Example 4

Reaching the defective function h is difficult using a randomly

generated test input.
Using concolic, we can reach the error state in two execution steps.
Ramdomly generate inputs: x = 26, y = 34.
Predicates x0 6= y0 and 2x0 6= x0 + 10 are formed during the concolic
execution.
Concolic testing calculates the PC x0 6= y0 ∧ x0 = x0 + 10 by negating
the last predicate of the earlier PC, this leads to the error statement.

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 26 / 1
 Symbolic testing techniques: State of the art

Modern constraint solvers can solve complicated non-linear functions,

improving symbolic testing techniques.
Many years of research spanning over several papers have extended
these techniques to handle pointers, files, multi-threaded programs,
dynamic data structures etc.
Some popular tools are CUTE (for C,UIUC), jCUTE (for Java, at
UIUC), PEX (for .NET, available as a Visual Studio add-in), SAGE
(Microsoft).
Several other companies have developed tools using such techniques,
NASA, IBM, Fujitsu etc.

Meenakshi D’Souza (International Institute of Information Technology

Symbolic Testing
Bangalore.) 27 / 1