Verification (cont.

Ch. 6

Black box testing

derives test cases from
specifications

Ch. 6

Systematic black-box
techniques
Syntax-driven testing
Decision table based testing
Cause-effect graph based testing

Ch. 6

Syntax-driven testing (1)

Consider testing an interpreter of
the following language
<expression> ::= <expression> + <term>|
<expression> - <term> | <term>
<term> ::= <term> * <factor> | <term> /
<factor> | <factor>
<factor> ::= ident | ( <expression>)

Ch. 6

Syntax-driven testing (2)

Apply complete coverage principle
to all grammar rules
Generate a test case for each rule of
the grammar
note, however that the test case might
also cover other rules

Note: the specification is formal, and

test generation can be automated
Ch. 6

Decision-table-based testing
The word-processor may present portions of text
in three different formats: plain text (p), boldface
(b), italics (i). The following commands may be
applied to each portion of text: make text plain
(P), make boldface (B), make italics (I), emphasize
(E), super emphasize (SE). Commands are
available to dynamically set E to mean either B or
I (we denote such commands as E=B and E=I,
respectively.) Similarly, SE can be dynamically set
to mean either B (command SE=B) or I (command
SE=I), or B and I (command SE=B+I.)

Ch. 6

*
*

*
*

SE

E=B

E=I

SE = B

SE = I

SE = B + I
action

*
p

Ch. 6

b,i

b,i

Cause effect graphs

B
I

O
R

P
A
N
D

E
E=B

SE
E=I
SE = B
SE = I

O
R

A
N
D

SE = B + I

The AND/OR graph represents the correspondence betw

causes and effects
Ch. 6

Coverage criterion
Generate all possible input
combinations and check outputs
May reduce the number by going
backwards from outputs
OR node with true output:
use input combinations with only one true
input

AND node with false output:

use input combinations with only one false
input
Ch. 6

Testing boundary conditions

Testing criteria partition input
domain in classes, assuming that
behavior is "similar" for all data
within a class
Some typical programming errors,
however, just happen to be at the
boundary between different classes
Ch. 6

10

Criterion
If x>y then S1; else S2; endif
After partitioning the input domain
D into several classes, test the
program using input values not only
inside the classes, but also at
their boundaries
This applies to both white-box and
black-box techniques
Ch. 6

11

Test Oracle
We must guarantee that our test is
accurate
We must verify, by running our
software on the selected test set,
that the results obtained do indeed
comply with the software goals, as
stated in the specification
Ch. 6

12

Test Oracle
Once we have selected a suitable test
set for an air traffic control system,
we should actually run the system by
applying it to real flights to verify
whether it does prevent accidents.

Ch. 6

13

Test Oracle
A test oracle determines whether a system
behaves correctly for test execution
An oracle is a mechanism used by
software testers and software engineers
for determining whether a test has passed
or failed. It is used by comparing the
output(s) of the system under test, for a
given test case input, to the outputs that
the oracle determines that product should
have.
Ch. 6

14

Purpose of Test Oracle

Sequential systems
Check functionality

Reactive Systems
Check functionality
Timing
Safety

Ch. 6

15

Complete Test Oracle

Generator
Provide predicted or expected results
for each test

Comparator
Compare predicted and obtained results

Evaluator
Determine whether the comparison
results are sufficient close to be a pass

Ch. 6

16

The oracle problem

How to inspect the results of test
executions to reveal failures?
How do we relate selected input
data the test cases with
corresponding outputs?
How do we verify the correctness
of the outputs we obtain?
Ch. 6

17

The oracle problem

Oracles are required at each stage
of testing
Automated test oracles are
required for running large amounts
of tests
Oracles are difficult to design - no
universal recipe

Ch. 6

18

Testing in the large

Module testing
testing a single module

Integration testing
integration of modules and subsystems

System testing
testing the entire system

Acceptance testing
performed by the customer
Ch. 6

19

Module testing
Scaffolding needed to create the
environment in which the module
should be tested
stubs
modules used by the module under test

driver
module activating the module under test

Ch. 6

20

Testing a functional module

PROCEDURE
UNDERTEST

STUB
CALL

DRIVER
CALL

ACCESSTONONLOCALVARIABLES

Ch. 6

21

Integration testing
Big-bang approach
first test individual modules in isolation
then test integrated system

Incremental approach
modules are progressively integrated
and tested
can proceed both top-down and bottom-up
according to the USES relation
Ch. 6

22

Integration testing and

USES relation
A
B

If integration and test

proceed bottom-up
only need drivers

C
D

Ch. 6

Otherwise, if we proceed
top-down only stubs are
needed

23

M1

Example

M2
M2,1

M2,2

M1 USES M2 and M2 IS_COMPOSED_OF {M2,1, M2,2}

CASE1
TestM1,providingastubforM2andadriverforM1
ThenprovideanimplementationforM2,1andastubforM2,2
CASE2
ImplementM2,2andtestitbyusingadriver,
ImplementM2,1andtestthecombinationofM2,1andM2,2
(i.e., M2) byusingadriver
Finally,implementM1andtestitwithM2,usingadriverfor
M1
Ch. 6
24

Testing OO programs
New issues

inheritance
genericity
polymorphism
dynamic binding

Open problems still exist

Ch. 6

25

Inheritance
Personnel

Consultant

Manager

Employee

Administartive_Staf

Ch. 6

Technical _Staf

26

How to test classes of the

hierarchy?
Flattening the whole hierarchy
and considering every class as a
totally independent component
does not exploit incrementality

Finding an ad-hoc way to take

advantage of the hierarchy

Ch. 6

27

A sample strategy
A test that does not have to be repeated
for any heir (subclass)
A test that must be performed for heir
class X and all of its further heirs
A test that must be redone by applying
the same input data, but verifying that
the output is not (or is) changed
A test that must be modified by adding
other input parameters and verifying that
the output changes accordingly
Ch. 6

28

Separate concerns in testing

Testing for functionality is not enough
Overload testing
Robustness testing
Regression testing
organize testing with the purpose of
verifying possible regressions of software
during its lifethat is, degradations of
correctness or other qualities due to later
modifications
Ch. 6

29

Testing concurrent and realtime systems

Nondeterminism inherent in
concurrency affects repeatability
For real-time systems, a test case
consists not only of input data, but
also of the times when such data
are supplied
Ch. 6

30

Analysis

Ch. 6

31

Symbolic execution
Can be viewed as a middle way
between testing and analysis
Executes the program on symbolic
values
One symbolic execution
corresponds to many actual
executions
Ch. 6

32

Example
Consider executing the following fragment
read(a); read(b);
x := a + b;
write(x);
Human analysis for the above program:
Let A and B be the first two values of the input file. They
are assigned to variables a and b, respectively. Thus, the
result of the assignment x := a + b is that x gets the value
A + B. Finally, A + B is printed.
Ch. 6

33

Symbolic x Numeric Execution

More generality: A and B represent any
actual values.
Allow us to conclude the program
computes the sum of the inputs
Executes the program on symbolic
values instead of numeric values

Ch. 6

34

Example(1)
read(a); read(b);
x := a +1; y := x *
b;
write(y);
Symbolic output:
read(a); read(b);
x:= a + 1;
x := x + b + 2;
write(x);
Symbolic output:

Ch. 6

35

Example(2)
1

x := y + 2

Consider executing the following fragment x > a

with x=X, y=Y, a=A
x := y + 2;
if x > a then
a := a + 2;
else
y := x + 3;
end if;
x := x + a + y;

a := a + 2

y := x + 3
3

x := x + a + y 4
Ch. 6

36

Example(2)
When control reaches the
conditional, symbolic values do not
allow execution to select a branch
One can choose a branch, and
record the choice in a path condition
Result:
<{a = A, y = Y + 5, x = 2 * Y + A + 7},
<1, 3, 4>, Y + 2 A>
execution path condition
path
Ch. 6

37

Example(2)
When control reaches the
conditional, symbolic values do not
allow execution to select a branch
One can choose a branch, and
record the choice in a path condition
Result:
<{a = A+2, y = Y, x = 2 * Y + A + 4},
<1, 2, 4>, Y + 2 > A>
execution path condition
path
Ch. 6

38

Symbolic execution rules (1)

symbolic state:
<symbolic_variable_values, execution_path, path_conditio

read (x)

removes any existing binding for x and adds

binding x = X, where X is a newly
introduced symbolic value

Write (expression)

Causes the symblic evaluation of the

expression and the binding
output(n) = computed_symbolic_value (n
counter associated with the output file)
initialized
to
1
and
automatically
incremented after each output statement.
Ch. 6

39

Symbolic execution rules (2)

x:= expression
construct symbolic value of expression,
SV; replace previous binding for x with x
= SV

After execution of the last statement

of a sequence that corresponds to
an edge of control graph, append
the edge to execution path
Ch. 6

40

Symbolic execution rules (3)

if cond then S1; else S2; endif
while cond loopendloop
condition is symbolically evaluated
eval (cond)

if eval (cond) true or false then

execution proceeds by following the
appropriate branch
otherwise, make nondeterministic
choice of true or false, and conjoin
eval (cond) (resp., not eval (cond)) to
the path condition
Ch. 6
41

Programs with arrays

Let any assignment to an array
value produce a new symbolic
value for the whole array and to
record suitable relationships
between the old value and the new
value.

Ch. 6

42

Programs with arrays

Let A1 be the symbolic value of array
a when statement a(i)= exp is
executed
Then, after execution of the
statement, a receives the new
symbolic value A2, denoted as A2 =
A1<i, exp>, a shorthand for
for all k if k = i then A2(k) = exp
else A2(k) = A1(k)
Ch. 6

43

Example
X denotes an integer array variable of length 5
1.
2.
3.
4.
5.
6.
7.
8.
9.

read( i ) ;
y := x( i );
x(3) := 9;
read( i );
x( i ) := 3 + y;
y := x( 2 );
read( i );
x( i ) := x( i ) 1;
y := y + x( i );

Ch. 6

44

Example(cont)
Let execution start in an initial state in which the
symbolic value X1 is bound to x.
1.
2.
3.
4.
5.

read( i ) ;
y := x( i );
x(3) := 9;
read( i );
x( i ) := 3 + y;

6.
7.
8.
9.

y := x( 2 );
read( i );
x( i ) := x( i ) 1;
y := y + x( i );

i = I1
y = X 1(I1)
x = X 2 where X2 = X1<3, 9>
i = I2
x = X3 where X3 = X2<I2, 3 + X1(I1)>
that is, X3(I2) = 3 + X1(I1)
y = X 3(2)
i = I3
x = X4 where X4 = X3<I3, X3(I3) 1>
y = X 3(2) + X4(I3)

Ch. 6

45

Example(cont)
The symbolic state expressed as a function of the
symbolic values read and xs initial value X1.
y = X1<3, 9><I2, 3 + X1(I1)>(2) +
X1<3, 9><I2, 3 + X1(I1)> <I3, X1<3, 9><I2, 3 +
X1(I1)>(I3) 1>(I3)

Ch. 6

46

Symbolic execution and

testing
The path condition describes the
data that traverse a certain path
Use in testing:
select path
symbolically execute it
synthesize data that satisfy the path
condition
they will execute that path
Ch. 6

47

Example (1)
found := false; counter := 1;
while (not found) and counter < number_of_items loop
if table (counter) = desired_element then
found := true;
end if;
counter := counter + 1;
end loop;
if found then
write ("the desired element exists in the table");
else
write ("the desired element does not exist in the table");
end if;

Ch. 6

48

Example (2)
1
2

3
found := true

6
5 counter := counter + 1
8
write "the desired
element does not
exist in the table"

9
write "the desired element exists in the table"
Ch. 6

49

Model checking
Correctness verification, in general, is
an undecidable problem
Model checking is a rather recent
verification technique based on the fact
that most interesting system properties
become decidable (i.e., algorithmically
verifiable) when the system is modeled
as a finite state machine
Ch. 6

50

Principles
Describe a given systemsoftware or
otherwiseas an FSM
Express a given property of interest as a
suitable formula
Verify whether the systems behavior does
indeed satisfy the desired property
this step can be performed automatically
the model checker either provides a proof that
the property holds or gives a counterexample in
the form of a test case that exposes the systems
failure to behave according to the property

Ch. 6

51

Properties and proofs

Property to be verified given through
a formula (in temporal logic)
For example, one can prove
there is always a computation that
allows a given process to enter the
critical region
there is no guarantee that the process
accesses the shared resource unless it
already owns it
Ch. 6

52

Debugging
The activity of locating and
correcting errors
It can start once a failure has been
detected
The goal is closing up the gap
between a fault and failure
memory dumps, watch points
intermediate assertions can help
Ch. 6

53