Data Flow Testing

The data flow testing method selects test paths of a program

according to the locations of definitions and uses of variables in
the program. A number of data flow testing strategies have been
studied and compared (e.g., [FRA88], [NTA88], [FRA93]).

To illustrate the data flow testing approach, assume that each

statement in a program is assigned a unique statement number
and that each function does not modify its parameters or global
variables.

For a statement with S as its statement number,

DEF(S) = {X | statement S contains a definition of X}
USE(S) = {X | statement S contains a use of X}

If statement S is an if or loop statement, its DEF set is empty and

its USE set is based on the condition of statement S. The
definition of variable X at statement S is said to be live at
statement S' if there exists a path from statement S to statement
S' that contains no other definition of X.

A definition-use (DU) chain of variable X is of the form [X, S, S'],

where S and S' are statement numbers, X is in DEF(S) and
USE(S'), and the definition of X in statement S is live at statement
S'.

One simple data flow testing strategy is to require that every DU

chain be covered at least once. We refer to this strategy as the
DU testing strategy. It has been shown that DU testing does not
guarantee the coverage of all branches of a program. However, a
branch is not guaranteed to be covered by DU testing only in rare
situations such as if-then-else constructs in which the then part
has no definition of any variable and the else part does not exist.
1
In this situation, the else branch of the if statement is not
necessarily covered by DU testing.

Data flow testing strategies are useful for selecting test paths of a
program containing nested if and loop statements. To illustrate
this, consider the application of DU testing to select test paths for
the PDL that follows:

To apply the DU testing strategy to select test paths of the

control flow diagram, we need to know the definitions and uses of
variables in each condition or block in the PDL. Assume that
variable X is defined in the last statement of blocks B1, B2, B3,
B4, and B5 and is used in the first statement of blocks B2, B3, B4,
B5, and B6. The DU testing strategy requires an execution of the
shortest path from each of Bi, 0 < i 5, to each of Bj, 1 < j 6.
(Such testing also covers any use of variable X in conditions C1,
2
C2, C3, and C4.) Although there are 25 DU chains of variable X,
we need only five paths to cover these DU chains. The reason is
that five paths are needed to cover the DU chain of X from Bi, 0 <
i 5, to B6 and other DU chains can be covered by making these
five paths contain iterations of the loop.

If we apply the branch testing strategy to select test paths of

the PDL just noted, we do not need any additional information. To
select paths of the diagram for BRO testing, we need to know the
structure of each condition or block. (After the selection of a path
of a program, we need to determine whether the path is feasible
for the program; that is, whether at least one input exists that
exercises the path.)

Since the statements in a program are related to each other

according to the definitions and uses of variables, the data flow
testing approach is effective for error detection. However, the
problems of measuring test coverage and selecting test paths for
data flow testing are more difficult than the corresponding
problems for condition testing.

3
 Aug 25,
Introduction 05

Product Development:
- Ineffective: organization focus on cost.
- Effective: make the process efficient

Define Quality:
- is conformance to request.
- is fitness for intended users.
-

Definitions of Software Quality:

- IEEE:
-

Summary of Defining Quality::

-
-

The Evolution of Quality:

1. Past: Quality Control
2. Now: Quality Assurance
3. Future: Quality Engineering

(Past) (Now) (Future)

Quality Control Quality Assurance Quality Engineering

4
 The Evolution of Quality

1. Quality Control:
- Quality Control is defined as a set of procedures designed to ensure that quality
standards and processes are adhered to, and that the final product meets or exceeds
the required technical and performance requirements, so it keeps quality at an
acceptable level by rejecting unacceptable.
- Prevent unacceptable products from being released to the customer.
- Problems:
o Not reduce defect. Fail
o Not improve process
o Not better product Request Development PC Inspection Pass
o Not motivate improvement

2. Quality Assurance:
- Covers all activities from design, development, production, installation, servicing and
documentation, this introduced the rules: "fit for purpose" and "do it right the first
time".
- Providing evidence that the quality function is being performed adequate.
- IEEE: A planned and systematic pattern of all actions necessary to provide adequate
confidence that the product conforms to established technical requirement.
- System of methods and procedures used to assure that the software product meet.
- Pros:
o More effective than quality because emphasize on development processing.
o Fix problems during and before development process
o Improve process reduce defect in a last manner
- Cos

Process & Design

Standards

Standard of Quality
o Separate from other process improve and software development activity.
o Motivation to improve is inconsistent
o Cost of making people monitors others.

3. Quality Engineering:
- Similar to Quality Assurance, but responsibility on everyone.
- Problems result in process change, not punishment of people.
- Finding error is good keeping them leak to customers.
- Appreciates that a competitive process is the way to remain a competitive.
- Measurement are used so decisions based on face (in addition to institution)
- Pride in quality
- Build quality in product through development
- Less adversarial
- Motivation
- Flexible to change the process in response to a problem
o Understand the problem & cause & process
- Knowledge is the foundation of successfully quality engineering.

5
 A Perspective on Testing

Definition:
The process of executing a program with the intend of finding errors.

Testing Objective:
- To improve quality by finding error.
- To provide confidence in the dependability of software.

Related Terms:
- Error: A measure of the difference between the
actual and the ideal. It occurs in some hardware or Fault, Failure, and Error: (Scenario)
software component.
- Fault: A condition that causes a system to fail in Consider for instance a system running on a
performing its required function. Fault is a multi-processor architecture: a fault in one
manifestation of an error. processor might cause it to crash (i.e., a
- Failure: The inability of a system or component to failure), which will be seen as a fault of the
system. Therefore, the ability of the system
perform a required function according to its
to function even in the presence of the failure
specifications. Failure is a departure of a system
of one processor will be regarded as fault-
from the service required.
tolerance instead of failure-tolerance. (Not all
faults cause immediate failure.)
Static vs. Dynamic:
- Static Analysis is analysis done on source code
without actually executing it.
o Defects like dead-code (i.e., unreachable code) can only be detected through
careful inspection.
o Similarly, non-conformance to organization-wide coding conventions can also
be detected through inspection process only.
o Syntax errors are caught by static analysis
- Dynamic Analysis is analysis done as a program is executing and is based on
intermediate values that result from the programs execution.
o E.g., A division by 0 error is caught by dynamic analysis.

Verification vs. Validation:

-
VerificationValidation
Actual user n

Verification:
o Are we building the right product?
o Does the software fulfill its requirement?
- Validation:
o Are we building the product right?
o Is implementing consistent with its specification?

6
 A Perspective on Testing (Cont.)

If testing doesn't find any defect?

- Low quality testing
- High quality software
- Metrics are needed
- Quality assessment (what percentage of defect captured, what percentage remain).
- Risk management.

Test Cases:
- Valuable (as much as source program)
- The information need to be in a test case:
o Input: Test Case ID

Preconditions
The actual inputs. Purpose
o Expected Output:
Postconditions Preconditions
Actual outputs.
o Optional: Inputs
ID
Purpose Expected Outputs
Execution history
- Testing Oracles: A testing oracle is any program, process, Postconditions
or body of data that specifies the expected outcome of a
set of tests. Oracles are often defined as a set of Execution History
input/expected outcome pairs. Date Result Version Run By

Relationship between specified, implemented, and tested behaviors:

Specification
(expected) 5 Program
2 6
(observed)
1
4 3

7
Test Cases
(verified)
- Region 2 and 5: There may be specified behaviors that are not tested.
- Region 1 and 4: Specified behaviors that are tested.
- Region 3 and 7: Test cases that correspond to unspecified behaviors.
- Region 2 and 6: There may be programmed behaviors that are not tested.
- Region 1 and 3: Programmed behaviors that are tested.
- Region 4 and 7: Test cases that correspond to unprogrammed behaviors.
- Note: We need to make the region where these sets all intersect (Region 1) as large
as possible.

7
 A Perspective on Testing (Cont.)

Testing strategy:
- Unit Testing: A unit is the smallest testable piece of software
(e.g., function).
- Component Testing: A component is an integrated aggregate
of one or more units (e.g., module).
- Integration Testing: This testing is used to demonstrate that
a combination of successfully tested components has bugs.
- System Testing: A system is a very large component.
System testing includes testing for:

Integration

System
Testing
o performance

Testing
o security
o system recovery from failure

Component
Testing
Concept map of testing:

Testing
Unit
Tester

Test implementation
of execution

Test set SW under test ? Result

Test case
Requirement
generation

assumption method Test purpose

Testing vs. Debugging:

- Debugging: The activity by which faults are identified and rectified.
- The purpose of testing is to show that a program has bugs.
- The purpose of debugging is to find the faults that led to the programs failure and to
design and implement the program changes that correct the faults.
- Testing is a demonstration of failure or apparent correctness.
- Debugging is a deductive process.
- Testing proves a programmers failure.
- Debugging is a programmers vindication.
- Testing can be automated to a large extent.
- Automatic debugging is still a dream.
- Much of testing can be done without design knowledge (by an outsider).
- Debugging is impossible without detailed design knowledge (by an insider).

8
 Testing Techniques

1. Functional (Black Box) Testing:

Input Output

- Program is treated as a black box.

- Program is subjected to inputs, and its outputs are verified for conformance to
specified behavior.
- Implementation details do not matter.
- Takes a users point of view.
- In principle, can detect all bugs in an infinite amount of time.
- The program test cases are based on specifications.
- Test planning can begin early in the software process; test case generation can be in
parallel.
- A complete functional test would consist of subjecting a program to all possible input
streams.

2. Structural (White Box) Testing:

- Aims at exercising the different control and data structures used in the program.
- Criteria are precise as they are based on program structures (i.e., are quite precise).
- Looks at implementation details.
- Takes a developers point of view.
- Is inherently finite but cannot detect all faults.
- Derivation of test cases according to program structure. Knowledge of the program is
used to identify additional test cases.
- Objective is to exercise all program statements (not all path combinations).
- One should design enough tests to ensure that every path is executed at least once.

Which testing technique is better?

- According to Venn diagrams, neither is sufficient.
- If all specified behavior have not been implemented, structural testing will never
been able to recognize this.
- If a program (pgm) implements behaviors that haven not been specified, this will
never be revealed by functional testing.

9
 Sep 1, 05
1. Testing Techniques: Functional Testing

10
11
Single Fault Assumption vs. Multi Fault Assumption:
- Single Fault Assumption: specifies that exactly one component is malfunctioning and
the cause of the problem.
- Multi Faults Assumption: more than one component are causing of the problem.

Dependent Variables vs. Independent Variables:

- Dependent Variables: such as month Feb 29 days depends on the year.

Physical vs. Logic:

- Some techniques require logic recognition of data dependencies such as equivalence
class. This requires more thought and more judgment.
- Other techniques are domain-based that have no recognition of data or logical
dependencies such as Boundary Value Testing.

12
 1.1 Functional Testing: Boundary Value Testing

1. Boundary Value Analysis (BVA):

- Objective: to use the values at their: Variable
values
o Minimum value
o Just above the min (min+1)
BVA test cases for a function of one variable
o Just below the max (max-1)
o Maximum value V2
- Rationale: errors occur more frequency at extreme values.
- Boundary value can be character, ordinary value,
- Requires 4n+1 test cases.
- Variables have discrete, bounded values.
- This technique is more effective with independent variables while with
dependent variables is not effective as independent variables. V1
- Works well when a program is a function of independent var represent
boundary physical quantity. BVA test cases for a function of two variables
- When no bounds exist, we have to create artificial bounds, as in the triangle problem.
Creating the lower bound and the maximum bound (called MAXINT in some
language).

2. Robustness Testing:
- An extension of BVA: include value below the minimum and above the maximum
values.
o Negative Testing: how does the program handle errors in input?
o May not be possible with some strongly by typed languages, such as compiler.
- In addition to the five values of variable, exceed the extremes with maximum and
minimum.
- Useful for exception handling to test behavior for invalid input values.
- Stress input boundary. Variable
- Acceptable response for invalid input? values
- Leads to exploratory testing (test hackers)
- Discover hidden functionality. Robustness test cases (one variable)

3. Worst-Case Testing: V2
- Do not consider single fault assumption.
- Explore situation with > 1 variables in their extreme values.
- More through; useful for critical software.
- Number of test cases is 5n.

4. Robust Worst-Case Testing: V1

- Discard single fault assumption.
- Allow the input values to simultaneously approach and exceed their boundaries.
Worst-Case Testing (Two variables)
- 7n test cases.

Thought on BVA:
- Does not require much thought.
- Usually assumes the variables are independent. Variables relationships not
considered.
- Values at certain boundaries may not have a meaning.

13
 Boundary Value Test Cases for ('Triangle' Problem)

Test Cases:

- Boundary Value Analysis:

- Worst Case:

Boundary Value Test Cases for ('NextDate' Problem)

14
Test Cases:

- Worst-Case:

Leap Year:

In the Gregorian calendar, a normal year consists of 365 days. Because the
actual length of a sidereal year (the time required for the Earth to revolve
once about the Sun) is actually 365.25635 days, a "leap year" of 366 days is
used once every four years to eliminate the error caused by three normal
(but short) years.

1) Any year that is evenly divisible by 4 is a leap year: for example, 1988,
1992, and 1996 are leap years. However, there is still a small error that must
be accounted for. To eliminate this error, the Gregorian calendar stipulates
that

2) a year that is evenly divisible by 100 (for example, 1900) is a leap year
only if it is also evenly divisible by 400. For this reason, the following years
ARE NOT leap years
1700, 1800, 1900, 2100, 2200, 2300, 2500, 2600 because they are evenly
divisible by 100 but NOT by 400. The following years ARE leap years
1600, 2000, 2400 because they are evenly divisible by both 100 and 400.
-

15
 Boundary Value Test Cases for ('Commission' Problem)

From one corner:

Test Cases:

- Output BVA:

16
 Sep 8, 05
1.2 Functional Testing: Equivalence Class Partitioning

Introduction:
- Two motivations:
o A sense of complete testing.
o Avoid redundancies.
- Several variation:
o Robust: consider invalid input.
o Weak: single fault assumption.
- Partition a set of values into a collection of mutually disjoint subsets (union is the
entire set).
o Completeness
o Disjointedness ensures a form of non-redundancy.
- Testing idea: identify test cases by using one element from each set.
- An Example: In triangle problem: {<1}, {1 200}, and {>200}
- Key idea in EC Testing:
o Wise selection of equivalent classes.
o Assuming f(X1, X2) where:
aX1d, with intervals [a,b), [b,c), [c,d]
cX2g, with intervals [e,f), [f,g]
Invalid values of X1, X2 are: X1< a, X2>d, X2<e, X2>g

Equivalence Class (EC) Partitioning:

- The sets in the partitions of the domain (input, output).
o Determine the boundaries
o Determine the equivalences.
- Motivation:
o A sense of completeness
o Minimize redundancies
- If a given set of values all cause a program to behave in exactly the same way, then
that is a candidate for an EC if a test case in one EC detects on defect, all other
test cases will detect the same defect.
- Input domain test cases focus on input, it may does not provide confidence about
calculation.

Some comments on EC Testing:

- The weak forms are not very comprehensive.
- If error condition are a high priority, robust form should be considered
- EC is strengthened with a hybrid approach with BVA
- Strong EC presumes the variables are independent. When dependence occurs DT is
appropriate.
- Several tries may be needed to choose the right ECs.

17
 Equivalence Class Test Cases for ('Mortgage Company' Problem)

Problem Description:
- Writes mortgage for people with monthly income of $1,000/month to $30,000/month.
- Writes a single mortgage for 1-4 homes
- Makes mortgages on condos, town houses, singled dwelling (not on duplex, mobile
home).
- Will make mortgage only for a person (not to corporations, trust, partnership)

Conditions:
- If an input represents a continuous range of values Select one valid & two invalid
classes.

invalid valid invalid

Income $
1,000 30,000
- If an input takes on discrete values (from a possible set) Select one valid & two
invalid classes.
invalid valid invalid

# of houses
-1 0 1 2 3 4 5 6

- If input conditions is specified as a set of valid values Select one valid EC (continue
all valid members) & one invalid EC.
Valid
Single mortgage Invalid
Town houses Duplex
Condo

- If an input specifies a "must be" condition Select one invalid EC (representing the
"must be" condition) & one invalid class that does not include the "must be"
condition.
Invalid
Corporations Valid
Partnership Person
Trust

Example of Test Cases:

Output Dwelling Applicant # of Monthly

Result Type Dwelling Income
OK Condo Person 2 $5,000
not OK Duplex Partnership 8 $100
not OK Town home Corporatio 9 $5,500
n

18
 Equivalence Class Test Cases for ('Triangle' Problem)

The Output Equivalence Classes:

Test Cases:

- Weak Normal EC:

- Weak Robust EC:

- Strong Robust EC (from one corner of the cube:

19
 Equivalence Class Test Cases for ('NextDay' Problem) Version 1

The Input Equivalence Classes:

- Valid Classes:

- Invalid Classes:

Test Cases:

- Weak Normal EC & Strong Normal EC:

- Weak Robust EC:

- Strong Robust EC (from one corner of the cube:

20
 Equivalence Class Test Cases for ('NextDay' Problem) Version 2

More careful selection of ECs:

- Reduce granularity: focus on more specific treatment of variables; How? Consider
input data:
o If not last day of month, next date increments day.
o If at the end of a month, day is set to 1, month is incremented.
o If at the end of a year, month, and day are set to 1, year is incremented.
o Leap year? When is the last day of a month.

The Output Equivalence Classes:

Test Cases:
- Weak Normal EC:

- Strong Normal EC (from one corner of the cube:

21
 Equivalence Class Test Cases for (Commission' Problem) Version 1

The Input Equivalence Classes:

- Valid Classes:

- Invalid Classes:

Test Cases:

- Strong Robust EC:

22
 Equivalence Class Test Cases for (Commission' Problem) Version 2

The Output Equivalence Classes:

Output Range EC Test Cases:

23
 Sep 15,
1.3 Functional Testing: Decision Tables 05

Introduction:
- Useful for capturing and presenting complex logical decision making.
- Prototype:
Rules
(IF) Conditions
(THEN) Actions

- Example:
R8 R7 R6 R5 R4 R3 R2 R1
T T T T F F F F Valid Size:
T T F F T T F F Valid quantity
T F T F F F T F Sufficient
Funds
Yes No No No No No No No Buy

- It can have multiple actions/conditions/rules.

- Well suited when logical relationship exists between variables.
- Choice of conditions greatly expands the table (look at the Triangle Problem below).
- Number of the rules:
o With n conditions imply 2n rules.
o If action sets of two rules are identical, there must be at least one condition
that allows two rules to be combined with a "-" entry. Look at the NextDay
example:
Rules 1-3 involve D1, D2, D3 for 30-day months Combine!
Rules 6-9 involve D1, D2, D3, D4 for 31-day months Combine!
- In testing cases:
o Conditions Input
o Action Output
o Rules Test cases

Summary of Decision Tables:

- Appropriate for:
o Prominent if-then-else
o Logical relationship among variables
o Cause and effect relations
o High cycloramic complexity
- Do not scale up well (2n combinations)
o Algebraically factor large tables
o Look for repeating patterns
- The combination will not work with Commission Problem because there is no
relationship between the input variables (Independent variables).

24
 Decision Tables Test Cases for ('Triangle' Problem)

The Rules:
c1: a,b,c form a triangle (NEEDS to be expanded)
c1: a<b+c?
c2: b<a+c?
c3: c<a+b?
c4: a=b?
c5: a=c?
c6: b=c?
a1: not a triangle
a2: scalene
a3: isosceles
a4: equilateral
a5: impossible

Decision Table:

R1 R1 R9 R8 R7 R6 R5 R4 R3 R2 R1
1 0
T T T T T T T T T T F C1
T T T T T T T T T F - C2
T T T T T T T T F - - C3
F F F F T T T T - - - C4
F F T T F F T T - - - C5
F T F T F T F T - - - C6
x x x A1
x A2
x x x A3
x A4
x x x A5

25
 Decision Tables Test Cases for ('NextDay' Problem) Version 1

The Rules:
M1 = {mon | mon has 30 days}
M2 = {mon | mon has 31 days}
M3 = {mon | mon is Feb}
D1 = {day | 1 <= day <= 28}
D2 = {day | day = 29}
D3 = {day | day = 30}
D4 = {day | day = 31}
Y1 = {year | year = 2000}
Y2 = {year | a leap year}
Y3 = {year | a common year}

Decision Table:

26
 Decision Tables Test Cases for
('NextDay' Problem) Problem: Rule 8 has unknown Version 2
decision

The Rules:
M1 = {mon | mon has 30 days}
M2 = {mon | mon has 31 days; not Dec}
M3 = {mon | mon is Dec}
M4 = {mon | mon is Feb}
D1 = {day | 1 <= day <= 27}
D2 = {day | day = 28}
D3 = {day | day = 29}
D4 = {day | day = 30}
D5 = {day | day = 31}
Y1 = {year | year is a leap year}
Y2 = {year | year is a common year}

Decision Table:

Reduced Decision Table:

Test Cases:

27
 Sep 22,
Retrospective of Functional Tests 05

Effort:
- Domain-based techniques have no recognition of data or logical dependencies
- The equivalent class techniques pay attention to data dependencies and to the
function itself
- The decision table techniques requires consider both data and logical dependencies
- Trade-off between test identification effort and test execution effort

Efficiency:
- The intuitive notion is that a set of test cases is just right (no gap , no redundancy)
- The gap is the possibility of untested functionality.

Effectiveness:
- Mandate a method, use it to generate test cases, and then run the test cases
- Know how effective a set of test cases is for finding faults present in a program

Appropriate Choices for Functional Testing:

- Look at the following table to see the appropriate testing techniques.
o Only Strong EC and Decision table can handle dependent variables.
o Decision table works only with dependent variables.
o Boundary tests work with physical variables not logical.

28
 A Brief Review of Graphs

Introduction:
- Directed graph G(V,E)
o Finite set V={n1, n2, , nm} of nodes.
o Finite set E={e1, e2, , ep} of nodes.
o Where each Ek={ni, nj} is an ordered pair
o (start node, terminal node)

- For example:
V={n1, n2, n3, n4},
E= {e1, e2, e3}={(n1,n2), (n2,n4), (n1,n3)}

e2
e1 n2 n4

n1
e3
n3

- Nodes of degrees
o Indeg (ni) = number of distinct edges that have ni as their terminal node.
o Outdeg (ni) = number of distinct edges that have ni as their start node.
o Src node : a node with indeg = 0
o Sink node: a node with outdeg = 0
o Transfer node: a node with (indeg != 0 & outdeg != 0).
- Directed path:
o a sequence of edges such that for any adjacent pairs of ei, ej, the terminal
node of ei is the start node of ej
- Connectness: for two nodes ni, nj :
o 0-connected: iff there is no path between ni & nj
o 2-connected: iff there is a path between ni & nj
o 3-connected: iff there is a path between ni to nj or and a path between nj to ni.
- Strongly connected graph:
o all pairs of nodes are 3-connected.
- Program graphs:
o Given a pgm in an imperative language, its graph is a directed graph in which
nodes are entire statements or fragment of statements and edges represent
flow of control.

29
 A Brief Review of Graphs (Cont.)

Graphs for ctrl structures:

Case

Sequence

Until (do/while)

While

- Connected compounds are collapsed into individual nodes

- A sequence of 2-connected nodes and a node with outdeg 2 can be mapped into a
single node:

30
 n1

n2

n3

- Areas bound by edges are called a region. You will see example later.
- For compound conditions in a pgm, a separate node is created for each condition, so
if (c1 and c2) needs to be divided to two conditions if (c1) and if (c2). For example:
(Look at handout 8).

31
 A Brief Review of Graphs (Cont.)

McCabe's Cyclomatic Complexity Metrics:

- Gives complexity
- Give the number of linearly independent paths. (in each branch, pick up one of them,
then go over the other in the next round) number of test cases

- Formula:
V(G) = #edges - #nodes +2

- You may count the designated node with/without its edges.

- A complex example when there is a loop required repeating the cycle many times.

Example:

F
A
G
D 20 times
return
C X
H
E
I

- Assuming N is the number of cycles that required the loop to be repeated:

o When N=1, 5 paths
o When N=2, 52 = 25 paths
o When N=3, 53 = 125
o When N=20, 51 + 52 + + 520 = number of test cases need to be generated
to cover all the paths, which is impossible.

32
 Quiz on Graphs

EECS814: Software Quality Assurance

Professor Hossein Saidian
Quiz 1 September 22, 2005 Name:

1. Draw a flow graph for the following code segment. For simplicity, assume that P1, P2, P3
are simple predicates:

S1;
if (P1) {
for (i = 1; P2; i++) {
if(P3)
S2;
}
If (P3) S3;
} else
S4;
S5;

33
 2. Testing Techniques: Structural Testing

Introduction:
- Complements functional testing.
- Based on implementation
- Powerful mathematical foundations

Representing graphs that are widely used for testing:

- At the unit-level:
1. Program Graph (pgm graph):
Given a program written in an imperative programming language, its program
graph is a direct graph in which nodes are program statements, and edges
represent flow of control (there is an edge from node i to node j iff the
statement corresponding to node j can be executed immediately after the
statement corresponding to node i)

- At the system-level behavior:

2. FSM (Finite State Machine)
is a directed graph in which states are nodes and transitions
are edges. Sources and sink states become initial and terminal
nodes, paths are modeled as paths, and so on. All the real-
time extensions of structured analysis use some form of finite
state machine, and nearly all forms of object-
oriented analyses require them.

3. Petri nets
A Petri net is a bipartite directed graph (P, T, In,
Out), in which P and T are disjoint sets of nodes, and In
In P T
and Out are sets of edges, where , and
Out T P
. Petri nets can do what FSM does, but not the other way round
(such as parallelism).

Structural Testing Techniques

- Control Flow-Based Testing:
1. Statement Coverage
2. Branch Coverage
3. Condition Coverage
4. Branch / Condition Coverage
5. Multi-Condition Coverage
6. Independent Path Coverage
- Data Flow-Based Testing:
1. All Definitions: each definition to a reachable use
2. All Uses: definition to each reachable use
3. All def-use: each definition to each reachable use

34
 2.1 Structural Testing: Control Flow-Based Testing

1. Statement Coverage (aka line coverage):

- Each executable statement is executed at least once
- Advantage: reliable simple
- Disadvantage: insensitive to certain ctrl structures
- Example:

int *p = NULL;
if (cond) p=&v;
*p = 100;

- Without a test case to force a false evaluation, the code is considered covered.
- Also, does not report whether loops reach termination condition; only whether the loop
body is executed.
- Statement coverage is blind to other classes of errors:

1. Assignment blindness: due to assignment of a value to a variable, error does not

propagate. For example:

if (cond)
x=1;
if (x+y>=0) // should be (y>0)
...

2. Equality blindness: due to an equality check of a variable, error does not

propagate. For example, we choose a=0 to check the following statement:

if (a==0)
...
if (b>3) // should be (a+b>3)
...

3. Self-blindness: conditional covers ups the error. For example, we choose x=0 to
check the following statement:

if(x<1) // should be (2*x<1)

...

35
 Sep 29,
2. Branch Coverage: 05

- Each branch of a decision must be executed

- Stronger than stmt coverage
- The compound conditions consider one node unlike path coverage. (Look at the example
in Branch / Condition Coverage section).

Subsumption Definition:

Criteria C1 subsumes criteria C2, iff:

- For all programs p being tested
with specifications s
- All test sets t
- t is adequate according to C1 for
- Path subsumes
testing p withbranch
respect to s implies
- Path subsumes statement
that t is adequate according to C2
for testing p with respect to s
- Example: (Test cases to satisfy Branch converge)

if ((x>1) && (y==1))

z=1;
else
z=0;
if ((x==2) || (y>1))
z++;
else
z=-1;

36
 Example on Statement Converge and Branch Converge ('Count' Program)
(Dependable Software Systems (Path Testing) @ Spiros Mancoridis)

The programming code for Count:

/* COUNT
This program counts the number of characters and lines in a text file.
INPUT: Text File
OUTPUT: Number of characters and number of lines.
*/

1 main (int argc, char *argv [])

2 {
3 int numChars = 0;
4 int numLines = 0;
5 char chr;
6 FILE *fp = NULL;
7
8 if (argc < 2)
9 {
10 printf ("\nusage: %s <filename>", argv[O]);
11 return (-1);
12 }
13 fp = fopen (argv[1], "r");
14 if (fp == NULL)
15 {
16 perror (argv[l]); /* display error message */
17 return (-2);
18 }
19 while (!feof(fp))
20 {
21 chr = getc (fp); /* read character */ .>
22 if (chr == '\n' ) /* if carriage return */ y
23 ++numLines;
24 else
25 ++numChars;
26 }
27 printf("\nNumber of characters = %d", numchars);
28 printf("\nNumber of lines = %d" , numLines) ;
29 }

Flowgraph for Count:

1 8 11 14 17 19 22 23 24 26 29

37
 Example on Statement Converge and Branch Converge (Cont.)

Test cases to satisfy Statement Coverage for Count:

i
g
j
F F F
a b c d e k
1 8 11 14 17 19 22 23 24 26 29
T T T T
F
h f

PATHS PROCESS LINKS TEST CASES

a b c d e f g h i j k l INPUT OUTPUT

ab None Usage: COUNT <filename>

agc Invalid
Input Filename
Error Message

aghdjkli Input
File with one character and no Carriage
Number
Return
of characters
at the end =
of1the lin
Number of lines = 0

aghdefli Input file

with
no characters and
Number
one carriage
of characters
return = 0
Number of lines = 1

Test cases to satisfy Branch Coverage for Count:

PATHS DECISIONS TEST CASES

8 14 19 22 INPUT OUTPUT

ab T None Usage: COUNT <filename>

agc F T Invalid Input Filename Error Message

aghdjkli Input
F FileFwith T,F
one character
F and no Carriage Return
Numberatof
the
characters
end of the
= line
1
Number of lines = 0

aghdefli F F Input
T,F
file with
T no characters and one
Number
carriage
ofreturn
characters = 0
Number of lines = 1

38
3. Condition Coverage:

- Each condition with a decision must assure all possible values.

- Potentially stronger than branch coverage.
- NOTE: you need at least to execute the true part.
- Example:

if ((x>1) && (y==1))

Z=1;
else
Z=0;

4. Branch / Condition Coverage:

- Try to cover all conditions and branches:

- Example:

if (x<y && x>2)

a++;
else
a=0;
if (x>0 && y>0)
b++;
else
b=0;

x<y
T
x>2
T

a=0 a++

y>0 Paths not covered in branch/condition coverage

T
x>0
T

b=0 b++

39
 Example on Statement Converge, Branch Converge, and Condition
Converge

Programming Code:

if (a > 3)
x++;
if (b==3)
y=0;

Flowgraph:

Path #4 Path #3Path #2 Path

a>3 T #1

x++

b=3 T

y=0

Test Cases:

Test Cases Paths Coverage Type

(a=6, b=3) 4 Statement Coverage
(a=1, b=1) 1
Branch Coverage
(a=6, b=3) 4
(a=6, b=3) 4 Condition Coverage
(a=1, b=1) 1
Branch/Condition Coverage
(a=6, b=3) 4

40
5. Multi-Condition Coverage:

- subsume condition & branch coverage

- Every combination of possible conditional values within a decision is chosen, so if we
have if(C1 & C2 & C3 & C4), we will have 16 possibilities.
- usually better than branch cover
- Differences between condition, branch, condition/branch, multi-condition coverage:

if (a && b)
X();
else
Y();

1 2
Condition Branch
Coverage3 Coverage

1. Condition Coverage:
if (True && False) Y();
if (False && True ) Y();
2. Branch Coverage:
if (True && True ) X();
if (False && True ) Y();
3. Branch / Condition Coverage:
if (True && True ) X();
if (False && False) Y();
4. Multi-conditions Coverage:
if (True && True ) X();
if (True && False) Y();
if (False && True ) Y();
if (False && False) Y();

6. Basic Path Testing:

- The cyclomatic complexity of a connected directed graph represents the # of linearly

independent cycles.
- To determine a set of basic paths:
1. Draw a flow graph
2. Pick a baseline (corresponding to normal execution)
3. Succeeding paths: retrace the baseline until a decision point is reached; flip
the decision and continue as much as the baseline as possible
4. Repeat until all decision have been flipped

41
 Example on Condition Converge and Multi-Condition Converge

Programming Code:

if (a>0 && c==1)

x++;
if (b==3 || d<0)
y=0;

Flowgraph:

a>0
T
c=1
T

x++

b=3 Paths not covered in branch/condition coverage

T
d<0 T

y=0

Test Cases:

1. Condition Coverage: Test the True and False condition of each expression.
Values Test
Statement
d c b a Case
s
#
x++; y=0; -1 True 1 True 3 True 1 True 1

2. Multi-Condition Coverage: Covers all the possibilities.

Vales Test
d c b a Case
Statements
#
x++; y=0; -1 True 1 True 3 True 1 True 1
1 False 1 True 3 True 0 False 2
-1 True 2 False 4 False 1 True 3
1 False 2 False 4 False 0 False 4

42
 Example on Path Converge ('Average' Program)

The programming code for Average:

/* AVERAGE
This procedure computes the average of 100 or fewer numbers that lie
between bounding values; it also computes the sum and the total number
valid.
*/

PROCEDURE average;

F IV
10
F
I 12 V 11

II 13
VI 1
F T
III
2
F
3

4
T
5

8 7

6 Regions
Predicate
INTERFACE RETURNS average, total.input, total.valid;
INTERFACE ACCEPTS value, minimum, maximum;
TYPE value[l:100] IS SCALAR ARRAY;
TYPE average, total.input, total.valid, minimum, maximum, sum IS SCALAR;
TYPE i IS INTEGER;

i = 1;
1 total.input = total.valid = 0;
sum = 0;
2 3
DO WHILE value[i]<>-999 AND total.input<100

4 increment total.input by 1
5 6
IF value[i]>=minimum AND value[i]<=maximum
THEN increment total.valid by 1;
7 sum = s sum + value[i]
ELSE skip
8 ENDIF
increment i by 1:
9 ENDDO
10

43
12
 IF total.valid>0
11 THEN average=sum / total.valid;
1 ELSE average =-999;
13 ENDIF
END average

Test cases to satisfy Path Coverage for Average:

# of independent path = # of complexity = # of regions = #e-#n+2 = of

predicates +1 = 6

Test Cases Paths #

(i=-999, total.valid=1) 1, 2, 10, 11, 13 1
(i=-999, total.valid=0) 1, 2, 10, 12, 13 2
(i=1, total.input=1000) 1, 2, 3, 10, 3
1, 2, 3, 4, 5, 6, 7, 8, 9, 2
(i=1, total.input=10, maximumiminimum) 4

1, 2, 3, 4, 5, 6, 8, 9, 2,
(i=1, total.input=10, maximum<iminimum) 5

(i=1, total.input=10, maximum<i<minimum) 1, 2, 3, 4, 5, 8, 9, 2, 6

44
 Example on Path Converge ('TotalPrice' Program)

The programming code for TotalPrice:

1 rushcharge = 0 1
1 if nextday = 'Y' T
2 rushcharge = 14.50 2
3 endif 3
3 tax = amount * 0.08 T
3 if amount >= 1000
4 shipcharge = amount * .06 + rushcharge 5 4
T
5 else if amount >= 200
6 shipcharge = amount * .08 + rushcharge 7 6
7 else if amount >= 100 T
8 shipcharge = 10 + rushcharge 9 8
9 else if amount >= 50 T
10 shipcharge = 9 + rushcharge
11 else if amount >= 25
11 10
T
12 shipcharge = 7 + rushcharge
13 else 13 12
13 shipcharge = 5 + rushcharge
14 endif 14
14 total = amount + tax + shipcharge

Test cases to satisfy Path Coverage for TotalPrice:

# of independent path = 7

AMT NextDa Path

#
y
AMT < 25 'Y' 1, 2, 3, 5, 7, 9, 11, 13,
1
14
(e.g. AMT 1000 'N' 1, 3, 4, 14
2
1500)
(e.g. 500) 200 AMT < 'N' 1,3,5,6,14
3
1000
(e.g. 150) 100 AMT < 200 'N' 1,3,5,7,8,14 4
(e.g. 75) 50 AMT < 100 'N' 1,3,5,7,9,10,14 5
(e.g. 35) 25 AMT < 500 'N' 1,3,5,7,9,11,12,14 6
(e.g. 15) AMT < 25 'N' 1,3,5,7,9,11,13,14 7

45
 Conclusion Example about Control Flow-Based Coverage

The programming code:

function P return INTEGER is 1

begin 2
X, Y: INTEGER; 3
READ (X); READ (Y); 4
while (X > 10) loop 5
X := X - 10; 6
exit when X = 10; 7
end loop; 8
if (Y < 20 and then X mod 2 = 0) 9
then 10
Y := Y + 20; 11
else 12
Y := Y - 20; 13
end if; 14
return 2 X + Y; 15
end P;

Flowgraph:

6 7
F 10
T
2,3,4 5 9 T 9` 14
F

12

Test Cases:

Test Cases Paths # Coverage Types

{4,5,6,7,9,9',10,14
(x=20, y=10) 1 Statement Coverage
}
{4,5,6,7,9,9',10,14
(x=20, y=10) } 1 Branch Coverage
(NOTE: 9 and 9` is consider
(x=15, y=30) {4,5,6,7,5,9,12,14 2 one node)
}
{4,5,6,7,9,9',10,14
(x=20, y=10) } 1 Condition Coverage
(x=5, y=30) {4,5,9,12,14} 2 (NOTE: you need at least to
(x=21, y=10) {4,5,6,7,5,9,9`,12, 3 execute the TRUE part)
14}
{4,5,9,12,14}
{4,5,6,7,5,9,12,14
(x=5, y=30) 1
}
(x=15, y=15) 2
{4,5,6,7,5,9,9`,12,
(x=21, y=10) 3 Path Coverage
14}
(x=20, y=10) 4
{4,5,6,7,9,9`,10,14
(x=15, y=30) 5
}
{4,5,6,7,8,12,14}

46
 Oct 20,
2.2 Structural Testing: Data Flow-Based Testing 05
2.2.1 Define/Use Testing

Introduction:
- Many failures involve because of execution of an incorrect variable definition:
o Incorrect assignment
o Predicate is faulty
o Definition is missing
- Explore sequence event related to the data (variable) state

- Examples:

Read (V1, V2, Vn)

o Definition V1, V2, Vn
Annotations:
Write (V1, V2, Vn)
o C-USE of V1, V2, Vn
- d: defined, created, initialized
Method(f1, f2, fn) (data decln, assignment, input)
o Definition of each formal parameter. - k: Killed, undefined, released
while B do S - u: used for something
o Predicate use of variables in B o computation
for (V=e1 to e2) o predicates
o Compute use of each variable in e1 & e2 - v: expression
o Definition of V o C-USE of all variables
o Predicate use of V in expression
if B then S1; else S2 o Definition of v
o Predicate use (P-USE) in B
case e1
o Predicate use (P-USE) in e1

Dataflow Anomalies:
- Suspicious and data in a program
o Not errors to be detected by the compiler
o Represent potential problem or bad programming style

- Example of anomalies:
o live variable problem; a variable is defined twice before its used:
X = a;
:
: no use of X
X = b;
o A variable is defined but never used
o A variable is used but never defined
o Unreferenced labels
o Unreachable code

47
Normal Dataflow Condition:

- du
- kd Dataflow Anomalies
- uu
- _d (last definition) dd define followed by define *
- k_ (ok, last thing done) dk defined then killed
- u_ (ok, last use) ku killed then used
__ nothing happens
_k nothing to kill
Example on Normal Dataflow Condition: _u undefined
d_ defined but not used
u_ normal unless the variable
should have been killed

* if a definition comes after

another definition, it does not
mean bad all the times.

Path 1 Path 2

Define X

Use Y
Kill Z

Define X Kill Z
Use X Use X
Use Z Define Z

Define Y
Use Z

Use Y
Use Z
48

Kill Y
Define Z
Dataflow
RespectDefinition:
to X
- Path 1 (the concern about define X in
- 2G(P)
nd program graph
node)
o
o _d P program
(normal)
k, oo
u, ddd
V P's Variables
(concern)
u
- PATH o (P)du (normal) d, k
- Patho2 Paths in P u d
- DEFo (v,n) <variable, node>
du (normal) d

Respectoto v
Y V
- Path1

o n G(P) u, k
o _u (bad)
o ud (ok) at n
o v defined
- USEo (v,n) Types of USE(v,n)
du(normal)
oo ukv (normal)
V P-use: # of out-degree 2
- Path 2 C-use: # of out-degree 1
oo _un (bad)
G(P)
oo udv is(ok)
used at n
o dk(definition-use)
- du-path (probable error)

Respectoto A
Z path in PATH(P) such that for some v V, there is DEF(v,m) & USE(v,n) such
- Path1 that m & n are initial and final nodes of the path.
o _k(definition-clean
- dc-path (problem) path)

o ku (problem)
oo uuFor(normal)
a v V, dc-path is a path PATH (P) with initial & final nodes DEF(v,m),
o ud USE(v,n), such that no other node in the path is a defining node of V.
(normal)
- Path 2
o _k (problem)
Example 1:
o kk (probable problem)
o kd (normal)
1 i :=0;
o du (normal)
2 while (i<2) DU-Paths USE DEF Variable
o uu (normal) <1-2> 2 1
3 {
4 cout << "in loop"; <1-2-3-4-5> 5 1
i
5 i++; <5-6-2> 2 5
6 } Disallowed 5 5

Example 2:
Test
1 sum :=0; DU-Paths USE DEF Variable
Cases *
2 read(n); n=0 <2-3-4> 4 2 n
3 i=1; n=5 6 5 num
4 while (i <= n) {
5 read (num);
n=5 6 1
6 sum += num n=0 9 1
n=5 sum
7 i++; Disallowed 6 6
8 } n=5 9 6
9 print(sum) n=0 4 3
n=5 7 3
n=5 i
4 7
n=5 Disallowed 7 7
* Note: Two sets of data cover all the DU-Paths:
n=0 and n=5

49
 Example on Define/Use Testing ('Commission' Program)

The programming code for Commission:

1 Program Commission (INPUT,OUTPUT)

2 Dim locks, stocks, barrels As Integer

3 Dim lockPrice, stockPrice, barrelPrice As Real
4 Dim totalLocks, totalStocks, totalBarrels As Integer
5 Dim IockSales, stockSales, barrelSales As Real
6 Dim sales, commission As Real

7 lockPrice = 45.0 Highlight DEF/USE Variables

8 stockPrice = 30.0
9 barrelPrice = 25.0 DEF: Italic & Underline
10 totalLocks = 0 P-use: Underline
11 totalStocks = 0 C-use: Italic
12 totalBarrels = 0

13 Input(locks)
14 While NOT(1ocks = -1) 'loop condition uses -1 to indicate end of data
15 Input(stocks, barrels)
30-33 34 7-13

35-37 38 14

39 15-19

20-28
40

29

41-42
16 totalLocks = totalLocks + locks
17 totalStocks = totalStocks + stocks
18 totalBarrels = totalBarrels + barrels
19 Input(stocks)
20 EndWhile

21 Output("Locks sold: ", totalLocks)

22 Output("Stocks sold: ", totalStocks)
23 Output("Barre1s sold: ", tota1Barrels)
24 lockSales = lockPrice * totalLocks
25 stockSales = stockPrice * totalStocks
26 barrelSales = barrelPrice * totalBarrels
27 sales = lockSales+stockSales+barrelSales
28 Output("Total sales: ", sales)

29 If (sales > 1800.0)

30 Then
31 commission = 0.10 * 1000.0
32 commission = commission + 0.15 * 800.0
33 commission = commission + 0.20*(sales-1800.0)
34 Else If (sales > 1000.0)
35 Then

50
36 commission = 0.10 * 1000.0
37 commission = commission + 0.15*(sales-1000.0)
38 Else commission = 0.10 * sales
39 EndIf
40 EndIf

41 Output("Commission is $", commission)

42 End Commission

51
 Oct 27,
Example on Define/Use Testing ('Commission' Program) (Cont.) 05

DEF/USE Variables in Commission:

USE at node DEF of node Variable
24 7 lockPrice
25 8 stockPrice
26 9 barrelPrice
16,21,24 10,16 totalLocks
17,22,25 11,17 totalStocks
18,23,26 12,18 totalBarrels
14,16 13,19 Locks
17 15 Stocks
18 15 Barrels
27 24 lockSales
27 25 stockSales
27 26 barrelSales
28,29,33,34,37,38 27 Sales
32,33,37,41 31,32,33,36,37,38 commission

DEF/USE Paths:
Du-Paths DEF USE DEF Variable #
Clear Node Node
P1=<7,8,9,10,11,,21,22,23,24> Yes 24 7 lockPrice 1
P2=<8,9,10,11,12,,22,23,24,25> Yes 25 8 stockPrice 2
P3=<9,10,11,12,13,,23,24,25,26> Yes 26 9 barrelPrice 3
P4=<10,11,12,13,14,15,16> Yes 16 10 totalLocks 4
P5=<P4,17,18,19,20,21> No 21 10 totalLocks 5
P6=<P5,22,23,24> No 24 10 totalLocks 6
Note: Ignoring the possible repetition of the
while-loop
P7=<16,16> or Yes 16 16 totalLocks 7
<16,17,18,19,20,14,15,16>
Disallowed as du-paths
P8=<16,17,18,19,20,21> Yes 21 16 totalLocks 8
P9=<P8,22,23,24> 24 16 totalLocks 9
P10=<11,12,13,14,15,16,17> Yes 17 11 totalStocks 10
P11=<P10,18,19,20,21,22> No 22 11 totalStocks 11
P12=<P11,23,24,25> No 25 11 totalStocks 12
Disallowed as du-paths Yes 17 17 totalStocks 13
P14=<17,18,19,20,21,22> No 22 17 totalStocks 14
P15=<P14,23,24,25> No 25 17 totalStocks 15
P16=<12,13,14,15,16,17,18> Yes 18 12 totalBarrels 16
P17=<P16,19,20,21,22,23> No 23 12 totalBarrels 17
P18=<P17,24,25,26> No 26 12 totalBarrels 18
Disallowed as du-paths Yes 18 18 totalBarrels 19
P20=<18,19,20,21,22,23> Yes 23 18 totalBarrels 20

52
 Example on Define/Use Testing ('Commission' Program) (Cont.)

P21=<P20,24,25,26> Yes 26 18 totalBarrels 21

P22=<13,14> Yes 14 13 Locks 22
P23=<P22,15,16> Yes 16 13 Locks 23
P24=<19,20,14> Yes 14 19 Locks 24
P25=<P24,15,16> Yes 16 19 Locks 25
P26=<15,16,17> Yes 17 15 Stocks 26
P27=<P26,18> Yes 18 15 Barrels 27
P28=<24,25,26,27> Yes 27 24 lockSales 28
P29=<25,26,27> Yes 27 25 stockSales 29
P30=<26,27> Yes 27 26 barrelSales 30
P31=<27,28> Yes 28 27 Sales 31
P32=<P31,29> Yes 29 27 Sales 32
P33=<P32,30,31,32,33> Yes 33 27 Sales 33
P34=<P32,34> Yes 34 27 Sales 34
P35=<P34,35,36,37> Yes 37 27 Sales 35
P36=<P32,38> Yes 38 27 Sales 36
P37=<31,32> Yes 32 31 commission * 37
P38=<31,32,33> Yes 33 31 commission 38
Not feasible n/a 37 31 commission 39
P40=<31,32,33,40,41> Yes 41 31 commission 40
Disallowed as du-paths Yes 32 32 commission 41
P42=<32,33> Yes 33 32 commission 42
Not feasible n/a 37 32 commission 43
P44=<32,33,40,41> Yes 41 32 commission 44
Not feasible n/a 32 33 commission 45
Disallowed as du-paths Yes 33 33 commission 46
Not feasible n/a 37 33 commission 47
P48=<33,40,41> Yes 41 33 commission 48
Not feasible n/a 32 36 commission 49
Not feasible n/a 33 36 commission 50
P51=<36,37> Yes 37 36 commission 51
P52=<36,37,39,40,41> Yes 41 36 commission 52
Not feasible n/a 32 37 commission 53
Not feasible n/a 33 37 commission 54
Disallowed as du-paths Yes 37 37 commission 55
P56=<37,39,40,41> Yes 41 37 commission 56
Not feasible n/a 32 38 commission 57
Not feasible n/a 33 38 commission 58
Not feasible n/a 37 38 commission 59
P60=<38,39,40,41> Yes 41 38 commission 60
* Note: In commission variable, you may consider only the real ones

53
DU-Path Test Coverage Metrics (Rapps-Weyuker dataflow metrics)

- All-Defs Criterion:

Annotations:
For every variable v V, T contains def-clear paths from every defining
node of V to a use of V
P Program
G(P) Graph of P
- All-Uses Criterion:
V Variables in P
For every variable v V, T contains def-clear paths from every defining T Paths in P
node of v to every use of v, and to successor node of each USE(v,n)

- All-P-Uses/Some-C-Uses:

For every variable v V, T contains def-clear paths from every defining node of v to
every predicate use of v; and if a definition of v has no P-uses, a def-clear leads to at
least one C-use

- All-C-Uses/Some-P-Uses:

For every variable v V, T contains def-clear paths from every defining node of v to
every C-use of v; and if a definition of v has no C-uses, a def-clear path leads to at
least one P-use

- All-DU-Paths:

For every variable v V, T contains def-clear paths from every defining node of v to
every use of v and to successor node of each USE(v,n) and that these paths are
either single loop traversals or they are cycle.

All-Paths

All-DU-Paths

All-Uses

All-C-Uses/Some-P-Uses All-P-Uses/Some-C-Uses

All-Defs All-P-Uses

All-Edges

All-Nodes

Subsumption Graph

54
Frankl and Weyuker's seven data flow criteria

- All-defs:
requires that for each definition of a variable X in P, the set of paths P executed by
the test set T contains a definition-clear subpath from the definition to at least one c-
use or one p-use of X.

- All-c-uses:
requires that for each definition of X in P, and each c-use of X reachable from the
definition, Pi contains a definition-clear subpath from the definition to the c-use.

- All-p-uses:
requires that for each definition of X in P, and each p-use of X reachable from the
definition, Pi contains a definition-clear subpath from the definition to the p-uses.

- All-c-uses/some-p-uses:
requires that for each definition of X in P, if there exists at least one c-use of X
reachable from the definition, Pi contains a definition-clear subpath from the
definition to at all reachable c-uses of X, otherwise, Pi contains a definition-clear
subpath from the definition to a reachable p-use of X.

- All-p-uses/some-c-uses:
requires that for each definition of X in P, if there exists at least one p-use of X
reachable from the definition, Pi contains a definition-clear subpath from the
definition to all reachable p-uses of X, otherwise, Pi contains a definition-clear
subpath from the definition to a reachable c-use of X.

- All-uses:
requires that for each definition of X in P, Pi contains a definition-clear subpath from
the definition to all reachable c-uses and p-uses of X.

- All-du-paths:
requires that for each definition of X in P, Pi contains all definition-clear subpaths
from the definition to all reachable c-uses and p-uses of X, such that each subpath
contains no loops, or contains one complete loop.

In addition: All-paths require that all paths through the program be executed.

55
 Nov 3, 05
2.2 Structural Testing: Data Flow-Based Testing
2.2.2 Slice-Based Testing

Introduction:
- A program slice is a subset of a pgm
- Enables programmers to view subsets of a pgm by filtering out code that is not
relevant.
- Why is it useful?
o More manageable.
o When testing, debugging, or understanding, most code is irrelevant.
o Pgm slicing provides a convenient way to filtering out irrelevant code
o Slices can be computed by static analysis.
- Definitions:
o P is a pgm
o V is a subset of variables at line #n
o A slice S(v,n) produces a portion of a pgm that contributes to the value of v
just before statement at n.
o S(v,n) must be derived from P (by deleting statements)
o S(v,n) must be syntactically correct.
o For all execution of P, the value of v in the execution of S(v,n) just before
location n must be the same value of in execution of P just before n.

Program Slicing Process:

1. Select the slicing criteria:
a. a variable (or set of variables)
b. a pgm location
2. Generate slices (may use such as tools: spyder, unravel, )
3. Perform testing on slices
4. Merge modified with the rest of the modified slices back into the original pgm

56
 Example on Slice-Based Testing

The programming code:

Slice S (mx, 22):
0 main() {
0 main() { 1 int mx, mn, av;
1 int mx, mn, av; 2 int tmp, num;
2 int tmp, sum, num; 4 tmp = readInt( ):
3 5 mx = tmp;
10 while(tmp >= 0)
4 tmp = readInt() : 11 {
5 mx = tmp; 12 if (mx < tmp)
6 mn = tmp; 13 mx = tmp;
7 sum = tmp; 18 tmp = readInt( );
8 num = 1; 19 }
9 22 printf("\nMax=%d", mx);
10 while(tmp >= 0) 27 }
11 {
12 if (mx < tmp) Slice S (sum, 25):
13 mx = tmp; 0 main() {
14 if (mn > tmp) 2 int tmp, num;
15 mn = tmp; 4 tmp = readInt( ):
16 sum += tmp; 7 sum = tmp;
17 ++num; 10 while(tmp >= 0)
18 tmp = readInt() ; 11 {
19 } 16 sum += tmp;
18 tmp = readInt( );
20 19 }
21 av = sum / num; 25 printf("\nSum=%d", sum);
22 printf("\nMax=%d", mx) ; 27 }
23 printf("\nMin=%d", mn);
24 printf("\nAvg=%d", av);
25 printf("\nSum=%d", sum); Slice S (mn, 23):
26 printf("\nNum=%d", num) ; 0 main() {
1 int mx, mn, av;
27 }
2 int tmp, num;
4 tmp = readInt( ):
6 mn = tmp;
10 while(tmp >= 0)
11 {
14 if (mn > tmp)
15 mn = tmp;
18 tmp = readInt( );
19 }
23 printf("\nMin=%d", mn);
27 }

Slice S (num, 26):

0 main() {
2 int tmp, num;
4 tmp = readInt( ):
8 num = 1;
10 while(tmp >= 0)
11 {
17 ++nun;
18 tmp = readInt( );
19 }
26 printf (" \nNum=%d", num);
27 }

57
 Mutation Testing

Mutation Testing:
- A testing techniques that focuses on measuring the adequacy of test cases
- Not a testing strategy
- Should be Slice S (av, 24): used in conjunction with
0 main() {
traditional techniques
1 int mx, mn, av;
- Goal: 2 int tmp, num; cause the mutant programs to
fail (thus 4 tmp = readInt( ): showing the effectiveness of
test 7 sum = tmp; cases)
- Faults are 8 num = 1; introduced into the pgm by:
1. 10 while(tmp >= 0) Creating many versions of
11 { pgm called mutants
16 sum += tmp;
2. Each mutant contains a single
17 ++num;
18 tmp = readInt( ); fault.
3. 19 } Test cases are applied to the
21 av = sum / num; original pgm & mutants
4. If 24 printf("\nAvg=%d", av); the original pgm of all
mutants generate the same
output, the test case is inadequate.
- A test case is shown to be adequate by finding at least one mutant that generates a
different output.
- Theoretical analysis as well experiments result show that it is an effective approach
to measure the adequacy of test cases

Categories of Mutation Operators:

Examples
Category
Mutation Original PGM
(x>5) if( x > y) Operand Replacement
(6>y)
(y>x)
b+(c-d) a=b*(c-d) Operator Replacement
b/(c-d)
(x>=y) if (x==y) Expression Modification
(x==++y) Such as: Replace an operator or insert
a new one
if(x==5) if (x==y) Statement Modification
a=5; a=5; Such as: Delete else part / delete if-
else a=1; then

Mutation Testing Process:

1. Generate test cases
2. Run each test case against the original pgm
a. If output is incorrect, the pgm must be modified & re-tested
b. If output is correct go to next step
3. Construct mutants (using a tool like Mothra)
4. Execute each test case against a alive mutant
a. If output of mutant differs from the original pgm, kill the mutant.

Mutation Score:
- A mutation score of a set of test cases is the percentage of non-equivalent mutants
killed.
D
Mutation Score * 100
NE
- (we need it to be 100%)
where D is dead mutants
N is # of mutants
E is # of equivalent mutants

58
 2.3 Structural Testing: Retrospective on Structure Testing

Observation:
- DF: for pgms that are computationally intensive
- Path/cyclomatic metric are more criteria metrics, not procedures:
o e.g., Basis Path gives a lower boundary on how much testing is necessary.
o Slice composition should be used to redevelop difficult sections of code
o Statement & Branch Coverage must be an absolute minimum (mandatory?)
testing reg.

When to stop testing?

1. When you run out of time.
2. When continued testing causes no new failures.
3. When continued testing reveals no new faults.
4. When you cannot think of any new test cases.
5. When you reach a point of diminishing returns.
6. When mandated coverage has been attained.
7. When all faults have been removed.

Trend of test coverage items:

Trend of test method effort:

59
 Types of Testing DON'T Like to See

During a particularly long and painful bout of REGRESSION TESTING our application
software, my buddies and I came up with this list of other types of testing we'd like not to
see:

AGGRESSION TESTING: If this doesn't work, I'm gonna kill somebody.

CONFESSION TESTING: Okay, Okay, I did program that bug.

CONGRSSIONAL TESTING: Are you now, or have you ever been a bug?

DEPRESSION TESTING: If this doesn't work, I'm gonna kill myself.

EGRESSION TESTING: Uh-oh, a bug... I'm outta here.

DIGRESSION TESTING: Well, it works, but can I tell you about my truck...

EXPRESSION TESTING: #@%^&*!!!, a bug.

OBSESSION TESTING: I'll find this bug if it's the last thing I do.

OPRESSION TESTING: Test this now!

POISSION TESTING: Alors! Regardez le poission!

REPRESSION TESTING: It's not a bug, it's a feature.

SECCESSION TESTING: The bug is dead! Long lives the bug!

SUGGESTION TESTING: Well, it works but wouldn't it be better if...

COMPRESSION TESTING: []

60
 3. Integration Testing

Introduction:
- Testing activities that integrate software components together.
- Definition: is a systematic technique for constructing the software architecture while
at the same time conducting tests to uncover errors associated with interfacing.
- Objective: to take unit tested components and build a program structure that has
been dictated by design.
- Several approaches:
1. Decomposition-Based Integration: using the functional decomposition tree.
a. Top-down Integration
b. Bottom-up Integration
c. Sandwich Integration: a combination of top-down and bottom-up
integration.
2. Call Graph-Based Integration: using a direct graph
a. Pair-Wise Integration
b. Neighborhood Integration
3. Path-Based Integration (MM-Path Based Integration)

Driver vs. Stub:

- Driver: is the main program that accepts test case data, passes such data to the
component (to be tested), and prints relevant results.
- Stubs: serve to replace modules that are subordinate to (called by) the component to
be tested.
- Drivers and stubs represent overhead because both are software that must be
written without delivering with the final software product.

61
 Figure: Driver vs. Stub (pressman, 2005)3.1 Integration Testing: Decomposition-
Based Integration

1. Top-down Integration:
- What are the steps for top-down integration?
1. The main control module is used as a test driver and stubs are
substituted for all components directly sub-ordinate to the
main control module.
2. Depending on the integration approach selected ( i.e., depth or
breadth first), sub-ordinate stubs are replaced one at a time
with actual components.
3. Tests are conducted as each component is integrated.
4. On completion of each set of tests, another stub is replaced
with the real components.
5. Regression testing may be conducted to ensure that new errors
have not been introduced.
The process continues from step 2 until the entire program
structure is built.

- What problems may be encountered when top-down integration is chosen?

1. Delay many tests until stubs are replaced with actual modules.
2. Develop stubs that perform limited functions that simulate the actual module, or
3. Integrate the software from the bottom of the hierarchy upward.

- Number of stubs required:

Stubs = Number of Nodes 1 (the driver)

2. Bottom-up Integration:
- What are the steps for bottom-up integration?
1. Low-level components are combined into clusters (sometimes
called builds) that perform a specific software sub-function.
2. A driver (a control program for testing) is written to
coordinate test case input and output.
3. The cluster is tested.
4. Drivers are removed and clusters are combined moving
upward in the program structure.

- Number of drivers required:

Drivers = Nodes - Leaves

62
 3.2 Integration Testing: Call Graph-Based Integration

Introduction
- Move away from a purely structural basis toward a behavioral basis.
- Eliminate the stub/driver development effort.

1. Pair-Wise Integration:
- Use actual code instead of stubs/drivers.
- Cover each edge.
- Restrict a test session to two pair of units (not big bang).

2. Neighborhood Integration:
- A neighbor is any node connected.
o A set of nodes that are one edge away from the given
node
o All intermediate predecessors (drives) and all immediate
successors (stubs).
- Pro: reduce the # of tests
- Con: problem isolating especially in large neighborhoods.

- Computing the number of neighborhoods:

Neighborhoods = interior nodes + source nodes

where Interior nodes = nodes (source nodes + sink nodes)

Neighborhoods = nodes - sink nodes

63
 Nov 10,
3.3 Integration Testing: Path-Based Integration 05

Introduction:
- Combined unit testing and integration testing.
- When a unit executes, some path of source statements is traversed.
- Suppose that a call goes to another unit along such a path:
o Unit testing: suppress the call statement because Path-Based Integration Pros:
control eventually returns to the calling unit anyway.
o Integration testing: abandon the single-entry, single- 1. Hybrid of functional and
exit percept and treat such calls as an exit followed by structural testing.
an entry. 2. Coupled with actual system
- We need to refine some of the program graph concepts. behavior.
3. Cost by elimination of stub
New Definition: and driver development.
- Source node:
o A statement fragment at which some program execution begins or resumes.
o Each module will have at least one.
o The first executable statement in a unit is a source node
- Sink node:
o A statement fragment at which program execution ends.
o E.g. final statement in a program.
- Module Execution Path (MEP): Hints in MM-Path:
o A sequence of statements that begins with a source node and
ends with a sink node with no intervening sink node. Nodes are Modules
o Program graph may have multiple source and sink nodes Edges are messages
- Message:
o A programming language mechanism by which one unit transfers control to
another unit.
- MM-Path:
o Module to module path (recall DD-Path: decision to decision path)
o Interleaved sequence of Module Execution Paths (MEP) and messages An MM-Path
o An example of Module Execution Paths (MEP) and MM-Path:
MEP(A,1) = <1, 2, 3, 6>
MEP(A,2) = <1, 2, 4>
MEP(A,3) = <5, 6>
MEP(B,1) = <1, 2>
MEP(B,2) = <3, 4>
MEP(C,1) = <1, 2, 4, 5>
MEP(C,2) = <1, 3, 4, 5>
- MM-Path Graph:
o Given a sequence of units, MM-Path graph is a directed
graph in which nodes are modules, execution paths and
edges correspond to messages and returns from one unit
to another.
o Assumption, in integration testing, unit testing has
already been done.
o The front figure shows an example of MM-Path Graph.

64
 Example on Integration Testing

The programming code:

1 Main integrationNextDate
Type Date
Month As Integer
Day As Integer
Year As Integer
EndType
Dim today As Date
Dim tomorrow As Date
2 GetDate(today) 'msg1
3 PrintDate(today) 'msg2
4 tomorrow = IncrementDate(today) 'msg3
5 PrintDate(tomorrow) 'msg4
6 End Main

7 Function isLeap(year) Boolean

8 If bear divisible by 4)
9 Then
10 If (year is NOT divisible by 100)
11 Then isLeap = True
12 Else
13 If bear is divisible by 400)
14 Then isLeap = True
15 Else isLeap = False
16 EndIf
17 EndIf
18 Else isLeap = False
19 EndIf
20 End (Function isleap)

21 Function lastDayOfMonth(month, year) Integer

22 Case month Of
23 Case 1: 1,3,5,7, 8, 10, 12
24 lastDayOfMonth = 3 1
25 Case 2: 4,6,9, 11
26 lastDayOfMonth = 30
27 Case 3: 2
28 If (isLeap(year)) 'msg5
29 Then IastDayOfMonth = 29
30 Else 1astDayOfMonth = 28
31 EndIf
32 EndCase
33 End (Function IastDayOfMonth)

65
 Example on Integration Testing (Cont.)

34 Function GetDate(aDate) Date

dim aDate As Date
35 Function ValidDate(aDate) Boolean 'within scope of GetDate
36 If ((aDate.Month > 0) AND (aDate.Month <=12)
37 Then monthOK = True
38 Else monthOK = False
39 EndIf
40 If (monthOK)
41 Then
42 If ((aDate.Day > 0) AND
(aDate.Day <= lastDayOfMonth(aDate.Month,
aDate.Year))'msg6
43 Then &yOK = True
44 Else dayOK = False
45 EndIf
46 EndIf
47 If ((aDate.Year > 1 8 11) AND (aDate.Year <= 20 12)
48 Then yearOK = True
49 Else yearOK = False
50 EndIf
51 If (monthOK AND dayOK AND yearOK)
52 Then ValidDate = True
53 Else ValidDate = False
54 EndIf
55 End (Function ValidDate)
' GetDate body begins here
56 Do
57 Output("enter a month")
58 Input(aDate.Month)
59 Output("enter a day")
60 Input(aDate.Day)
61 Output("enter a year")
62 Input(aDate.Year)
63 GetDateMonth = aDate.Month
64 GetDate-Day = aDate.Day
65 GetDate.Year = aDate.Year
66 Until (ValidDate(aDate)) 'msg7
67 End (Function GetDate)

68 Function IncrementDate(aDate) Date

69 If (aDate.Day < lastDayOfMonth(aDate.Month)) 'msg8
70 Then aDate.Day = aDate.Day + 1
71 Else aDate.Day = 1
72 If (aDate.Month = 12)
73 Then aDate.Month = 1
74 aDate.Year = aDate.Year + 1
75 Else aDate.Month = aDate.Month + 1
76 EndIf
77 EndIf
78 End (IncrementDate)

66
 Example on Integration Testing (Cont.)

1. Functional decomposition of integration version:

They will cause problem with pairwise

integration because they are never
directly called by Main program.

2. Call graph of integration version:

3. MM-Path Based Integration (for May 27, 2002):

Main (1, 2)
msg1
GetDate (34, 56, 57,58, 59,60,61, 62, 63, 64, 65, 66)
msg7
ValidDate (35, 36,37, 39,40, 41, 42))
msg6
lastDayOfMonth (21,22,23,24,32, 33)'point of message quiessence
ValidDate (43.45,46,47,48, 50, 51, 52, 54, 55)
GetDate (67)
'Main (3)

67
 Static Program Testing Analysis:
1. Hoare Logic

Introduction:
- Hoare logic (also known as FloydHoare logic) is a formal system developed by the
British computer scientist C. A. R. Hoare, and subsequently refined by Hoare and
other researchers. It was published in Hoare's 1969 paper "An axiomatic basis for
computer programming". The purpose of the system is to provide a set of logical
rules in order to reason about the correctness of computer programs with the rigour
of mathematical logic.
- A programming language specification consists of a syntactic description and a
semantic description.
o Syntactic description: symbols we can use in a language
o Semantic Description: what phrases in a programming language mean.
- In specific phrase, using natural language to describe three things:
1. Pre-conditions
2. Invariant
3. Post-conditions

Prototype:
{P} C {Q}
where P and Q are assertions and C is a command. P is called {P} C {Q} reads:
the precondition and Q the postcondition. Assertions are
formulas in predicate logic. Whenever P holds of the state
before the execution of C, then
Two Types: Q will hold afterwards.
- Partial Correctness: if C does not terminate, then there is no
"after", so Q can be any statement at all. Indeed, one can choose Q to be false to
express that C does not terminate.
- Total Correctness: if C terminates and at termination Q is true, the expression
exhibits. Termination would have to be proved separately.

Loop Invariant:
- A constant (unchanging) predicate (constraint, fact)
- Unaffected by group of mathematical operations (program statements) under
consideration
o often the important identification of the program (states? the acceptance?
look a little deeply)
- Establishing a loop invariant:
o Define a predicate I that shows the logical relationship between i, s, and b

I: 1 i 11 ^ s = sum(b[k]) k=0, ..., i-1

\\ i is changing
o Show that I is true before the loop and after each iteration of the loop so that
it is true after the completion of the loop
o If I is true at these place, together with falsity for loop guard, show that the
post condition holds

68
 Example on Hoare Logic

Objective:
A program that stores in the sum of array b[0,,10] in S

The predicate I:
I: 1 i 11 ^ s = sum(b[k]) k=0, ..., i-1

The programming code with comments:

Precondition: True
i:=1;
(1) I is true s:= b[0];

while (i<11)
s:=s+b[i];
(2) is true i:=i+1;

(3)I is true end while

postcondition: s = sum(b[k])
k=0,, 10

Prove that I is True all the times:

1. Before the loop:
i=1, s=b[0]

Does this affect I?

I : 1 i 11 ^ s = sum(b[k]) k=0, ..., i-1

1 1 11 ^ b[0]= sum(b[k]) k=0,i-1

True ^ True

True

2. Inside the loop

s = s + b[i];
i = i + 1;

Do these affect I?
I : 1 i 11 ^ s = sum(b[k]) k=0, ..., i-1

1 1 < 11 ^ s+b[i]= sum(b[k]) k=0,i-1

0 1 < 11 ^ s+b[i]= sum(b[k]) k=0,i

0 1 < 11 ^ s+b[i]= sum(b[k]) k=0,i-1+b[i]

0 1 < 11 ^ s+b[i]= sum(b[k]) k=0,i-1+b[i]

0 1 < 11 ^ s = sum(b[k]) k=0,i-1

3. Post condition: (when i=11)

I : 1 i 11 ^ s = sum(b[k]) k=0, ..., i-1

1 11 11 ^ s = sum(b[k]) k=0, ..., 10

69

True ^ True

70
 Static Program Testing Analysis:
2. Design by Contract (DBC)

Introduction:
- Software entities have obligations to other entities based upon formalized rules
between them.
- A functional specification, or 'contract', is created for each module in the system
before it is coded. Program execution is then viewed as the interaction between the
various modules as bound by these contracts.
- routines have explicit preconditions that the caller must satisfy before calling the
routine, and explicit postconditions that describe the conditions that the routine will
guarantee to be true after the routine finishes
- a contract takes the following general form: "If you, the caller, set up certain
preconditions, then I will establish certain other results when I return to you. If you
violate the preconditions, then I promise nothing."
- DBC is novel in recognizing that these contracts are so crucial to software correctness
that they should be part of the design process. In effect, DBC advocates writing the
assertions first.
- The notion of a contract extends down to the method/procedure level; the contract
for each method will normally contain the following pieces of information:
o Acceptable and unacceptable inputs
o Return values, and their meanings
o Error and exception conditions that can occur
o Side-effects
o Preconditions
o Postconditions
o invariants
o (Rarer) Performance guarantees, e.g., for time or space used

DBC Syntax:
- When you write a module, as part of this program, you have to follow the syntax:
Requirement
---
---
---
Programs
---
---
Ensure
---
---

- If a requirement is met, then the ensure is generated to be generated.

71
 Glossary

Taken from ADRION, W. R., BRANSTAD, M. A. & CHERNIAVSKY, J. C. (1982) Validation, Verification, and Testing of
Computer Software. ACM Comput. Surv., 14, 159-192.

Audit.
See DOD Development Reviews.

Black Box Testing.

See Functional Testing.

Boundary Value Analyses.

A selection technique in which test data are chosen to lie along "boundaries" of input
domain (or output range) classes, data structures, procedure parameters, etc. Choices often
include maximum, minimum, and trivial values or parameters. This technique is often called
stress testing. (See Section 2.4.)

Branch Testing.
A test method satisfying coverage criteria that require that for each decision point each
possible branch be executed at least once. (See Section 2.5.)

Cause-Effect Graphing.
Test data selection technique. The input and output domains are partitioned into classes and
analysis is performed to determine which input classes cause which effect. A minimal set of
inputs is chosen that will cover the entire effect set. (See Section 2.4.)

Certification.
Acceptance of software by an authorized agent usually after the software has been validated
by the agent, or after its validity has been demonstrated to the agent.

Critical Design Review.

See DOD Development Reviews.

Complete Test Set.

A test set containing data that causes each element of a prespecified set of Boolean
conditions to be true. Additionally, each element of the test set causes at least one condition
to be true. (See Section 2.2.)

Consistent Condition Set.

A set of Boollean conditions such that complete test sets for the conditions uncover the
same errors. (See Section 2.2.)

Cyclomatic Complexity.
The cyclomatic complexity of a program is equivalent to the number of decision statements
plus
1. (See Section 2.5.)

DD (decision-to-decision) Path.
A path of logical code sequence that begins at an entry or decision statement and ends at a
decision statement or exit. (See Section 2.5.)

Debugging. The process of correcting syntactic and logical errors detected during coding.
With the primary goal of obtaining an executing piece of code, debugging shares with
testing certain techniques and strategies, but differs in its usual ad hoc application and local
scope.

Design-Based Functional Testing.

The application of test data derived through functional analysis (see Functional Testing)
extended to include design functions as well as requirement functions. (See Section 2.4.)

72
DOD Development Reviews.
A series of reviews required by DOD directives. These include
(1) The Systems Requirements Review is an examination of the initial progress during the
problem definition stage and of the convergence on a complete system configuration.
Test planning and test documentation are begun at this review.
(2) The System Design Review occurs when the system definition has reached a point where
major system modules can be identified and completely specified along with the
corresponding test requirements. The requirements for each major subsystem are
examined along with the preliminary test plans. Tools required for verification support are
identified and specified at this stage.
(3) The Preliminary Design Review is a formal technical review of the basic design approach
for each major subsystem or module. The revised requirements and preliminary design
specifications for each major subsystem and all test plans, procedures, and
documentation are reviewed at this stage. Development and verification tools are further
identified at this stage. Changes in requirements will lead to an examination of the test
requirements to maintain consistency.
(4) The Critical Design Review occurs just prior to the beginning of the construction stage.
The complete and detailed design specifications for each module and all draft test plans
and documentation are examined. Again, consistency with previous stages is reviewed,
with particular attention given to determining if test plans and documentation reflect
changes in the design specifications at all levels.
(5) Two audits, the Functional Configuration Audit and the Physical Configuration Audit are
performed. The former determines if the subsystem performance meets the
requirements. The latter audit is an examination of the actual code. In both audits,
detailed attention is given to the documentation, manuals and other supporting material.
(6) A Formal Qualification Review is performed to determine through testing that the final
coded subsystem conforms with the final system specifications and requirements. It is
essentially the subsystem acceptance test.

Driver.
Code that sets up an environment and calls a module for test. (See Section 1.3.)

Dynamic Analysis.
Analysis that is performed by executing the program code. (See Section 2.7.)

Dynamic Assertion.
A dynamic analysis technique that inserts assertions about the relationship between
program variables into the program code. The truth of the assertions is determined as the
program executes. (See Section 2.7.)

Error Guessing. Test data selection technique. The selection criterion is to pick values that
seem likely to cause errors. (See Section 2.4.}

Exhaustive Testing. Executing the program with all possible combinations of values for
program variables. (See Section 2.1.)

Extrenal Test Data. Test data that is at the extreme or boundary of the domain of an input
variable or which produces results at the boundary of an output domain. (See Section 2.4.)

Formal Qualification Review.

See DOD Development Reviews.

Functional Configuration Audit.

See DOD Development Reviews.

Functional Testing.
Application of test data derived from the specified functional requirements without regard to
the final program structure. (See Section 2.4.)

Infeasible Path.

73
A sequence of program statements that can never be executed. (See Section 2.5.)

Inspection.
A manual analysis technique in which the program {requirements, design, or code) is
examined in a very formal and disciplined manner to discover errors. (See Section 2.2.)

Instrumentation.
The insertion of additional code into the program in order to collect information about
program behavior during program execution. (See Section 2.7.)

Invalid Input (Test Data for Invalid Input Domain).

Test data that lie outside the domain of the function the program represents. (See Section
2.1.)

Life-Cycle Testing.
The process of verifying the consistency, completeness, and correctness of the software
entity at each stage in the development. (See Section 1.)

Metric-Based Test Data Generation.

The process of generating test sets for structural testing based upon use of complexity
metrics or coverage metrics. (See Section 2.5.)

Mutation Analysis.
A method to determine test set thoroughness by measuring the extent to which a test set
can discriminate the program from slight variants (mutants) of the program. (See Section
2.6.)

Oracle. A mechanism to produce the "correct" responses to compare with the actual
responses of the software under test. (See Section 2.1.)

Path Expressions.
A sequence of edges from the program graph which represents a path through a program.
(See Section 2.5.)

Path Testing.
A test method satisfying coverage criteria that each logical path through the program be
tested. Often paths through the program are grouped into a Testing of Computer Software
187 finite set of classes; one path from each class is then tested. (See Section 2.5.)

Preliminary Design Review.

See DOD Development Reviews.

Program Graph.
Graphical representation of a program. (See Section 2.5.)

Proof of Correctness.
The use of techniques of mathematical logic to infer that a relation between program
variables assumed true at program entry implies that another relation between program
variables holds at program exit. (See Section 2.2.)

Regression Testing.
Testing of a previously verified program required following program modification for
extension or correction. (See Section 1.4.)

Simulation.
Use of an executable model to represent the behavior of an object. During testing the
computational hardware, the external environment, and even code segments may be
simulated. (See Section 2.2.)

Self-Validating Code.

74
Code which makes an explicit attempt to determine its own correctness and to proceed
accordingly. (See Section 2.7.)

Special Test Data.

Test data based on input values that are likely to require special handling by the program.
(See Section 2.4.)

Statement Testing.
A test method satisfying the coverage criterion that each statement in a program be
executed at least once during program testing. (See Section 2.5.)

Static Analysis,
Analysis of an program that is performed without executing the program. (See Section 2.7.)

Stress Testing.
See Boundary Value Analysis.

Structural Testing.
A testing method where the test data are derived solely from the program structure. (See
Section 2.5.)

Stub.
Special code segments that, when invoked by a code segment under test, will simulate the
behavior of designed and specified modules not yet constructed. (See Section 1.3.)

Symbolic Execution.
A static analysis technique that derives a symbolic expression for each program path. (See
Section 2.7.)

System Requirements Review.

See DOD Development Reviews.

Test Data Set.

Set of input elements used in the testing process. (See Section 2.1.)

Test Driver.
A program that directs the execution of another program against a collection of test data
sets. Usually the test driver also records and organizes the output generated as the tests are
run. (See Section 1.3.)

Test Harness.
See Test Driver.

Testing.
Examination of the behavior of a program by executing the program on sample data sets.

Valid Input (test data for a valid input domain).

Test data that lie within the domain of the function represented by the program. (See Section
2.1.)

Validation.
Determination of the correctness of the final program or software produced from a
development project with respect to the user needs and requirements. Validation is usually
accomplished by verifying each stage of the software development life cycle.

Verification.
In general, the demonstration of consistency, completeness, and correctness
of the software at each stage and between each stage of the development life cycle.

Walk-Through.

75
A manual analysis technique in which the module author describes the module's structure
and logic to an audience of colleagues. (See Section 2.2.)

White Box Testing.

See Structural Testing.

76