Learning Objectives

Testing
What is testing?
Why test?
How testing increase our confidence in program correctness?
When to test?
Functional testing

What to test?
How to test?
Different types of testing
How to assess test adequacy?

August 26, 2010 1

 Testing
When should we test?
Separate phase for testing? OR
At the end of every phase? OR
During each phase, simultaneously with
all development and maintenance
Functional testing

activities

August 26, 2010 2

 Terminology
Program - Sort
Specification
Input: p - array of n integers, n > 0
Output: q - array of n integers such that
q[0] ≤ q[1] ≤ … ≤ q[n]
Functional testing

Elements in q are a permutation of elements in p, which are

unchanged
Description of requirements of a program

August 26, 2010 3

 Tests
Test input (Test case)
A set of values given as input to a program
Includes environment variables
{2, 3, 6, 5, 4}
Test set
Functional testing

A set of test cases

{ {0}, {9, 8, 7, 6, 5}, {1, 3, 4, 5}, {2, 1, 2, 3} }

August 26, 2010 4

 Oracle
Function that determines whether or not the results of executing a
program under test is as per the program’s specifications
Functional testing

Software output Problems

Under Test
Test Failure? • Correctness of Oracle
Compare Success? • Correctness of specs
Cases • Generation of Oracle
Oracle output • Need more formal
specs

August 26, 2010 5

 Correctness
Program Correctness
A program P is considered with respect to a specification S, if and
only if:
For each valid input, the output of P is in accordance with the
specification S
Functional testing

What if the specifications are themselves incorrect?

August 26, 2010 6

 Errors, defects, faults
Often used interchangeably.
Error:
Mistake made by programmer.
Human action that results in the software containing a fault/defect.
Defect / Fault:
Functional testing

Manifestation of the error in a program.

Condition that causes the system to fail.
Synonymous with bug

August 26, 2010 7

 Failure
Incorrect program behavior due to a fault in the program.
Failure can be determined only with respect to a set of requirement
specs.
For failure to occur, the testing of the program should force the
erroneous portion of the program to be executed.
Functional testing

August 26, 2010 8

 Errors and failure

Error revealing
Inputs
inputs cause failures
Functional testing

Program

Erroneous outputs
Outputs indicate failures

August 26, 2010 9

 Nonexecution-Based Testing
Person creating a product should not be the only one responsible for
reviewing it.
A document is is checked by a team of software professionals with a
range of skills.
Increases chances of finding a fault.
Functional testing

Types of reviews
Walkthroughs
Inspections

August 26, 2010 10

 Walkthroughs

TEAM

Rep from Specs team Manager from Specs team

Functional testing

Client representative Rep of team from next phase

Rep from SQA group

Reviewer prepares two lists:
Items that the reviewer does not understand
Items that the reviewer believes are incorrect

August 26, 2010 11

 Managing walkthroughs
Distribute material for walkthrough in advance.
Includes senior technical staff.
Chaired by the SQA representative.
Task is to record fault for later correction
Not much time to fix it during walkthrough
Functional testing

Other individuals are trained to fix it better

Cost of 1 team vs cost of 1 person
Not all faults need to be fixed as not all “faults” flagged are
incorrect

August 26, 2010 12

 Managing walkthroughs (contd.)
Two ways of doing walkthroughs
Participant driven
Present lists of unclear items and incorrect items
Rep from specs team responds to each query
Document driven
Functional testing

Person responsible for document walks the participants through the

document
Reviewers interrupt with prepared comments or comments triggered
by the presentation
Interactive process
Not to be used for the evaluation of participants

August 26, 2010 13

 Inspections
Proposed by Fagan for testing
designs
code
An inspection goes beyond a walkthrough
Five formal stages
Functional testing

Stage 1 - Overview
Overview document (specs/design/code/ plan) to be prepared by
person responsible for producing the product.
Document is distributed to participants.

August 26, 2010 14

 Inspections (contd.)
Stage 2 - Preparation
Understand the document in detail.
List of fault types found in inspections ranked by frequency used for
concentrating efforts.
Stage 3 - Inspection
Functional testing

Walk through the document and ensure that

Each item is covered
Every branch is taken at least once
Find faults and document them (don’t correct)
Leader (moderator) produces a written report

August 26, 2010 15

 Inspections (contd.)
Stage 4 - Rework
Resolve all faults and problems
Stage 5 - Follow-up
Moderator must ensure that every issue has been resolved in some
way
Functional testing

Team
Moderator-manager, leader of inspection team
Designer - team responsible for current phase
Implementer - team responsible for next phase
Tester - preferably from SQA team

August 26, 2010 16

 What to look for in inspections
Is each item in a specs doc adequately and correctly addressed?
Do actual and formal parameters match?
Error handling mechanisms identified?
Design compatible with hardware resources?
What about with software resources?
Functional testing

August 26, 2010 17

 What to record in inspections
Record fault statistics
Categorize by severity, fault type
Compare # faults with average # faults in same stage of
development
Find disproportionate # in some modules, then begin checking other
Functional testing

modules
Too many faults => redesign the module
Information on fault types will help in code inspection in the same
module

August 26, 2010 18

 Pros and cons of inspection
High number of faults found even before testing (design and code
inspections)
Higher programmer productivity, less time on module testing
Fewer faults found in product that was inspected before
If faults are detected early in the process there is a huge savings
Functional testing

What if walkthroughs are used for performance appraisal?

August 26, 2010 19

 Execution-based testing
Execution-based testing is a process of inferring certain behavioral
properties of a product, based, in part, on the results of executing
the product in a known environment with selected inputs.
Depends on environment and inputs
How well do we know the environment?
Functional testing

How much control do we have over test inputs?

Real time systems

August 26, 2010 20

 Reliability
Probability of failure-free operation of a product for a given time
duration
How often does the product fail?
mean time between failures
How bad are the effects?
Functional testing

How long does it take to repair it?

mean time to repair
Failure behavior controlled by
Number of faults
Operational profile of execution

August 26, 2010 21

 Robustness
How well does the product behave with
range of operating conditions
possibility of unacceptable results with valid input?
possibility of unacceptable results with invalid input?
Should not crash even when not used under permissible conditions
Functional testing

August 26, 2010 22

 Performance
To what extent does the product meet its requirements with regard
to
response time
space requirements
What if there are too many clients/ processes, etc...
Functional testing

August 26, 2010 23

 Levels of testing
Unit testing
Integration testing
System testing
Acceptance testing
Functional testing

August 26, 2010 24

 Functional testing
Learning objectives

What is functional testing?

How to perform functional testing?
What are clues, test requirements, and test specifications?
Functional testing

How to generate test inputs?

What are equivalence partitioning, boundary value testing, domain
testing, state testing, and decision table testing?

August 26, 2010 25

 What is functional testing?
When test inputs are generated using program specifications, we say that
we are doing functional testing.
Functional testing tests how well a program meets the functionality
requirements.
Functional testing

August 26, 2010 26

 The methodology
The derivation of test inputs is based on program specifications.
Clues are obtained from the specifications.
Clues lead to test requirements.
Test requirements lead to test specifications.
Test specifications are then used to actually execute the program under
Functional testing

test.

August 26, 2010 27

 Specifications-continued

Two types of pre-conditions are considered:

Validated: those that are required to be validated by the
program under test and an error action is required to be
Functional testing

performed if the condition is not true.

Assumed: those that are assumed to be true and not checked
by the program under test.

August 26, 2010 28

 Preconditions for sort
Validated:
N>0
On failure return -1; sorting considered unsuccessful.
Functional testing

Assumed:
The input sequence contains N integers.
The output area has space for at least N integers.

August 26, 2010 29

 Post-conditions
A post-condition specifies a property of the output of a program.
The general format of a post-condition is:
if condition then effect-1 {else effect-2}
Example:
For the sort program a post-condition is:
Functional testing

if N>0 then {the output sequence has the same elements as in

the input sequence and in ascending order.}

August 26, 2010 30

 Incompleteness of specifications

Specifications may be incomplete or ambiguous.

Example post-condition:
if user places cursor on the name field then read a string
Functional testing

This post-condition does not specify any limit on the length of the
input string hence is incomplete.

August 26, 2010 31

 Ambiguous specifications
It also does not make it clear as to
whether a string should be input only after the user has placed the
cursor on the name field and clicked the mouse or simply placed
the cursor on the name field.
and hence is ambiguous.
Functional testing

August 26, 2010 32

 Clues: summary
Clues are:
Pre-conditions
Post-conditions
Variables, e.g. A is a length implying thereby that its value cannot
be negative.
Functional testing

Operations, e.g. “search a list of names” or “find the average of

total scores”
Definitions, e.g. “filename(name) is a name with no spaces.”

August 26, 2010 33

 Clues-continued
Ideally variables, operations and definitions should be a part of at
least one pre- or post-condition.
However, this may not be the case as specifications are not always
written formally.
Hence look out for variables, operations, and definitions within a
Functional testing

specification!

August 26, 2010 34

 Test requirements checklist
Obtaining clues and deriving test requirements can become a tedious task.
To keep it from overwhelming us it is a good idea to make a checklist of clues.
This checklist is then transformed into a checklist of test requirements by
going through each clue and deriving test requirements from it.
Functional testing

August 26, 2010 35

 Equivalence partitioning
The input domain is usually too large for exhaustive testing.
It is therefore partitioned into a finite number of sub-domains for the
selection of test inputs.
Each sub-domain is known as an equivalence class and serves as a source of at
least one test input.
Functional testing

August 26, 2010 36

 Equivalence partitioning

Input domain Input domain

partitioned into four
sub-domains.
Functional testing

2
1
3

Too many Four test inputs, one

test inputs. selected from each sub-domain.

August 26, 2010 37

 How to partition?
Inputs to a program provide clues to partitioning.
Example 1:
Suppose that program P takes an input X, X being an integer.
For X<0 the program is required to perform task T1 and for X>=0
Functional testing

task T2.

August 26, 2010 38

 How to partition?-continued
The input domain is prohibitively large because X can assume a large
number of values.
However, we expect P to behave the same way for all X<0.
Similarly, we expect P to perform the same way for all values of
X>=0.
Functional testing

We therefore partition the input domain of P into two sub-domains.

August 26, 2010 39

 Two sub-domains

One test case:

X=-3 Equivalence class
Functional testing

X<0 X>=0
Equivalence class Another test case:
X=-15

All test inputs in the X<0 sub-domain are considered equivalent.

The assumption is that if one test input in this sub-domain reveals
an error in the program, so will the others.

This is true of the test inputs in the X>=0 sub-domain also.

August 26, 2010 40

 Non-overlapping partitions
In the previous example, the two equivalence classes are non-overlapping. In
other words the two sub-domains are disjoint.
When the sub-domains are disjoint, it is sufficient to pick one test input
from each equivalence class to test the program.
Functional testing

August 26, 2010 41

 Non-overlapping partitions
An equivalence class is considered covered when at least one test has been
selected from it.
In partition testing our goal is to cover all equivalence classes.
Functional testing

August 26, 2010 42

 Overlapping partitions
Example 2:
Suppose that program P takes three integers X, Y and Z. It is known
that:
X<Y
Z>Y
Functional testing

August 26, 2010 43

 Overlapping partitions

X<Y, Z<=Y
X=2, Y=3, Z=1
X>=Y, Z<=Y
X<Y X=15, Y=4, Z=1
Functional testing

X<Y, Z>Y X>=Y

X=3, Y=4, Z=7
Z>Y Z<=Y

X>=Y, Z>Y
X=15, Y=4, Z=7

August 26, 2010 44

 Overlapping partition-test selection

In this example, we could select 4 test cases as:

X=4, Y=7, Z=1 satisfies X<Y
X=4, Y=2, Z=1 satisfies X>=Y
X=1, Y=7, Z=9 satisfies Z>Y
X=1, Y=7, Z=2 satisfies Z<=Y
Functional testing

Thus, we have one test case from each equivalence class.

August 26, 2010 45

 Overlapping partition-test selection

However, we may also select only 2 test inputs and satisfy all four
equivalence classes:
X=4, Y=7, Z=1 satisfies X<Y and Z<=Y
X=4, Y=2, Z=3 satisfies X>=Y and Z>Y
Thus, we have reduced the number of test cases from 4 to 2 while
Functional testing

covering each equivalence class.

August 26, 2010 46

 Partitioning using non-numeric data

In the previous two examples the inputs were integers. One can
derive equivalence classes for other types of data also.
Example 3:
Suppose that program P takes one character X and one string Y as
inputs. P performs task T1 for all lower case characters and T2 for
Functional testing

upper case characters. Also, it performs task T3 for the null string
and T4 for all other strings.

August 26, 2010 47

 Partitioning using non-numeric data

X: lc, Y: not null

X: UC, Y: not null

X: lc
Functional testing

X:UC
X: lc, Y: null
Y: null Y: not null

lc: Lower case character

X: UC, Y: null
UC: Upper case character
null: null string.

August 26, 2010 48

 Non-numeric data
Once again we have overlapping partitions.
We can select only 2 test inputs to cover all four equivalence classes. These
are:
X: lower case, Y: null string
X: upper case, Y: not a null string
Functional testing

August 26, 2010 49

 Guidelines for equivalence partitioning

Input condition specifies a range: create one for the valid case and two for
the invalid cases.
e.g. for a<=X<=b the classes are
a<=X<=b (valid case)
X<a and X>b (the invalid cases)
Functional testing

August 26, 2010 50

 Guidelines-continued

Input condition specifies a value: create one for the valid value and
two for incorrect values (below and above the valid value). This may
not be possible for certain data types, e.g. for boolean.
Input condition specifies a member of a set: create one for the valid
value and one for the invalid (not in the set) value.
Functional testing

August 26, 2010 51

 Sufficiency of partitions
In the previous examples we derived equivalence classes based on the
conditions satisfied by the input data.
Then we selected just enough tests to cover each partition.
Think of the advantages and disadvantages of this approach!
Functional testing

August 26, 2010 52

 Boundary value analysis (BVA)
Another way to generate test cases is to look for boundary values.
Suppose a program takes an integer X as input.
In the absence of any information, we assume that X=0 is a boundary.
Inputs to the program might lie on the boundary or on either side of
the boundary.
Functional testing

August 26, 2010 53

 BVA: continued
This leads to 3 test inputs:
X=0, X=-20, and X=14.
Note that the values -20 and 14 are on either side of the boundary and are
chosen arbitrarily.
Notice that using BVA we get 3 equivalence classes. One of these three
Functional testing

classes contains only one value (X=0), the other two are large!

August 26, 2010 54

 BVA: continued
Now suppose that a program takes two integers X and Y and that x1<=X<=x2
and y1<=Y<=y2.
Functional testing

14
1 5 2
y2
10
11
8 9 6
13
y1 12 3
4 7

x1 x2
August 26, 2010 55
 BVA-continued
In this case the four sides of the rectangle represent the boundary.
The heuristic for test selection in this case is:
Select one test at each corner (1, 2, 3, 4).
Select one test just outside of each of the four sides of the
Functional testing

boundary (5, 6, 7, 8)

August 26, 2010 56

 BVA-continued
Select one test just inside of each of the four sides of the
boundary (10, 11, 12, 13).
Select one test case inside of the bounded region (9).
Select one test case outside of the bounded region (14).
How many equivalence classes do we get?
Functional testing

August 26, 2010 57

 BVA -continued
In the previous examples we considered only numeric data.
BVA can be done on any type of data.
For example, suppose that a program takes a string S and an integer X as inputs.
The constraints on inputs are:
length(S)<=100 and a<=X<=b
Can you derive the test cases using BVA?
Functional testing

August 26, 2010 58

 BVA applied to output variables

Just as we applied BVA to input data, we can apply it to output data.

Doing so gives us equivalence classes for the output domain.
We then try to find test inputs that will cover each output equivalence class.
Functional testing

August 26, 2010 59

 Summary
Specifications, pre-conditions, and post-conditions.
Clues, test requirements, and test specifications.
Clues from code.
Test requirements catalog.
Equivalence partitioning and boundary value analysis.
Functional testing

August 26, 2010 60