From Pressman, Software Engineering

a practitioners approach, Chapter 14

and Pezze + Young, Software Testing
and Analysis, Chapters 10-11
Today, well talk about testing how to
test software. The question is: How do
we design tests? And well start with
functional testing.

Functional Testing
Software Engineering
Andreas Zeller Saarland University

Functional testing is also called blackbox testing, because we see the

program as a black box that is, we
ignore how it is being written

in contrast to structural or white-box

testing, where the program is the base.

If the program is not the base, then what

is? Simple: its the specification.

If the program is not the base, then what

is? Simple: its the specification.

Testing Tactics

Functional

Structural

black box

white box

Tests based on spec

Test covers as much
specified behavior
as possible

Tests based on code

Test covers as much
implemented behavior
as possible

Why Functional?
Functional

Structural

black box

white box

Program code not necessary

Early functional test design has benefits

reveals spec problems assesses testability gives additional

explanation of spec may even serve as spec, as in XP

Structural testing can not detect that

some required feature is missing in the
code
Functional testing applies at all
granularity levels (in contrast to
structural testing, which only applies to
unit and integration testing)

Why Functional?
Functional

Structural

black box

white box

Best for missing logic defects

Common problem: Some program logic was simply forgotten

Structural testing would not focus on code that is not there

Applies at all granularity levels

unit tests integration tests system tests regression tests

2,510,588,971 years, 32 days, and 20

hours to be precise.

A Challenge
class Roots {
// Solve ax2 + bx + c = 0
public roots(double a, double b, double c)
{ }
// Result: values for x
double root_one, root_two;
}

Which values for a, b, c should we test?

assuming a, b, c, were 32-bit integers, wed have (232)3 1028 legal inputs
with 1.000.000.000.000 tests/s, we would still require 2.5 billion years

Note that in 900 million years, due to

increase of the luminosity of the sun,
CO2 levels will be toxic for plants; in 1.9
billion years, surface water will have
evaporated (source: Wikipedia on
Earth)

Life Cycle of the Sun

Note that in 900 million years, due to

increase of the luminosity of the sun,
CO2 levels will be toxic for plants; in 1.9
billion years, surface water will have
evaporated (source: Wikipedia on
Earth)

Life Cycle of the Sun

10

None of this is crucial for the

computation, though.

11

12

A Challenge
class Roots {
// Solve ax2 + bx + c = 0
public roots(double a, double b, double c)
{ }
// Result: values for x
double root_one, root_two;
}

Which values for a, b, c should we test?

assuming a, b, c, were 32-bit integers, wed have (232)3 1028 legal inputs
with 1.000.000.000.000 tests/s, we would still require 2.5 billion years

13

One might think that picking random

samples might be a good idea.

Random Testing
Pick possible inputs uniformly
Avoids designer bias

A real problem: The test designer can make the same logical
mistakes and bad assumptions as the program designer
(especially if they are the same person)

But treats all inputs as equally valuable

14

However, it is not. For one, we dont

care for bias we specifically want to
search where it matters most. Second,
random testing is unlikely to uncover
specific defects. Therefore, we go for
functional testing.

Why not Random?

Defects are not distributed uniformly
Assume Roots applies quadratic equation
and fails if b2 4ac = 0 and a = 0

Random sampling is unlikely to choose

a = 0 and b = 0

15

The main steps of a systematic

approach to functional program testing
(from Pezze + Young, Software Testing
and Analysis, Chapter 10)

Systematic Functional Testing

Functional
specification

identify

Independently
testable feature
identify

derive

Representative
values

Model
derive

generate

Test case
specifications

Test case

16

Testable Features
Functional
specification

identify

Independently
testable feature

into
Decompose systemRepresentative
identify

derive

independently testable features (ITF)

Model

values

An ITF need not correspond to units or

subsystems of the software

derive

generate

Test case
case testing, ITFs are exposed
ForTestsystem
specifications

through user interfaces or APIs

17

Just one roots is a unit and thus

provides exactly one single testable
feature.

Testable Fatures
class Roots {
// Solve ax2 + bx + c = 0
public roots(double a, double b, double c)
{ }
// Result: values for x
double root_one, root_two;
}

Ask
audience

What are the independently testable features?

18

Every single function becomes an

independently testable feature. Some
functions (like memory access, for
instance) are dependent on each other,
though: to retrieve a value, you must
first store it.
(Note how the calculator shows the
#years required for the Roots
calculation.)

Testable Fatures
Consider a multi-function
calculator

What are the independently

testable features?

Ask
audience
19

The main steps of a systematic

approach to functional program testing
(from Pezze + Young, Software Testing
and Analysis, Chapter 10)

Testable Features
Functional
specification

identify

Independently
testable feature
identify

derive

Representative
values

Model
derive

generate

Test case

Test case
specifications

20

The main steps of a systematic

approach to functional program testing
(from Pezze + Young, Software Testing
and Analysis, Chapter 10)

Representative Values
Functional
Try to select
specification

identify

Independently

inputs
testable feature
that are especially
identify
derive
valuable

Usually by

Representative
values

Model

choosing
representatives of equivalence derive
classes that
generate
are apt to fail often or not Test
at allcase

Test case

specifications

21

Needles in a Haystack
To find needles,

look systematically

We need to find out

what makes needles special

22

We can think of all the possible input values to a program as

little boxes ... white boxes that the program processes
correctly, and colored boxes on which the program fails. Our
problem is that there are a lot of boxes ... a huge number,
and the colored boxes are just an infinitesimal fraction of the
whole set. If we reach in and pull out boxes at random, we
are unlikely to find the colored ones.
Systematic testing says: Lets not pull them out at random.
Lets first subdivide the big bag of boxes into smaller groups
(the pink lines), and do it in a way that tends to concentrate
the colored boxes in a few of the groups. The number of
groups needs to be much smaller than the number of boxes,
so that we can systematically reach into each group to pick
one or a few boxes.
Functional testing is one variety of partition testing, a way of
drawing the orange lines so that, when one of the boxes
within a orange group is a failure, many of the other boxes in
that group may also be failures. Functional testing means
using the program specification to draw pink lines.
(from Pezze + Young, Software Testing and Analysis,
Chapter 10)

Systematic Partition Testing

Failure (valuable test case)
The space of possible input values
(the haystack)

No failure

Failures are sparse in

the space of possible
inputs ...

If we systematically test some

cases from each part, we will
include the dense parts

... but dense in some

parts of the space

Functional testing is one way of

drawing orange lines to isolate
regions with likely failures

23

How do we choose equivalence

classes? The key is to examine input
conditions from the spec. Each input
condition induces an equivalence class
valid and invalid inputs.

Equivalence Partitioning
Input condition

Equivalence classes

range

one valid, two invalid

specific value

one valid, two invalid

member of a set

one valid, one invalid

boolean

one valid, one invalid

(larger and smaller)

(larger and smaller)

24

How do we choose representatives rom

equivalence classes? A greater number
of errors occurs at the boundaries of an
equivalence class rather than at the
center. Therefore, we specifically look
for values that are at the boundaries
both of the input domain as well as at
the output.

Boundary Analysis
Possible test case

Test at lower range (valid and invalid),

at higher range(valid and invalid), and at center

25

(from Pezze + Young, Software Testing

and Analysis, Chapter 10)

Example: ZIP Code

Input:

5-digit ZIP code

Output:

Ask
list of cities audience

What are

representative
values to test?
26

(from Pezze + Young, Software Testing

and Analysis, Chapter 10)

Valid ZIP Codes

1. with 0 cities
as output
(0 is boundary value)

2. with 1 city
as output
3. with many cities
as output
27

(from Pezze + Young, Software Testing

and Analysis, Chapter 10)

Invalid ZIP Codes

4. empty input
5. 14 characters
(4 is boundary value)

6. 6 characters
(6 is boundary value)

7. very long input

8. no digits
9. non-character data
28

Special ZIP Codes

How about a ZIP code that reads
12345; DROP TABLE orders; SELECT
* FROM zipcodes WHERE zip =

Or a ZIP code with 65536 characters

This is security testing
29

Generally, random inputs are easier to

generate, but less likely to cover parts of
the specification or the code.
See Gutjahr (1999) in IEEE
Transactions on Software Engineering
25, 5 (1999), 661-667

Gutjahrs Hypothesis
Partition testing
is more effective
than random testing.

30

The main steps of a systematic

approach to functional program testing
(from Pezze + Young, Software Testing
and Analysis, Chapter 10)

Representative Values
identify

Functional
specification

Independently
testable feature
identify

derive

Representative
values

Model
derive

generate

Test case
specifications

Test case

31

The main steps of a systematic

approach to functional program testing
(from Pezze + Young, Software Testing
and Analysis, Chapter 10)

Model-Based Testing
identify

Functional
specification

Independently
testable feature

Have a formal model

that specifies softwareidentify

behavior

derive

Representative
as
Models typically come
values
finite state machines and
decision structures
Test case

Model

derive

generate

Test case

specifications

32

As an example, consider these steps

modeling a product maintenance
process
(from Pezze + Young, Software Testing
and Analysis, Chapter 14)

Finite
State
Machine

33

based on these (informal)

requirements
(from Pezze + Young, Software Testing
and Analysis, Chapter 14)

34

Coverage Criteria
Path coverage: Tests cover every path
State coverage: Every node is executed
Transition coverage: Every edge is executed
Not feasible in practice due to infinite number of paths

A minimum testing criterion

Typically, a good coverage criterion to aim for

35

With five test cases (one color each), we

can achieve transition coverage
(from Pezze + Young, Software Testing
and Analysis, Chapter 14)

Transition
Coverage

36

Finite state machines can be used to

model for a large variety of behaviors
and thus serve as a base for testing.

State-based Testing
Protocols (e.g., network communication)
GUIs (sequences of interactions)
Objects (methods and states)
37

empty
acct

open

Heres an example of a finite state

machine representing an Account class
going through a number of states.
Transition coverage means testing each
Account method once.
(From Pressman, Software Engineering
a practitioners approach, Chapter 14)

set up
acct

setup Accnt

deposit
(initial)

Account states

deposit
working
acct

balance
credit
accntInfo

withdraw

withdrawal
(final)

dead
acct

nonworking
acct

close

38
Figure 14.3 State diagram for Account class (adapted from [KIR94])

A decision table describes under which

conditions a specific outcome comes to
be. This decision table, for instance,
determines the discount for a purchase,
depending on specific thresholds for the
amount purchased.
(from Pezze + Young, Software Testing
and Analysis, Chapter 14)

Decision Tables
Education

Individual

Education account

Current purchase >

Threshold 1
Current purchase >
Threshold 2
Special price <
scheduledprice
Special price <
Tier1
Special price <
Tier2

Out

Edu
Special
No
Special Tier1 Special Tier2 Special
discount price discount price discount price discount Price

39

Condition Coverage
Basic criterion: Test every column
Compound criterion: Test every combination
Modified condition decision criterion (MCDC):
Dont care entries () can take arbitrary values

Requires 2n tests for n conditions and is unrealistic

like basic criterion, but additionally, modify

each T/F value at least once
Again, a good coverage criterion to aim for

Nicolas and I were going

through the slides and found
that in the Functional testing
lecture, on slide 39, the Basic
criterion is swapped with the
Compound criterion
description, at least from what
we know from the Structural
testing chapter from the
Pezze&Young book. Are we

40

We modify the individual values in

column 1 and 2 to generate four
additional test cases but these are
already tested anyway. For instance,
the modified values in column 1 are
already tested in column 3.
(from Pezze + Young, Software Testing
and Analysis, Chapter 14)

MCDC Criterion
Education

Individual

Education account

T
F

Current purchase >

Threshold 1
Current purchase >
Threshold 2
Special price <
scheduledprice
Special price <
Tier1
Special price <
Tier2

Edu
Special
No
Special Tier1 Special Tier2 Special
discount price discount price discount price discount Price

Out

41

This also applies to changing the other

values, so adding additional test cases
is not necessary in this case.
(from Pezze + Young, Software Testing
and Analysis, Chapter 14)

MCDC Criterion
Education

Individual

Education account

Current purchase >

Threshold 1
Current purchase >
Threshold 2
Special price <
scheduledprice
Special price <
Tier1
Special price <
Tier2

TF

Out

Edu
Special
No
Special Tier1 Special Tier2 Special
discount price discount price discount price discount Price

42

MCDC Criterion
Education

Individual

Education account

T
F

Current purchase >

Threshold 1
Current purchase >
Threshold 2
Special price <
scheduledprice
Special price <
Tier1
Special price <
Tier2

Edu
Special
No
Special Tier1 Special Tier2 Special
discount price discount price discount price discount Price

Out

43

However, if we had not (yet) tested the

individual accounts, the MC/DC criterion
would have uncovered them.
(from Pezze + Young, Software Testing
and Analysis, Chapter 14)

MCDC Criterion
Education

Individual

Education account

Current purchase >

Threshold 1
Current purchase >
Threshold 2
Special price <
scheduledprice
Special price <
Tier1
Special price <
Tier2

T
F

Out

Edu
Special
No
Special Tier1 Special Tier2 Special
discount price discount price discount price discount Price

44

Established by a number of studies

done by E. Weyuker at AT&T. Any
explicit relationship between coverage
and error detection would mean that we
have a fixed distribution of errors over
all statements and paths, which is
clearly not the case.

Weyukers Hypothesis

The adequacy of a coverage criterion

can only be intuitively defined.

45

To decide where to put most effort in

testing, one can also examine the past
i.e., where did most defects occur in the
past. The above picture shows the
distribution of security vulnerabilities in
Firefox the redder a rectangle, the
more vulnerabilities, and therefore a
likely candidate for intensive testing.
The group of Andreas Zeller at Saarland
University researches how to mine such
information automatically and how to
predict future defects.

Learning from the past

46

Evidence: several studies, including

Zellers own evidence :-)

Paretos Law

Approximately 80% of defects

come from 20% of modules

47

The main steps of a systematic

approach to functional program testing
(from Pezze + Young, Software Testing
and Analysis, Chapter 10)

Model-Based Testing
Functional
specification

identify

Independently
testable feature
identify

derive

Representative
values

Model
derive

generate

Test case

Test case
specifications

48

The main steps of a systematic

approach to functional program testing
(from Pezze + Young, Software Testing
and Analysis, Chapter 10)

Deriving Test Case Specs

identify

specification
testable feature
Now:
need
to
take
care
of
combinations

Typically, one Representative

Input
values enumerated in previous
step
Functional
Independently
identify

uses models and

values
representative
values to generate
generate
test cases
Test case

derive

Model

derive

Test case
specifications

49

Many domains come as a combination

of individual inputs. We therefore need
to cope with a combinatorial explosion.

Combinatorial Testing
IIS

Server

Windows

OS

Linux

Apache

MySQL

Oracle

Database

50

Combinatorial Testing
Eliminate invalid combinations
Cover all pairs of combinations
Combinations typically generated
IIS only runs on Windows, for example

such as MySQL on Windows and Linux

automatically

and hopefully tested automatically, too

51

Pairwise testing means to cover every

single pair of configurations

Pairwise Testing
IIS

Windows

Apache

IIS

Linux

Apache

MySQL Oracle

IIS

Linux

MySQL Oracle

Windows

Apache

Windows

IIS

Linux

Windows

Apache

MySQL Oracle

Linux

MySQL Oracle

52

In practice, such testing needs hundreds

and hundreds of PCs in every possible
configuration Microsoft, for instance,
has entire buildings filled with every
hardware imaginable
Source: http://www.ci.newton.ma.us/
MIS/Network.htm

Testing environment
Millions of configurations
Testing on dozens of different machines
All needed to find & reproduce problems
53

The main steps of a systematic

approach to functional program testing
(from Pezze + Young, Software Testing
and Analysis, Chapter 10)

Deriving Test Case Specs

Functional
specification

identify

Independently
testable feature
identify

derive

Representative
values

Model
derive

generate

Test case

Test case
specifications

54

The main steps of a systematic

approach to functional program testing
(from Pezze + Young, Software Testing
and Analysis, Chapter 10)

Deriving Test Cases

Functional
specification

identify

Independently
testable feature

Implement test cases in code

Requires building scaffolding

Representative
identify

derive

i.e., drivers and stubs values

Model
derive

generate

Test case
specifications

Test case

55

Heres an example for automated unit

tests the well-known JUnit

Unit Tests
Directly access units (= classes, modules,
components) at their programming
interfaces

Encapsulate a set of tests as a single

syntactical unit

Available for all programming languages

(JUNIT for Java, CPPUNIT for C++, etc.)

56

The main steps of a systematic

approach to functional program testing
(from Pezze + Young, Software Testing
and Analysis, Chapter 10)

Deriving Test Cases

Functional
specification

identify

Independently
testable feature
identify

derive

Representative
values

Model
derive

generate

Test case

Test case
specifications

57

The main steps of a systematic

approach to functional program testing
(from Pezze + Young, Software Testing
and Analysis, Chapter 10)

Systematic Functional Testing

Functional
specification

identify

Independently
testable feature
identify

derive

Representative
values

Model
derive

generate

Test case

Test case
specifications

58

Summary

59