TESTING

Why not test software product?

And if we don’t test it properly

 Adverse Effects of
Faulty Software
• Communications: Loss or corruption of
communication media, non delivery of
data.
• Space Applications: Lost lives, launch
delays.
• Defense and Warfare: Misidentification of
friend or foe.
 Adverse Effects of Faulty
Software (Cont’d)
• Transportation: Deaths, delays, sudden
acceleration, inability to brake.
• Safety-critical Applications: Death,
injuries.
• Electric Power: Death, injuries, power
outages, long-term health hazards
(radiation).
 Adverse Effects of Faulty
Software (Cont’d)
• Money Management: Fraud, violation of
privacy, shutdown of stock exchanges and
banks, negative interest rates.
• Control of Elections: Wrong results
(intentional or non-intentional).
• Control of Jails: Technology-aided escape
attempts and successes, accidental release
of inmates, failures in software controlled
locks.
• Law Enforcement: False arrests and
imprisonments.
 Bug in Space Code
• Project Mercury’s FORTRAN code had the
following fault: DO
I=1.10 instead of ... DO I=1,10
• The fault was discovered in an analysis of why
the software did not seem to generate results
that were sufficiently accurate.
• The erroneous 1.10 would cause the loop to be
executed exactly once!
 Military Aviation Problems
• An F-18 crashed because of a missing
exception condition:
if ... then ... without the else clause that
was thought could not possibly arise.
• In simulation, an F-16 program bug caused
the virtual plane to flip over whenever it
crossed the equator, as a result of a missing
minus sign to indicate south latitude.
 Year Ambiguities
• In 1992, Mary Bandar received an
invitation to attend a kindergarten in
Winona, Minnesota, along with others born
in '88.
• Mary was 104 years old at the time.
 Year Ambiguities (Cont’d)
• Mr. Blodgett’s auto insurance rate tripled
when he turned 101.
• He was the computer program’s first driver
over 100, and his age was interpreted as
1.
• This is a double blunder because the
program’s definition of a teenager is
someone under 20!
 Dates, Times, and Integers
• The number 32,768 = 215 has caused
all sorts of grief from the overflowing of
16-bit words.
• A Washington D.C. hospital computer
system collapsed on September 19,
15
1989, 2 days after January 1, 1900,
forcing a lengthy period of manual
operation.
 Dates, Times, and Integers
(Cont’d)
• COBOL uses a two-character date field …
• The Linux term program, which allows
simultaneous multiple sessions over a
single modem dialup connection, died
word wide on October 26, 1993.
• The cause was the overflow of an int
variable that should have been defined as
an unsigned int.
 Shaky Math
• In the US, five nuclear power plants were
shut down in 1979 because of a program
fault in a simulation program used to
design nuclear reactor to withstand
earthquakes.
• This program fault was, unfortunately,
discovered after the power plants were
built!
 Shaky Math (Cont’d)
• Apparently, the arithmetic sum of a set of
numbers was taken, instead of the sum of
the absolute values.
• The five reactors would probably not have
survived an earthquake that was as strong
as the strongest earthquake ever recorded
in the area.
Therac-25 Radiation “Therapy”

• In Texas, 1986, a man received between

16,500-25,000 rads in less than 1 sec, over
an area of about 1 cm.
• He lost his left arm, and died of complications
5 months later.
• In Texas, 1986, a man received at least 4,000
rads in the right temporal lobe of his brain.
• The patient eventually died as a result of the
overdose.
 Therac-25 Radiation
“Therapy” (Cont’d)
• In Washington, 1987, a patient received
8,000-10,000 rads instead of the
prescribed 86 rads.
• The patient died of complications of the
radiation overdose.
 AT&T Bug: Hello? ... Hello?
• In mid-December 1989, AT&T installed
new software in 114 electronic switching
systems.
• On January 15, 1990, 5 million calls were
blocked during a 9 hour period nationwide.
 AT&T Bug (Cont’d)
• The bug was traced to a C program that
contained a break statement within an
switch clause nested within a loop.
• The switch clause was part of a loop.
Initially, the loop contained only if
clauses with break statements to exit
the loop.
• When the control logic became
complicated, a switch clause was
added to improve the readability of the
code ...
 Bank Generosity
• A Norwegian bank ATM consistently
dispersed 10 times the amount
required.
• Many people joyously joined the queues
as the word spread.
 Bank Generosity (Cont’d)
• A software flaw caused a UK bank to
duplicate every transfer payment request
for half an hour. The bank lost 2 billion
British pounds!
• The bank eventually recovered the funds
but lost half a million pounds in potential
interest.
 Bug in BoNY Software
• The Bank of New York (BoNY) had a $32
billion overdraft as the result of a 16-bit
integer counter that went unchecked.
• BoNY was unable to process the incoming
credits from security transfers, while the
NY Federal Reserve automatically debited
BoNY’s cash account.
 Bug in BoNY Software (Cont’d)
• BoNY had to borrow $24 billion to cover
itself for 1 day until the software was fixed.
• The bug cost BoNY $5 million in interest
payments.

Hence, we have to test

We test because

We know that we are fallible

- Paul C J
Human is to Err
- ?
 Discussion …
• Do you think bug free software is
unattainable?
– Are their technical barriers that make this
impossible?
– Is it just a question of time before we can do
this?
– Are we missing technology or processes?
Ultimate goal for software testing

Quality
Assurance
Some known statistics for you
Most bugs are not because of
mistakes in the code …

• Specification (~= 55%)

• Design (~= 25%)
• Code (~= 15%)
• Other (~= 5%)
 Relative cost of bugs
“bugs found later cost more to fix”

• Cost to fix a bug increases exponentially (10 x)

– i.e., it increases tenfold as time increases
• E.g., a bug found during specification costs $1 to
fix.
• … if found in design cost is $10
• … if found in code cost is $100
• … if found in released software cost is $1000
 Course objective and scope
The course aims at providing

• A sound conceptual foundation in the area of Software

Testing Methodologies with emphasis on concepts
and techniques for testing and analysis of software.
• Software testing at unit, subsystem and system levels;
• Specification based testing and code based testing;
Model based testing.
.
• Techniques and methods for
software test generation and validation.
• The Software Analysis: Static and Dynamic.
• Testing Object Oriented Software.
• The types of software testing:
regression and interoperability.
• The software test process and management
 The course
• Introduction
• Specification Based Testing .
• Code Based Testing
• Testing object oriented software
• Testing Types
• Test process and Management
 Introduction
• Why testing?
• Basic definitions
• Test case
• Testing life cycle
• Taxonomy
• Testing levels
 Specification Based Testing
• Equivalence Class
• Boundary Value Analysis
• Decision Tables
• Cause-Effect Graphs
• Special Value Testing
 Code Based Testing
• Statement, branch, multiple condition,
loop testing
• Path testing, McCabe path testing
• Data Flow
• Program Proving
• Static Analysis
Testing object oriented software
• Introduction to OO testing concepts
• State Based testing

Testing Types
• Integration testing
• System Testing
Test process and Management
• Software Process and Testing
• Review and Inspections
• Software Test Management- Test Plans
and Reports
• Reliability Analysis
Introduction

Testing has two fundamental purposes:

.

1. To evaluate quality or acceptability

of that which is being tested.

2. To discover problems or errors.

Further expanded into 3 groups of objectives

Demonstration Detection Prevention

Show that the system can be Discover defects, errors, and Provide information to
used with acceptable risk. deficiencies Prevent or reduce the
Number of errors

Show that products are ready Determine system capabilities Reduce the number of early
for integration or use. and limitations errors propagated through
to later phases
Demonstrate functions under Determine quality of Clarify system specifications
special conditions components, work products, and performance
and the system
Identify ways to avoid risks
And problems in the future.
 Basic definitions
Error (mistakes) - we call them BUGS
Fault ( defect) – result of an error
Failure – occurs when faults execute
Incident – symptom of failure
Test- is obviously concerned with all the above and
is the art of exercising software with test cases
Test case- has an identity and is associated with
program behavior
• Mistake – a human action that produces an incorrect
result.
• Fault [or Defect] – an incorrect step, process, or data
definition in a program.
• Failure – the inability of a system or component to
perform its required function
• within the specified performance requirement.
• Error – the difference between a computed, observed,
or measured value or
• condition and the true, specified, or theoretically correct
value or condition.
• Specification – a document that specifies in a
complete, precise, verifiable manner, the requirements,
design, behavior, or other characteristic of a system or
component, and often the procedures for determining
whether these provisions have been satisfied.
 Test case
Contents of a Test Case

"Boilerplate": author, date, purpose, test case ID

Pre-conditions (including environment)

Inputs

Expected Outputs

Observed Outputs

Pass/Fail
Taxonomy -- Boris Bezier

• Mild: e.g., a misspelled output.

• Moderate: e.g., bug impacts performance.
• Annoying: e.g., bills for $0.00 are sent.
• Disturbing: e.g., ATM won’t give you
money because of bug.
• Serious: e.g., Your paycheck transaction is
not recorded.
• Very Serious: e.g., Your paycheck is
deposited to the wrong account.
• Catastrophic: e.g., Bug starts a war.
 A Testing Life Cycle
Fix
Error
Requirement Fault
Specs Error Resolution
Fault
Fault
Design Isolation
Error
Fault
Fault
Coding
Classification
Fault incident
Testing
 Program Behaviors

Specification Program
(expected) (observed)

Missing Functionality Extra Functionality

(sins of omission) (sins of commission)

"Correct"
Portion
 Testing Program Behavior

Specification Program
(expected) (observed)

Test Cases
(verified)
 Basic Approaches

Specification Program

Functional Structural
(Black Box) (White Box)
establishes confidence seeks faults
Mainline Functional Testing Techniques
Specification based testing ( Functional)

1. Boundary Value Testing

2. Equivalence Partitions

3. Robustness Testing

4. Special Value Testing

5. Output Domain (Range) Checking

6. Cause and Effect Graphs

 Functional Testing

Ultimately, any program can be viewed as a mapping from its

Input Domain to its Output Range:

Output = F ( Input )

Domain Range
Functional testing uses information about functional mappings
to identify test cases: