Introduction to Software

Testing

Why Do We Test Software?

 Testing in the 21st Century

 Today’s software market : Industry is going

 is much bigger through a revolution in
 is more competitive what testing means to
 has more users the success of software
 products
Embedded Control Applications
 airplanes, air traffic control  PDAs
 spaceships  memory seats
 watches  DVD players
 ovens  garage door openers
 remote controllers  cell phones

 Agile processes put increased pressure on testers

 Tests are key to functional requirements – but who builds those tests ?
 Software is a Skin that
Surrounds Our Civilization
Software Faults, Errors & Failures
 Software Fault : A static defect in the software
 Defect
 It is synonymous of fault
 Software Failure : External, incorrect behavior with
respect to the requirements or other description of the
expected behavior
 Software Error : An incorrect internal state that is the
manifestation of some fault

Faults in software are equivalent to design mistakes in

hardware.
 A Concrete Example
Fault: Should start
searching at 0, not 1

public static int numZero (int [ ] arr)

Test 1
{ // Effects: If arr is null throw NullPointerException [ 2, 7, 0 ]
// else return the number of occurrences of 0 in arr Expected: 1
int count = 0; Actual: 1
for (int i = 1; i < arr.length; i++)
{ Error: i is 1, not 0, on Test 2
if (arr [ i ] == 0) the first iteration [ 0, 2, 7 ]
{ Failure: none Expected: 1
count++; Actual: 0
}
} Error: i is 1, not 0
return count; Error propagates to the variable count
} Failure: count is 0 at the return statement

Spectacular Software Failures
NASA’s Mars lander: September 1999, crashed THERAC-25 design
due to a units integration fault Mars Polar
Lander crash
site?

 THERAC-25 radiation machine : Poor testing of

safety-critical software can cost lives : 3 patients Ariane 5:
were killed exception-handling
 Ariane 5 explosion : Very expensive bug : forced self
destruct on maiden
 Intel’s Pentium FDIV fault : Public relations flight (64-bit to 16-bit
nightmare.1 in 9 billion floating point divides conversion: about
with random parameters would produce 370 million $ lost)
inaccurate results
 Northeast Blackout of 2003
508 generating units
and 256 power
plants shut down

Affected 10 million
people in Ontario,
Canada
Affected 40 million
people in 8 US states

Financial losses of
$6 Billion USD

The alarm system in the energy management system failed due to a

software error and operators were not informed of the power
overload in the system
 Costly Software Failures
 NIST report, “The Economic Impacts of Inadequate
Infrastructure for Software Testing” (2002)
 Inadequate software testing costs the US alone between $22 and
$59 billion annually
 Better approaches could cut this amount in half
 Huge losses due to web application failures
 Financial services : $6.5 million per hour (just in USA!)
 Credit card sales applications : $2.4 million per hour (in USA)
 In Dec 2006, amazon.com’s BOGO offer turned into a
double discount
 2007 : Symantec says that most security vulnerabilities are
due to faulty software

World-wide monetary loss due to poor software is staggering

 Testing in the 21st Century
 More safety critical, real-time software
 Enterprise applications means bigger programs, more users
 Paradoxically, free software increases our expectations !
 Security is now all about software faults
 Secure software is reliable software
 The web offers a new deployment platform
 Very competitive and very available to more users
 Web apps are distributed
 Web apps must be highly reliable
 What Does This Mean?

Software testing is getting more

important

What are we trying to do when we test ?

What are our goals ?
 Validation & Verification (IEEE)

 Validation : The process of evaluating software at the end

of software development to ensure compliance with
intended usage

 Verification : The process of determining whether the

products of a given phase of the software development
process fulfill the requirements established during the
previous phase
 Cost of Not Testing
Poor Program Managers might say:
“Testing is too expensive.”
 Testing is the most time consuming and
expensive part of software development
 Not testing is even more expensive
 If we have too little testing effort early, the cost
of testing increases
 Planning for testing after development is
prohibitively expensive
 Cost of Late Testing
60
Assume $1000 unit cost, per fault, 100 faults $480K
50
40 $360K
Fault origin (%)
30
$20K $100K Fault detection (%)
20
$13K
10 Unit cost (X)
$6K
0
s gn s t
e nt s i
Te s t s t n t
em De t Te Te m
e
ir ni ion e m l o y
u U t p
q / ra st e
Re og g Sy - D
r t e s t
P In Po

Software Engineering Institute; Carnegie Mellon University; Handbook CMU/SEI-96-HB-002

Difference
Summary: Why Do We Test
Software ?
A tester’s goal is to eliminate faults as
early as possible

•
Improve quality
•
Reduce cost
•
Preserve customer satisfaction
Common Software Problems
 Incorrect calculation
 Incorrect data edits & ineffective data edits
 Incorrect matching and merging of data
 Data searches that yields incorrect results
 Incorrect processing of data relationship
 Incorrect coding / implementation of business
rules
 Inadequate software performance
Common Software Problems

Confusing or misleading data

Software usability by end users &

Obsolete Software

Unreliable results or performance

Inadequate support of business needs

Incorrect or inadequate interfaces
with other systems

Inadequate performance and security
controls

Incorrect file handling
Objectives of Testing

 Executing a program with the intent of finding

an error.
 To check if the system meets the requirements
and be executed successfully in the Intended
environment.
 To check if the system is “ Fit for purpose”.
 To check if the system does what it is expected
to do.
Objectives of Testing

 A good test case is one that has a probability of

finding an as yet undiscovered error.
 A successful test is one that uncovers a yet
undiscovered error.
 A good test is not redundant.
 A good test should be “best of breed”.
 A good test should neither be too simple nor
too complex.
Objective of a Software Tester

 Find bugs as early as possible and make sure they get

fixed.
 To understand the application well.
 Study the functionality in detail to find where the
bugs are likely to occur.
 Study the code to ensure that each and every line of
code is tested.
 Create test cases in such a way that testing is done to
uncover the hidden bugs and also ensure that the
software is usable and reliable
EVOLUTION OF SOFTWARE TESTING

Debugging-oriented Phase (Before 1957) This phase is the early

period of testing. At that time, testing basics were unknown.
Programs were written and then tested by the programmers until
they were sure that all the bugs were removed. The term used for
testing was checkout, focused on getting the system to run.

Demonstration-oriented Phase (1957–78) The term ‘debugging’

continued in this phase. However, in 1957, Charles Baker pointed
out that the purpose of checkout is not only to run the software but
also to demonstrate the correctness according to the mentioned
requirements. Thus, the scope of checkout of a program increased
from program runs to program correctness. Moreover, the purpose
of checkout was to show the absence of errors.
EVOLUTION OF SOFTWARE TESTING
Destruction-oriented Phase (1979–82) This phase can be
described as the revolutionary turning point in the history of
software testing. Myers changed the view of testing from ‘testing is
to show the absence of errors’ to ‘testing is to fi nd more and more
errors.’ He separated debugging from testing and stressed on the
valuable test cases if they explore more bugs.
Evaluation-oriented Phase (1983–87) With the concept of early
testing, it was realized that if the bugs were identifi ed at an early
stage of development, it was cheaper to debug them as compared to
the bugs found in implementation or post-implementation phases.
This phase stresses on the quality of software products such that it
can be evaluated at every stage of development. In fact, the early
testing concept was established in the form of verifi cation and
validation activities which help in producing better quality
software.
EVOLUTION OF SOFTWARE TESTING

Prevention-oriented Phase (1988–95) The evaluation model

stressed on the concept of bug-prevention as compared to the
earlier concept of bug-detection. With the idea of early detection of
bugs in earlier phases, we can prevent the bugs in implementation
or further phases. Beyond this, bugs can also be prevented in other
projects with the experience gained in similar software projects.
Process-oriented Phase (1996 onwards) In this phase, testing was
established as a complete process rather than a single phase
(performed after coding) in the software development life cycle
(SDLC). The testing process starts as soon as the requirements for
a project are specifi ed and it runs parallel to SDLC. Moreover, the
model for measuring the performance of a testing process has also
been developed like CMM. The model for measuring the testing
process is known as Testing Maturity Model (TMM).
GOALS OF SOFTWARE TESTING
GOALS OF SOFTWARE TESTING
Test Case

In the simplest form, a test case is a set of conditions or variables

under which a tester determines whether the software satisfies
requirements and functions properly.
A test case usually contains:
(Simplest Form)
TEST Case ID
Description
Inputs
Expected result
Actual result (once tested)
Pass/Fail
Test Case

A,B
if(A>B)
cout<<”A is Greater”;
else
cout<<”B is Greater”;
Endif
Test Case

A,B
if(A>B)
cout<<”A is Greater”;
else
cout<<”B is Greater”;
Endif
Test Case- TC1
A=10,B=20 Expected Output- B is Greater
Test Case- TC2
A=20,B=10 Expected Output- A is Greater
 Testing Principles

• Effective Testing not Exhaustive Testing

• Testing is not a single phase performed in SDLC
• Destructive approach for constructive testing
• Early Testing is the best policy.
• The Pareto principle applies to software testing.The
probability of the existence of an error in a section of a
program is proportional to the number of errors already
found in that section.
• Testing strategy should start at the smallest module level and
expand toward the whole program.
 Testing Principles

• Testing should also be performed by an independent team.

• All tests should be traceable to customer requirements.
• Everything must be recorded in software testing.
• Invalid inputs and unexpected behavior have a high probability
of finding an error.
• Testers must participate in specification and design reviews.
SOFTWARE TESTING AS A PROCESS

SOFTWARE
TESTING
AS A
PHASE

SOFTWARE
TESTING
AS A
PROCESS
SOFTWARE TESTING AS A PROCESS

Software testing process must be planned, specified, designed, implemented, and

quantified. Testing must be governed by the quality attributes of the software product.
Thus, testing is a dual-purpose process, as it is used to detect bugs as well as to establish
confi dence in the quality of software.

An organization, to ensure better quality software, must adopt a testing process and
consider the following points:

Testing process should be organized such that there is enough time for important and
critical features of the software

Testing techniques should be adopted such that these techniques detect maximum bugs.

Quality factors should be quantified so that there is a clear understanding in running the
testing process. In other words, the process should be driven by quantified quality goals.
In this way, the process can be monitored and measured. Testing procedures and steps
must be defined and documented.

There must be scope for continuous process improvement.