Introduction to Software

Testing

Why Do We Test Software?

CS 300 SOFTWARE ENGINEERING

WEEK 14
 Testing in the 21st Century
 Software defines behavior
– network routers, finance, switching networks, other infrastructure

 Today’s software market : Industry is going through a

– is much bigger revolution in what testing
– is more competitive means to the success of
– has more users software products
 Embedded Control Applications
– airplanes, air traffic control – PDAs
– spaceships – memory seats
– watches – DVD players
– ovens – garage door openers
– remote controllers – cell phones

 Agile processes put increased pressure on testers

– Programmers must unit test – with no training or education!
– Tests are key to functional requirements – but who builds those tests ?
2
 Software is a Skin that
Surrounds Our Civilization

3
Software Faults, Errors & Failures
 Software Fault : A static defect in the software

 Software Failure : External, incorrect behavior with

respect to the requirements or other description of
the expected behavior

 Software Error : An incorrect internal state that is the

manifestation of some fault
Faults in software are equivalent to design mistakes
in hardware.
Software does not degrade.

4
 Fault and Failure Example
 A patient gives a doctor a list of symptoms
– Failures
 The doctor tries to diagnose the root cause, the
ailment
– Fault
 The doctor may look for anomalous internal
conditions (high blood pressure, irregular heartbeat,
bacteria in the blood stream)
– Errors
Most medical problems result from external attacks
(bacteria, viruses) or physical degradation as we age.
Software faults were there at the beginning and do
not “appear” when a part wears out.

5
 A Concrete Example
Fault: Should start
searching at 0, not 1

public static int numZero (int [ ] arr) Test 1

{ // Effects: If arr is null throw NullPointerException [ 2, 7, 0 ]
// else return the number of occurrences of 0 in arr Expected: 1
int count = 0; Actual: 1
for (int i = 1; i < arr.length; i++)
{ Error: i is 1, not 0, on Test 2
if (arr [ i ] == 0) the first iteration [ 0, 2, 7 ]
{ Failure: none Expected: 1
count++; Actual: 0
}
} Error: i is 1, not 0
return count; Error propagates to the variable count
} Failure: count is 0 at the return statement

6
 The Term Bug
 Bug is used informally
 Sometimes speakers mean fault, sometimes error, sometimes
failure … often the speaker doesn’t know what it means !
 This class will try to use words that have precise, defined, and
unambiguous meanings

“an analyzing process must

equally have been performed
in order to furnish the
“It has been just so in all of my inventions. The
Analytical Engine with the
first step is an intuition, and comes with a burst, necessary operative data; and
then difficulties arise—this thing gives out and that herein may also lie a
[it is] then that 'Bugs'—as such little faults and possible source of error.
difficulties are called—show themselves and Granted that the actual
months of intense watching, study and labor are mechanism is unerring in its
requisite. . .” – Thomas Edison processes, the cards may give
it wrong orders. ” – Ada,
Countess Lovelace (notes on
Babbage’s Analytical Engine)

7
 Spectacular Software Failures
 NASA’s Mars lander: September 1999, THERAC-25 design
crashed due to a units integration fault
Mars Polar
Lander crash
site?

 THERAC-25 radiation machine : Poor

testing of safety-critical software can costAriane 5:
lives : 3 patients were killed exception-handling
 Ariane 5 explosion : Millions of $$ bug : forced self
destruct on maiden
 Intel’s Pentium FDIV fault : Public relations flight (64-bit to 16-bit
nightmare conversion: about
370 million $ lost)

We need our software to be dependable

Testing is one way to assess dependability

8
 Northeast Blackout of 2003
508 generating
units and 256
power plants shut
down

Affected 10 million
people in Ontario,
Canada
Affected 40 million
people in 8 US
states
Financial losses of
$6 Billion USD

The alarm system in the energy management system failed

due to a software error and operators were not informed of
the power overload in the system
9
 Costly Software Failures
 NIST report, “The Economic Impacts of Inadequate
Infrastructure for Software Testing” (2002)
– Inadequate software testing costs the US alone between
$22 and $59 billion annually
– Better approaches could cut this amount in half
 Huge losses due to web application failures
– Financial services : $6.5 million per hour (just in USA!)
– Credit card sales applications : $2.4 million per hour (in
USA)
 In Dec 2006, amazon.com’s BOGO offer turned into a double discount
 2007 : Symantec says that most security vulnerabilities are due to faulty
software

World-wide monetary loss due to poor software is staggering

10
 Spectacular software Failures
• Boeing A220 : Engines failed after
software update allowed excessive
vibrations
• Boeing 737 Max : Crashed due to overly
aggressive software flight overrides
(MCAS)
• Toyota brakes : Dozens dead, thousands of
crashes
• Healthcare website : Crashed
repeatedly on launch—never load
tested

• Northeast blackout : 50 million people, $6

billion USD lost … alarm system failed

Software testers try to find faults before

the faults find users
11
 Testing in the 21st Century
 More safety critical, real-time software
 Embedded software is ubiquitous … check your
pockets
 Enterprise applications means bigger programs,
more users
 Paradoxically, free software increases our
expectations !
 Security is now all about software faults
– Secure software is reliable software
 The web offers a new deployment platform
– Very competitive and very available to more users
– Web apps are distributed
– Web apps must be highly reliable
Industry desperately needs our inventions ! 12
The True Cost of Software Failure

Fail watch analyzed news articles for

2016
• 606 reported software failures
• Impacted half the world’s population
• Cost a combined $1.7 trillion US dollars

Poor software is a significant drag on the

world’s economy

Not to mention frustrating

13
 What Does This Mean?

Software testing is getting more

important

What are we trying to do when we test ?

What are our goals ?

14
 Validation & Verification (IEEE)

 Validation : The process of evaluating software at the

end of software development to ensure compliance
with intended usage

 Verification : The process of determining whether the

products of a given phase of the software
development process fulfill the requirements
established during the previous phase

15
 Testing Goals Based on Test
Process Maturity
 Level 0 : There’s no difference between testing and
debugging
 Level 1 : The purpose of testing is to show
correctness
 Level 2 : The purpose of testing is to show that the
software doesn’t work
 Level 3 : The purpose of testing is not to prove
anything specific, but to reduce the risk of using the
software
 Level 4 : Testing is a mental discipline that helps all
IT professionals develop higher quality software
16
 Level 0 Thinking
 Testing is the same as debugging

 Doesnot distinguish between incorrect

behavior and mistakes in the program

 Does not help develop software that is reliable

or safe

This is what we teach undergraduate CS majors

17
 Level 1 Thinking
 Purpose is to show correctness
 Correctness is impossible to achieve
 What do we know if no failures?
– Good software or bad tests?
 Test engineers have no:
– Strict goal
– Real stopping rule
– Formal test technique
– Test managers are powerless

This is what hardware engineers often expect

18
 Level 2 Thinking
 Purpose is to show failures

 Looking for failures is a negative activity

 Puts
testers and developers into an
adversarial relationship

 What if there are no failures?

This describes most software companies.
How can we move to a team approach ??

19
 Level 3 Thinking
 Testing can only show the presence of failures
 Whenever we use software, we incur some
risk
 Risk may be small and consequences
unimportant
 Risk may be great and consequences
catastrophic
 Testers and developers cooperate to reduce
risk

This describes a few “enlightened” software companies

20
 Level 4 Thinking
A mental discipline that increases quality
 Testing is only one way to increase quality
 Test engineers can become technical leaders of the
project
 Primary responsibility to measure and improve
software quality
 Their expertise should help the developers

This is the way “traditional” engineering works

21
 Where Are You?

Are you at level 0, 1, or 2 ?

Is your organization at work at level

0, 1, or 2 ?
Or 3?

We hope to teach you to become

“change agents” in your workplace …
Advocates for level 4 thinking
22
Tactical Goals : Why Each Test ?
If you don’t know why you’re conducting
each test, it won’t be very helpful
 Writtentest objectives and requirements must
be documented
 What are your planned coverage levels?
 How much testing is enough?

 Common objective – spend the budget …

test until the ship-date …
–Sometimes called the “date criterion”

23
 Here! Test This!
Offutt’s first “professional” job
Big software– big
MicroSteff program
software system
for the mac

V.1.5.1 Jan/2007
Jan/2011

1.44 MB
MF2-HD
DataLife
Verdatim

A stack of computer printouts—and no documentation

24
 Why Each Test ?
If you don’t start planning for each test when the
functional requirements are formed, you’ll never
know why you’re conducting the test
 1980: “The software shall be easily
maintainable”
 Threshold reliability requirements?
 What fact does each test try to verify?
 Requirements definition teams need testers!

25
 Cost of Not Testing
Poor Program Managers might say:
“Testing is too expensive.”

 Testing is the most time consuming and

expensive part of software development
 Not testing is even more expensive
 If we have too little testing effort early, the
cost of testing increases
 Planning for testing after development is
prohibitively expensive

26
 Cost of Late Testing
60
Assume $1000 unit cost, per fault, 100 faults
50
40
Fault origin (%)
30
Fault detection (%)
20
Unit cost (X)
10
0

Software Engineering Institute; Carnegie Mellon University; Handbook CMU/SEI-96-HB-002

27
 Summary:
Why Do We Test Software ?

A tester’s goal is to eliminate faults

as early as possible

• Improve quality
• Reduce cost
• Preserve customer satisfaction

28
 Software
Testing Life
Cycle

(STLC)
 STLC with QA
Software Testing Life Cycle is a sequence of activities conducted during testing process
to ensure software quality goals are met.
There are 6 major phases in STLC model:

Requirement
Analysis
Test Planning Test Case
PROCESSEnvironment
Development OF STLC MODEL
Setup Test
Execution
Test Closure
 Requirement Analysis
QA interacts with various stakeholders like Client,
Business Analyst, System Analyst, Technical Lead/
Development Manager to understand the requirements in
detail to develop the project.
During this phase the QA takes an important decisions like
what are the testing types & techniques to be performed,
feasibility for automation testing implementation, etc.
 Test Planning
 It is a document which contains the plan related to all testing
activities which needs to be done to deliver a quality product.
 Senior QA Manager determines test plan strategy along with
efforts and cost estimates for the project.
 The focus of the is to describe:

What cannot be Tools used for

What to test
tested testing
Risks and
Resources Test duration contingencies
planning
 Test Case Development
– Testing team write down all test cases.
– Prepare test data.
– Test cases are reviewed by peer members or QA lead.
– Requirement Traceability Matrix is prepared. The
RTM is an industry accepted format for tracking
requirements where each test cases is mapped with the
requirements using tis RTM. We can track backward
and forward traceability.
 Test Environment Setup
 Decides the software and hardware conditions under
which a work product is tested.
 It can be done in parallel with test case development
phase.
 Generic thing may not repeat for every story.
 Needs to be discussed in sprint grooming.
 We can have multiple QA server.
 Test Execution
 The testers will carry out the testing based on the test plans
and the test cases prepared. Bugs are reported to the
development team. The development team resolves the
bugs and the system is retested to ensure that it is bug free
and ready to go live.
 Test Closure Activities
 Test closure activities are done mostly after the product is
delivered
 Test closure activities mainly comprise of four types:
– Ensure Test Completion
– Handover Test Artifacts
– Project Retrospectives
– Archive Test Work Products
 Further reading..
 https://www.softwaretestinghelp.com/types-of-
software-testing/