SE3633 Software Testing

Software Fault, Error and Failure

• Software Fault
• A static defect in the software.
• Software Failure
• External, incorrect behavior with respect to the requirements or other
description of the expected behavior.
• Software Error
• An incorrect internal state that is the manifestation of some fault.
• Faults in software are equivalent to design mistakes in hardware.
• Software does not degrade.

Introduction to Software Testing, Edition 2 (Ch 1)

A Concrete Example Fault: Should start
searching at 0, not 1
Test 1
public static int numZero (int [ ] arr) [ 2, 7, 0 ]
{ // Effects: If arr is null throw NullPointerException Expected: 1
// else return the number of occurrences of 0 in arr Actual: 1
int count = 0;
for (int i = 1; i < arr.length; i++)
{ Error: i is 1, not 0, on Test 2
if (arr [ i ] == 0) [ 0, 2, 7 ]
the first iteration
{ Expected: 1
Failure: none
count++; Actual: 0
}
} Error: i is 1, not 0
return count; Error propagates to the variable count
} Failure: count is 0 at the return statement

Introduction to Software Testing, Edition 2 (Ch 1)

Validation & Verification (IEEE)
• Validation
• The process of evaluating software at the end of software development to
ensure compliance with intended usage

• Verification
• The process of determining whether the products of a given phase of the
software development process fulfill the requirements established during the
previous phase

IV&V stands for “independent verification and validation”

Introduction to Software Testing, Edition 2 (Ch 1)

 Testing Goals Based on Test Process Maturity
• Level 0
• There’s no difference between testing and debugging
• Level 1
• The purpose of testing is to show correctness
• Level 2
• The purpose of testing is to show that the software doesn’t work
• Level 3
• The purpose of testing is not to prove anything specific, but to reduce the risk of
using the software
• Level 4
• Testing is a mental discipline that helps all IT professionals develop higher quality
software
Introduction to Software Testing, Edition 2 (Ch 1)
Level 0 Thinking
• Testing is the same as debugging

• Does not distinguish between incorrect behavior and

mistakes in the program

• Does not help develop software that is reliable or safe

This is what we teach undergraduate CS majors

Introduction to Software Testing, Edition 2 (Ch 1)

Level 1 Thinking
• Purpose is to show correctness
• Correctness is impossible to achieve
• What do we know if no failures?
• Good software or bad tests?
• Test engineers have no:
• Strict goal
• Real stopping rule
• Formal test technique
• Test managers are powerless
This is what hardware engineers often expect
Introduction to Software Testing, Edition 2 (Ch 1)
Level 2 Thinking
• Purpose is to show failures
• Looking for failures is a negative activity
• Puts testers and developers into an adversarial relationship
• What if there are no failures?
This describes most software companies.
How can we move to a team approach ??

Introduction to Software Testing, Edition 2 (Ch 1)

Level 3 Thinking
• Testing can only show the presence of failures
• Whenever we use software, we incur some risk
• Risk may be small and consequences unimportant
• Risk may be great and consequences catastrophic
• Testers and developers cooperate to reduce risk
This describes a few “enlightened” software companies

Introduction to Software Testing, Edition 2 (Ch 1)

Level 4 Thinking
A mental discipline that increases quality
• Testing is only one way to increase quality
• Test engineers can become technical leaders of the project
• Primary responsibility to measure and improve software quality
• Their expertise should help the developers

This is the way “traditional” engineering works

Introduction to Software Testing, Edition 2 (Ch 1)

Where Are You?
Are you at level 0, 1, or 2 ?

Is your organization at work at level 0, 1, or 2 ? Or 3?

We hope to teach you to become “change agents” in your workplace …

Advocates for level 4 thinking

Introduction to Software Testing, Edition 2 (Ch 1)

Tactical Goals : Why Each Test ?
If you don’t know why you’re conducting each test, it won’t be very
helpful
• Written test objectives and requirements must be
documented
• What are your planned coverage levels?
• How much testing is enough?
• Common objective – spend the budget … test until the ship-
date …
• Sometimes called the “date criterion”
Introduction to Software Testing, Edition 2 (Ch 1)
Here! Test This!
Offutt’s first “professional” job
Big software– big
MicroSteff program
software system
for the mac

V.1.5.1 Jan/2007
Jan/2011

1.44 MB
MF2-HD
DataLife
Verdatim

A stack of computer printouts—and no documentation

Introduction to Software Testing, Edition 2 (Ch 1)
Why Each Test ?
If you don’t start planning for each test when the functional
requirements are formed, you’ll never know why you’re conducting the
test

• 1980: “The software shall be easily maintainable”

• Threshold reliability requirements?
• What fact does each test try to verify?
• Requirements definition teams need testers!

Introduction to Software Testing, Edition 2 (Ch 1)

Cost of Not Testing
Poor Program Managers might say:
“Testing is too expensive.”
• Testing is the most time consuming and expensive part of
software development
• Not testing is even more expensive
• If we have too little testing effort early, the cost of testing
increases
• Planning for testing after development is prohibitively
expensive
Introduction to Software Testing, Edition 2 (Ch 1)
Cost60of Late Testing
Assume $1000 unit cost, per fault, 100 faults
50
40
Fault origin (%)
30
Fault detection (%)
20
10 Unit cost (X)

Software Engineering Institute; Carnegie Mellon University; Handbook CMU/SEI-96-HB-002

Introduction to Software Testing, Edition 2 (Ch 1)