Fault, Error, and Failure

Testing, Quality Assurance, and Maintenance

Winter 2018

Prof. Arie Gurfinkel

based on slides by Prof. Lin Tan and others

Terminology, IEEE 610.12-1990

Fault -- often referred to as Bug [Avizienis’00]

–A static defect in software (incorrect lines of code)
Error
–An incorrect internal state (unobserved)
Failure
–External, incorrect behaviour with respect to the
expected behaviour (observed)

Not used consistently in literature!

2
2
What is this?

A fault?

An error?

A failure?

We need to describe specified

and desired behaviour first!

3
3
Bernd Bruegge & Allen H. Dutoit. Object-Oriented Software Engineering: Using UML, Patters, and Java
Erroneous State (“Error”)

4
4
Bernd Bruegge & Allen H. Dutoit. Object-Oriented Software Engineering: Using UML, Patters, and Java
Design Fault

5
5
Bernd Bruegge & Allen H. Dutoit. Object-Oriented Software Engineering: Using UML, Patters, and Java
Mechanical Fault

6
6
Bernd Bruegge & Allen H. Dutoit. Object-Oriented Software Engineering: Using UML, Patters, and Java
Example: Fault, Error, Failure
public static int numZero (int[] x) {
//Effects: if x==null throw NullPointerException
// else return the number of occurrences of 0 in x
int count = 0;
for (int i = 1; i <x.length; i++) {
if (x[i]==0) {
count++;
Error State: Expected State:
}
x = [2,7,0] x = [2,7,0]
}
i =1 i =0
return count;
count =0 count =0
PC=first iteration for PC=first iteration for
Fix: for(int i=0; i<x.length; i++)
x = [2,7,0], fault executed, error, no failure
x = [0,7,2], fault executed, error, failure
State of the program: x, i, count, PC
7
7
Exercise: The Program

/* Effect: if x==null throw NullPointerException.

Otherwise, return the index of the last element
in the array ‘x’ that equals integer ’y’.
Return -1 if no such element exists. */

public int findLast (int[] x, int y) {

for (int i=x.length-1; i>0; i--) {
if (x[i] == y) { return i; }
}
return -1;
}

/* test 1: x=[2,3,5], y=2;

expect: findLast(x,y) == 0
test 2: x=[2,3,5,2], y=2;
expect: findLast(x,y) == 3 */
8
8
Exercise: The Problem

Read this faulty program, which includes a test case that

results in failure. Answer the following questions.
• (a) Identify the fault, and fix the fault.
• (b) If possible, identify a test case that does not execute the fault.
• (c) If possible, identify a test case that executes the fault, but does not
result in an error state.
• (d) If possible identify a test case that results in an error, but not a
failure. Hint: Don't forget about the program counter.
• (e) For the given test case ‘test1’, identify the first error state. Be sure
to describe the complete state.

9
9
 States
State 0:
• x = [2,3,5]
• y= 2
• i = undefined
• PC = findLast(...)

10
10
States

11
11
States
Incorrect Program

Correct Program

12
12
Exercise: Solutions (1/2)

(a) The for-loop should include the 0 index:

• for (int i=x.length-1; i >= 0; i--)

(b) The null value for x will result in a NullPointerException before the loop test is
evaluated, hence no execution of the fault.
• Input: x = null; y = 3
• Expected Output: NullPointerException
• Actual Output: NullPointerException

(c) For any input where y appears in a position that is not position 0, there is no
error. Also, if x is empty, there is no error.
• Input: x = [2, 3, 5]; y = 3;
• Expected Output: 1
• Actual Output: 1

13
13
Exercise: Solutions (2/2)

(d) For an input where y is not in x, the missing path (i.e. an incorrect PC on the final
loop that is not taken, normally i = 2, 1, 0, but this one has only i = 2, 1, ) is an error,
but there is no failure.
• Input: x = [2, 3, 5]; y = 7;
• Expected Output: -1
• Actual Output: -1
(e) Note that the key aspect of the error state is that the PC is outside the loop
(following the false evaluation of the 0>0 test. In a correct program, the PC should
be at the if-test, with index i==0.
• Input: x = [2, 3, 5]; y = 2;
• Expected Output: 0
• Actual Output: -1
• First Error State:
– x = [2, 3, 5]
– y = 2;
– i = 0 (or undefined);
– PC = return -1;

14
14
RIP Model

Three conditions must be present for an error to

be observed (i.e., failure to happen):
• Reachability: the location or locations in the program
that contain the fault must be reached.
• Infection: After executing the location, the state of the
program must be incorrect.
• Propagation: The infected state must propagate to
cause some output of the program to be incorrect.

15
15
HOW DO WE DEAL WITH
FAULTS, ERRORS, AND
FAILURES?
16
16
Addressing Faults at Different Stages

Fault Fault Fault

Avoidance Detection Tolerance
Better Design, Testing, Redundancy,
Better PL, ... Debugging, ... Isolation, ...

17
17
Declaring the Bug
as a Feature

18
18
Bernd Bruegge & Allen H. Dutoit. Object-Oriented Software Engineering: Using UML, Patters, and Java
Modular Redundancy: Fault Tolerance

19
19
Bernd Bruegge & Allen H. Dutoit. Object-Oriented Software Engineering: Using UML, Patters, and Java
Patching: Fixing the Fault

20
20
Bernd Bruegge & Allen H. Dutoit. Object-Oriented Software Engineering: Using UML, Patters, and Java
Testing: Fault Detection

21
21
Bernd Bruegge & Allen H. Dutoit. Object-Oriented Software Engineering: Using UML, Patters, and Java
Testing vs. Debugging

Testing: Evaluating software by observing its

execution
Debugging: The process of finding a fault given a
failure

Testing is hard:
• Often, only specific inputs will trigger the fault into creating a
failure.
Debugging is hard:
• Given a failure, it is often difficult to know the fault.

22
22
Testing is hard

if ( x - 100 <= 0 )
if ( y - 100 <= 0 )
if ( x + y - 200 == 0 )
crash();

Only input x=100 & y=100 triggers the crash

If x and y are 32-bit integers, what is the
probability of a crash?
• 1 / 264

23
23