Software

Testing
What & Why

Introduction

1
 Software Testing
 What is software testing ?

 Why do we need to test ?

 Cannot we live without testing ?

 How do we handle software bugs reported by the customers ?

 Who should work hard to avoid frustrations of failures ?

The developers normally ask :

How did these bugs escape from testing ?
2
 Some Software failures
Ariane 5
It took the European Space Agency 10 years and $7 billion to produce
Ariane 5, a giant rocket capable of hurling a pair of three-ton satellites
into orbit with each launch and intended to give Europe overwhelming
supremacy in the commercial space business.

The rocket was destroyed after 39 seconds of its launch, at an altitude of

two and a half miles along with its payload of four expensive and
uninsured scientific satellites.

3
 Some Software failures
When the guidance system’s own computer tried to convert one piece of
data the sideways velocity of the rocket from a 64 bit format to a 16 bit
format; the number was too big, and an overflow error resulted after 36.7
seconds.
When the guidance system shutdown, it passed control to an identical,
redundant unit, which was there to provide backup in case of just such a
failure.
Unfortunately, the second unit had failed in the identical manner, a few
milliseconds before.

Read the record report of Prof. J.L. Lions, Chairman Inquiry Board
and Director General of ESA prepared on 19th July, 1996 for
Identification of the Causes of Failure of Ariane 5:
4
 Some Software failures

Y2K problem:

It was simply because of using only last two digits of the year.

The 4 digit date format, like 1964, was shortened to 2 digit format,
like 64.

5
 Some Software failures
The Patriot Missile
o First time used in Gulf war
o Used as a defence from Iraqi Scud missiles
o Failed several times including one that killed 28 US soldiers in
Dhahran, Saudi Arabia

Reasons:
A small timing error in the system’s clock accumulated to the point that
after 14 hours, the tracking system was no longer accurate.
In the Dhahran attack, the system had been operating for more than 100
hours.

6
 Some Software failures
The Space Shuttle
Part of an abort scenario for the Shuttle requires fuel dumps to lighten the
spacecraft.

It was during the second of these dumps that a (software) crash occurred.

The fuel management module, which had performed one dump and
successfully exited, restarted when recalled for the second fuel dump...

7
 Experience of Windows XP

Released on October 25,2001

Same day company posted 18 megabytes of patches on its website for

bug fixes, compatibility updates and enhancements.

8
 Testing Process

It is basically the process of testing the newly developed software,

prior to actual use.

Good testing involves much more than just running the program a
few times to see whether it works as per specifications.

9
 What is testing ?
There are many definitions of testing. A few of them are given below:

1. Testing is the process of demonstrating that errors are not

present.
2. The purpose of testing is to show that a program performs its
intended functions correctly.

3. Testing is the process of establishing confidence that a program

does what it is supposed to do.
Incorrect

Testing

Value to the program

10
 Definition
The most appropriate definition is :

“Testing is the process of executing a program with the intent of finding

faults.”

Our input selection should have a higher probability of finding faults.

11
 What is Software testing ?

Consider a program, minimum() that reads

a set of integers and prints the smallest
integer.

12
13
 What is Software testing ?
.
Some test cases

Sr. Inputs Expected Observed Match ?

No. output output
Size Set of integers

5 6,9,2,16,19 2 2 Yes
1
7 96,11,32,9,39,99,91 9 9 Yes
2
3 7 31,36,42,16,65,76,81 16 16 Yes

4 6 28,21,36,31,30,38 21 21 Yes

5 6 106,109,88,111,114,116 88 88 Yes
6 6 61,69,99,31,21,69 21 21 Yes
7 4 6,2,9,5 2 2 Yes
8 4 99,21,7,49 7 7 Yes

14
 Situations

In the minimum() program we concentrate on some typical and

critical situations :

(i) A very short list (of inputs) with the size of 1,2 or 3
elements.
(ii) An empty list i.e. of size 0.

(iii) A list where the minimum element is the first or last element.

(iv) A list where the minimum element is negative.

(v) A list where all elements are negative.

Cont….
15
 Situations
(vi) A list where some elements are real numbers.

(vii) A list where some elements are alphabetic characters.

(viii) A list with duplicate elements.

(ix) A list where one element has a value greater than

maximum permissible limit of an integer.

16
 Situations
In the previous table, we have selected elements in every list to
cover essentially the same situation :

 A list of moderate size.

 Containing all positive integers.
 Minimum is somewhere in the middle.

17
 Some critical / typical situations of program Minimum()

Case number with Inputs Expected Observed o/p Match ?

description o/p
Size Set of Integers

a 1 90 90 2147483647 No
Case 1 b 2 12,10 10 2147483647 No
A very short list with c 2 10,12 10 2147483647 No
size 1,2 or 3 d 3 12,14,36 12 14 No
e 3 36,14,12 12 14 No
f 3 14,12,36 12 12 Yes
Case 2
An empty list i.e. of Error 2147483647 No
a 0 ----
size 0 message

Case 3 a 5 10,23,34,81,97 No
10 23
A list where minimum
element is first or last b 5 97,81,34,23,10 No
10 23
element

Case 4 a 4 10,-2,5,23 -2 2 No
A list where minimum
element is negative b 4 5,-25,20,36 -25 20 No

Cont…. 18
 Case number with Inputs Expected Observed o/p Match
description o/p ?
Size Set of Integers

Case 5 a 5 -23,-31,-45,-56,-78 -78 31 No

A list where all
elements are b 5 -6,-203,-56,-78,-2 -203 56 No
negative

Case 6 a 5 12,34.56,6.9,62.14,19 6.9 34 (Program does not take No

A list where some values for index 3,4,5)
elements are real b 5 2,3,5,6,9 2 838993460 (Program No
numbers does not take any array
value)

Case 7 a 5 23,2 I,26,6,9 6 2 (No value for index No

A list where some 3,4,5)
elements are b 1I 2,3,4,9,6,5,11,12,14,21, 2 2147483647 (Program No
22 does not take any index
characters
value)
Case 8 a 5 3,4,6,9,6 3 4 No
A list with duplicate
elements b 5 13,6,6,9,15 6 6 Yes

Case 9
Value greater than a 5 530,4294967297,23, 23 1 No
maximum permissible 46,59
limit of an integer

19
What are the possible reasons for so many
failures ?
Possible reasons

Case 1 While handling the minimum, the base value of the index and / or end value
A very short list with of the index of the usable array has not been handled properly.
size 1,2 or 3 See line no. 25 & 26

Case 2 Program proceeds without checking the size of the array.

An empty list i.e. of See line 18 & 19
size 0

Case 3 Same as for case 1

A list where minimum
element is the first or
last element

Case 4 The program converts all negative integers into positive integers
A list where minimum See line no. 22
element is negative

Case 5 Same as for case 4

A list where all
elements are negative
20
Cont….
What are the possible reasons for so many
failures ?
Possible reasons

Case 6 Scanf() has unpredictable behaviour for inputs not according to specified
Some elements are format.
real numbers

Case 7 Same as for case 6

Some elements are
alphabetic characters

Case 8 (a) Same as case 1

A case with duplicate (b) We are getting correct results because minimum value is in the
elements middle of the list and all values are positive.

Case 9 This is hardware dependent problem. This is the case of the overflow of
A list with one value maximum permissible value of the integer. Here, 32 bits integers are used.
greater than maximum
permissible limit of an
integer

21
 Reasons for observed outputs

Cases Observed Remarks

output

1 (a) 2147483647 The program has ignored first & last values of the list. This is the maximum
1 (b) 2147483647 value of a 32 bit integer to which variable minimum is initialized.
1 (c) 2147483647

1 (d) 14 The program has ignored first and last value of the list. Middle value is 14.
1 (e) 14

1 (f) 12 Ignored first & last value. Fortunately, middle value is minimum & thus the
result is correct.

2 (a) 2147483647 Same as 1 (a)

3 (a) 23 Ignored first & last value. 23 is the minimum value in the remaining list.
3 (b) 23

4 (a) 2 Ignored first & last value. It has also converted negative integer(s) to positive
4 (b) 20 integer(s).

Cont…. 22
 Reasons for observed outputs

Cases Observed Remarks

output

5 (a) 31 Same as case 4

5 (b) 56

6 (a) 34 After getting ‘.’ of 34.56, program was terminated & 34 was displayed.12 was
6 (b) 858993460 ignored due to first value:
For 6(b): Garbage value

7 (a) 2 After getting ‘I’, program terminated adruptly.

7 (b) 2147483647 Maximum value of integer

8 (a) 4 Same as case 3

8 (b) 6 Fortunately result is correct. Although, first & last value are ignored.

9 (a) 1 The program displays this value due to the overflow of the 32 bit signed integer
data type used in the program

23
 Modification in the program minimum ()
1. The program has ignored the first & last values of the list.
Line no. 25 & 26
Two faults
25 “i=1;” “i=0;”
26 “while (i<No-1)” “while (i<=No-1)”

2. The program proceeds without checking the size of the array

Line no. 17 & 18
After line 18
If (No <=0)
{
printf (“Invalid size specified”);
}
24
Modification in the program minimum ()
3. Program has converted negative values to positive values

Line no. 22

Delete this line

Modified program in fig. 2

25
26
 Results of modified program
Case number with Inputs Expected Observed o/p Match ?
description o/p
Size Set of Integers

a 1 90 90 90 Yes
Case 1 b 2 12,10 10 10 Yes
A very short list with c 2 10,12 10 10 Yes
size 1,2 or 3 d 3 12,14,36 12 12 Yes
e 3 36,14,12 12 12 Yes
f 3 14,12,36 12 12 Yes

Case 2 Error Error message

a 0 ---- Yes
An empty list i.e. of message
size 0

Case 3 a 10 Yes
5 10,23,34,81,97 10
A list where minimum
element is first or last b 10 Yes
5 97,81,34,23,10 10
element
Case 4 a 4 10,-2,5,23 -2 -2 Yes
A list where minimum
element is negative b 4 5,-25,20,36 -25 -25 Yes

Cont…. 27
 Case number with Inputs Expected Observed o/p Match
description o/p ?
Size Set of Integers

Case 5 a 5 -23,-31,-45,-56,-78 -78 -78 Yes

A list where all
elements are b 5 -6,-203,-56,-78,-2 -203 -203 Yes
negative

Case 6 a 5 12,34.56,6.9,62.14,19 6.9 12 No

A list where some
elements are real b 5.4 2,3,5,6,9 2 838993460 (Program No
numbers does not take any array
value)

Case 7 a 5 23,2 I,26,6,9 6 2 (No value for index No

A list where some 3,4,5)
elements are b 1I 2,3,4,9,6,5,11,12,14,21, 2 2147483647 (Program No
22 does not take any index
characters
value)
Case 8 a 5 3,4,6,9,6 3 3 Yes
A list with duplicate
elements b 5 13,6,6,9,15 6 6 Yes

Case 9
Value greater than a 5 530,4294967297,23, 23 1 No
maximum permissible 46,59
limit of an integer

28
 Results of modified program

Case 6 & case 7 are failed due to scanf() function parsing problem

 Do not use scanf()

 If use scanf(), display a warning message for the user before

using scanf()

29
30
 Objective
Objective is to identify weak areas.

Most appropriate definition is :

Testing is the process of executing a program with the intent of
finding faults.

31
 Why should we test ?
Software testing is expensive

not testing is more expensive and dangerous

Running a car without brakes

How much testing ?

Do we have methods to know it?

Do we have techniques to quantify it ?

32
The cost to fix errors increases drastically from specification phase to
test phase & then to maintenance phase.

33
 Who should do testing ?
 Team work
 Depends on complexity & size of the project

34
 Who should do testing ?

Developers should have a reduced role in testing

Testing persons should be curious, critical but non judge mental and
good communicators.

Testers should ask questions that developers might find not be able
to ask themselves or are awkward, irritating, insulting or even
threatening to the developers.

Most of the times, testing persons are different from developers.

Developers provide guidelines during testing.

35
 Role of Persons
Sr. Persons Roles
No.
Customer Provides funding, give requirements, approve changes
1
& some test results.
Project Manager Plans & manages the project.
2

Software Design, codes & builds the software, participates in

3 developer (s) code reviews & testing, fixes bugs, defects & short
comings.

Testing Create test plans & test specifications based on the

4 Co-ordinator (s) requirements and functional and technical documents.

5 Testing persons (s) Execute the tests and documents results.

36
 What should we test ?
We should test the program’s responses to every possible input.

All valid inputs Invalid inputs

Suppose a program requires two 8 bits integers as inputs.

Total combinations = 28 x 28
Time required to execute a test case = 1second
Total time = 18 hours

37
 What should we test ?
• How to choose reasonable number of test cases out of large pool of test
cases?

• What is the bottom line for testing?

• Execute every statement at least once
• Execute every possible path at least once
• Execute every exit of branch statement at least once

38
 What should we test ?
Executing every statement of the program at least once may seem to be
a reasonable goal.

However, many portions of the program may be missed with this type
of criteria.

39
Consider the following piece This code can be represented
of code: graphically as:

1. If (x>0)
A
2. {
3. a=a+b;
4. }
5. If (y>10) C B
6. {
7. c=c+d;
8. }
E D

40
Line Numbers Symbol for
representation

1 A
2,3,4 B
5 C
6,7,8 D
end E

41
 What should we test ?
What are the possible paths in this graph???
The possible paths are: ACE, ABCE, ACDE and ABCDE
How many test cases are sufficient for 100% statement coverage??

If we choose x=9 and y=15, all statements are covered.

Hence, only one test case is sufficient for 100% statement
coverage.

But we have missed other three paths.

Therefore this coverage may not be sufficient, even that may42also
be difficult to achieve in the real life programs.
 What should we test ?

Another dimension is to execute all possible paths of the

program.
A program path can be traced through the code from the start of the
program to program termination.

Two paths differ if the program executes different statements in

each, or executes the same statements but in different order. A
program may have many paths.

43
 Sequences
B

A F
D
G
C X EXIT
H
E
< 20 times
I
through the
loop

Here’s an example that shows that there are too many paths to
test in even a fairly simple program. This is from Myers, The Art
of Software Testing. 44
 Sequences
B

A F
D
G
C X EXIT
H
E
< 20 times
I
through the
loop

There are 5 ways to get from A to X. One of them is ABX--EXIT

45
 Sequences
B

A F
D
G
C X EXIT
H
E
< 20 times
I
through the
loop

A second path is ACDFX--EXIT

46
 Sequences
B

A F
D
G
C X EXIT
H
E
< 20 times
I
through the
loop

Third: ACDGX--EXIT

47
 Sequences
B

A F
D
G
C X EXIT
H
E
< 20 times
I
through the
loop

Fourth: ACEHX--EXIT

48
 Sequences
B
A F
D
G
C X EXIT
H
E
I < 20 times
through the
loop

Fifth: ACEIX--EXIT. There are 5 ways to get from A to the EXIT

if you go through X only once.
49
 Sequences
B

A F
D
G
C X EXIT
H
E
< 20 times
I
through the
loop
But we can go through X more than once. Here’s another path.
ACEHXABX--EXIT.

50
 Sequences
B

A F
D
G
C X EXIT
H
E
< 20 times
I
through the
loop
There are 5 ways to get to X the first time, 5 more to get back to
X the second time, so there are 5 x 5 = 25 cases for reaching
EXIT by passing through X twice.
51
 Sequences
Analyzing Myers’ example

• There are 51 + 52 + ... + 519 + 520 = 1014 = 100 trillion

paths through the program.

• It would take only a billion years to test every path

(if one could write, execute and verify a test case
every five minutes).

52
 The Impossibility Of Testing Everything

• In his monumental book Testing Object-Oriented Systems,

Robert Binder provides an excellent example of the
impossibility of testing "everything."

• Consider the following program:

int blech (int j)

{
j = j -1; // should be j = j + 1
j = j / 30000;
return j;
}
53
 The Impossibility Of Testing Everything
Note that the second line is incorrect! The function blech accepts an integer
j, subtracts one from it, divides it by 30000 (integer division, whole
numbers, no remainder) and returns the value just computed.

If integers are implemented using 16 bits on this computer executing this

software, the lowest possible input value is -32768 and the highest is
32767. Thus there are 65,536 possible inputs into this tiny program.

Will you have the time (and the stamina) to create 65,536 test cases? Of
course not.

So which input values do we choose?

Consider the following input values and their ability to detect this defect.

54
 Input (j) Expected Result Actual Result

1 0 0

42 0 0

40000 1 1

-64000 -2 -2

Note that none of the test cases chosen have detected this
defect.
In fact only four of the possible 65,536 input values will find this
defect.
What is the chance that we will choose all four?
What is the chance we will choose one of the four?
55
• 30000
• 29999
• -29999
• -30000

56
Limitation of Testing:::

Complete Testing Is Impossible

57
 Complete testing?
What do we mean by "complete testing"?
• Complete "coverage": Tested every line / branch / basis path?
• Testers not finding new bugs?
• Test plan complete?

Complete testing must mean that, at the end of

testing, you know there are no remaining
unknown bugs.

• After all, if there are more bugs, you can find them if you do more
testing. So testing couldn't yet be "complete."
58
 The Limitation Of Testing

There is a popular belief that we can test a

program completely:

 Some junior level programming texts even claim to tell us how

to do it: Test the program’s response to all possible inputs, or
test all possible paths through the program.

59
 The Limitation Of Testing
 Many managers also believe in the possibility of complete
testing. They even order the staff to do it, and assure each
other that it is being done.
 Sales brochures from some software testing companies
promise that they will fully test our code.

 Test coverage analyzers are sometimes marketed with the

promise of telling us, whether we have fully tested the code,
and what further testing we must do to achieve complete
testing.

 Many salespeople believe their software products are fully

tested and error free, and pass on this claim to customers.

60
 The Limitation Of Testing
No matter, how hard we try, how cleverly we plan, how much time
we spend, and how many staff and computers we use, we still can
not do enough testing. We still miss bugs. Some of the reasons are:

 The domain of possible inputs is too large to test

 There are too many possible paths through the program to test

 The user interface issues (and thus the design issues) are too
complex to completely test.
 If “3+3 should be 7” is written in specifications, and program
gives output of 7 (instead of 6); is it a bug?

61
 Measurement of the progress of testing

What is the status of testing ?

 We count things

 How to interpret such counts ?

 Is experiencing more failures a good news or a bad ?

The answer could be either.

62
Measurement of the progress of testing
Is high bug count a measure of the progress of testing????
A high bug count could mean that testing was thorough and very
few bugs remain. Or,
it could mean that the software simply has lots of bugs and, even
though many have been exposed, lots of them remain.
These counts may be illusive and do not help us to measure the
progress of testing.

63
 Measurement of the progress of testing
When should we release the software??
When to release the software is a sensitive decision & should be based
on the status of testing.
Although in the absence of testing standards,
– “economics”,
– “time to market” and
– “gut feeling”

have become important issues over technical considerations for the

release of any software.
Many reliability models are available with many limitations and are not
universally acceptable. 64
 Measurement of the progress of testing

Software companies are facing serious challenges in testing their

products and these challenges are growing bigger as software grows
more complex.
The first & most important thing is to recognize the complex nature of
testing and take it seriously.
Researchers should also realize that companies are ready to fund good
research ideas. However, they expect more practical and less academic
work in order to implement immediately.
We all should work hard to bridge the gap between academic research
and real industry requirements. If we do so, problems shall be minimized
& we may be able to develop standards for software testing.
65
 Software Testing

 Error, Mistake, Bug, Fault and Failure

People make errors. A good synonym is mistake.

This may be a syntax error or misunderstanding of specifications. Sometimes,

there are logical errors.
When people make mistakes while coding, we call these mistakes “bugs”.

Errors tend to propagate; a requirements error may be magnified during design

and amplified still more during coding.

66
 Software Testing
A fault is the result of an error. A fault is the representation of an error, where
representation is the mode of expression, such as narrative text, Unified
Modeling Language diagrams, hierarchy charts, and source code etc.
Defect is a good synonym for fault and bug (as per ISTQB)
International Software Testing Qualifications Board

We have faults of commission and faults of omission.

•An error of omission results in a fault in which something is missing that should be
present in the representation.

•A fault of commission occurs when we enter something into a representation that is

incorrect.

Faults of omission occur when we fail to enter correct information.

Of these two types, faults of omission are more difficult to detect and resolve.
67
•Incident—When a failure occurs, it may or may not be readily apparent to the
user (or customer or tester). An incident is the symptom associated with a
failure that alerts the user to the occurrence of a failure.
A failure occurs when a fault executes.
A particular fault may cause different failures, depending on how it has been
exercised.

The failures only occur in an executable representation, which is usually taken

to be source code, or more precisely, loaded object code.

This definition relates failures only to faults of commission

How can we deal with failures that correspond to faults of omission?

What about faults that never happen to execute, or perhaps do not execute for
a long time?

Reviews prevent many failures by finding faults; in fact, well-done reviews can
find faults of omission 68
Test—Testing is obviously concerned with errors, faults, failures, and incidents. A test is the act
of exercising software with test cases.

A test has two distinct goals:

•to find failures or
•to demonstrate correct execution.

69
 As per ISTQB
Error /Mistake
A human action that produces an incorrect result.

A bug/defect/fault
An imperfection or deficiency in a work product where it does not meet its requirements
or specifications.

Failure
An event in which a component or system does not perform a required function within
specified limits.

70
How do we deal with Errors and
Faults?

71
 Verification and Validation
 Verification and Validation
Verification: (as defined by IEEE/ANSI)
It is a process of evaluating a system or component to determine
whether the products of a given development phase satisfy the
conditions imposed at the start of that phase.
 Verification is the process of evaluating, reviewing, inspecting
and doing desk checks of work products such as requirement
specifications, design specification and code.
 It can be applied to all those things that can be reviewed in
the early phases to make sure that what comes out of that phase
is what we expected to get.
 It is a ‘human testing’ as it involves looking at the documents
72
on paper.
 Verification and Validation

Validation: (as defined by IEEE/ANSI)

It is a process of evaluating a system or component during or

at the end of development process to determine whether it
satisfies the specified requirements.

 It involves executing the actual software.

 It is a computer based testing process.
 Testing = Verification + Validation

73
 Verification and Validation

 Verification and validation are complementary: The

effectiveness of error detection suffers if one or the other is
not done.

 Each of these provides filters that are designed to catch

different kinds of problems in the product.

 Historically: Testing has been largely validation oriented.

74
 Software Testing
 Test, Test Case and Test Suite
Test and Test case terms are used interchangeably. In practice, both are
same and are treated as synonyms. Test case describes an input
description and an expected output description.

Test Case ID
Section-I Section-II
(Before Execution) (After Execution)
Purpose : Execution History:
Pre condition: (If any) Result:
Inputs: If fails, any possible reason (Optional);

Expected Outputs: Any other observation:

Post conditions: Any suggestion:
Written by: Run by:
Date: Date:

Fig. 2: Test case template

The set of test cases is called a test suite. Hence any combination of test
75
cases may generate a test suite.
 Software Testing
 Deliverables and Milestones
Different deliverables are generated during various phases of software
development.
The examples are source code, software requirement and specification
(SRS) document, software design document (SDD), installation guide, user
reference manual etc.
Milestones are the events that are used to ascertain the status of the
project.
Example, finalization of SRS is a milestone. Completion of SDD is another
milestone.
The milestones are essential for monitoring and planning the progress of
the development.

76
 Software Testing
 Alpha, Beta and Acceptance Testing
The term Acceptance Testing is used when the software is developed for a
specific customer. A series of tests are conducted to enable the customer to
validate all requirements. These tests are conducted by the end user / customer
and may range from adhoc tests to well planned systematic series of tests.

The terms alpha and beta testing are used when the software is developed as a
product for anonymous customers.
Alpha Tests are conducted at the developer’s site by some potential
customers. These tests are conducted in a controlled environment. Alpha
testing may be started when formal testing process is near completion.

Beta Tests are conducted by the customers / end users at their sites. Unlike
alpha testing, developer is not present here. Beta testing is conducted in a real
environment that cannot be controlled by the developer.
77
 Software Testing
 Static and Dynamic Testing

Static Testing should be started at earlier phases of software development and

gives very good results at reasonable cost.

Dynamic Testing refers to execute the code and see how it performs with
specific inputs.
All validation activities come in this category where execution of program is
essential.

Static Testing refers to testing activities without executing the code.

All verification activities like inspections, walkthroughs, reviews etc come under
this category of testing.
78
 Software Testing
 Testing and Debugging

The purpose of testing is to find faults and find them as early as possible.

The process used to determine the cause of the fault and to remove it is
known as debugging.

79