Software Testing

Testing 1
 Background

◼ Main objectives of a project: High Quality & High

Productivity (Q&P)
◼ Quality has many dimensions
◼ reliability, maintainability, interoperability etc.
◼ Reliability is perhaps the most important
◼ Reliability: The chances of software failing
◼ More defects => more chances of failure =>
lesser reliability
◼ Hence Quality goal: Have as few defects as
possible in the delivered software

Testing 2
 Faults & Failure

◼ Failure: A software failure occurs if the behavior

of the s/w is different from expected/specified.
◼ Fault: cause of software failure
◼ Fault = bug = defect
◼ Failure implies presence of defects
◼ A defect has the potential to cause failure.
◼ Definition of a defect is environment, project
specific

Testing 3
 Role of Testing

◼ Reviews are human processes - can not catch all

defects
◼ Hence there will be requirement defects, design
defects and coding defects in code
◼ These defects have to be identified by testing
◼ Therefore testing plays a critical role in ensuring
quality.
◼ All defects remaining from before as well as new
ones introduced have to be identified by testing.

Testing 4
 Detecting defects in Testing

◼ During testing, a program is executed with a set

of test cases
◼ Failure during testing => defects are present
◼ No failure => confidence grows, but can not say
“defects are absent”
◼ Defects detected through failures
◼ To detect defects, must cause failures during
testing

Testing 5
 Test Oracle

◼ To check if a failure has occurred when executed

with a test case, we need to know the correct
behavior
◼ I.e. need a test oracle, which is often a human
◼ Human oracle makes each test case expensive
as someone has to check the correctness of its
output

Testing 6
 Role of Test cases

◼ Ideally would like the following for test cases

◼ No failure implies “no defects” or “high quality”
◼ If defects present, then some test case causes a failure
◼ Psychology of testing is important
◼ should be to ‘reveal’ defects(not to show that it works!)
◼ test cases must be “destructive”
◼ Role of test cases is clearly very critical
◼ Only if test cases are “good”, the confidence
increases after testing

Testing 7
 Test case design

◼ During test planning, have to design a set of

test cases that will detect defects present
◼ Some criteria needed to guide test case
selection
◼ Two approaches to design test cases
◼ functional or black box
◼ structural or white box
◼ Both are complimentary; we discuss a few
approaches/criteria for both

Testing 8
 Black Box testing

◼ Software tested to be treated as a block box

◼ Specification for the black box is given
◼ The expected behavior of the system is used to
design test cases
◼ i.e test cases are determined solely from
specification.
◼ Internal structure of code not used for test case
design

Testing 9
 Black box Testing…

◼ Premise: Expected behavior is specified.

◼ Hence just test for specified expected behavior
◼ How it is implemented is not an issue.
◼ For modules,specification produced in design
specify expected behavior
◼ For system testing, SRS specifies expected
behavior

Testing 10
 Black Box Testing…
◼ Most thorough functional testing - exhaustive
testing
◼ Software is designed to work for an input space
◼ Test the software with all elements in the input space
◼ Infeasible - too high a cost
◼ Need better method for selecting test cases
◼ Different approaches have been proposed

Testing 11
 Equivalence Class partitioning

◼ Divide the input space into equivalent classes

◼ If the software works for a test case from a
class then it is likely to work for all
◼ Can reduce the set of test cases if such
equivalent classes can be identified
◼ Getting ideal equivalent classes is impossible
◼ Approximate it by identifying classes for which
different behavior is specified

Testing 12
 Equivalence class partitioning…

◼ Rationale: specification requires same behavior

for elements in a class
◼ Software likely to be constructed such that it
either fails for all or for none.
◼ E.g. if a function was not designed for negative
numbers then it will fail for all the negative
numbers
◼ For robustness, should form equivalent classes
for invalid inputs also

Testing 13
 Equivalent class partitioning..

◼ Every condition specified as input is an equivalent

class
◼ Define invalid equivalent classes also
◼ E.g. range 0< value<Max specified
◼ one range is the valid class
◼ input < 0 is an invalid class
◼ input > max is an invalid class
◼ Whenever that entire range may not be treated
uniformly - split into classes

Testing 14
 Equivalent class partitioning..

◼ Should consider equivalent classes in outputs

also and then give test cases for different
classes
◼ E.g.: Compute rate of interest given loan
amount, monthly installment, and number of
months
◼ Equivalent classes in output: + rate, rate = 0 ,-rate
◼ Have test cases to get these outputs

Testing 15
 Equivalence class…

◼ Once equivalent classes selected for each of

the inputs, test cases have to be selected
◼ Two approaches
1. Select each test case covering as many valid
equivalent classes as possible
2. Or, have a test case that covers at most one valid
class for each input plus a separate test case for
each invalid class

Testing 16
 Example

◼ Consider a program that takes 2 inputs – a

string s and an integer n
◼ Program determines n most frequent characters
◼ Tester believes that programmer may deal with
different types of characters separately
◼ A set of valid and invalid equivalence classes is
given

Testing 17
 Example..

Input Valid Equivalent Class Invalid Equivalent class

s:string 1: Contains numbers 1: non-ascii char

2: Lower case letters 2: string length > N
3: upper case letters
4: special chars
5: string length between 0-
N(max)

n:int 6: Int in valid range 3: Int out of range

Testing 18
 Example…

◼ Test cases (i.e. s , n) with first method

◼ s : string of length < N with lower case, upper case,
numbers, and special chars, and n=5 (one test case)
◼ Plus test cases for each of the invalid equivalent
classes (three test cases)
◼ Total test cases: 1+3= 4
◼ With the second approach
◼ A separate string for each type of char (i.e. a string
of numbers, one of lower case, …) + invalid cases
◼ Total test cases will be 5 + 2 = 7

Testing 19
 Boundary value analysis

◼ Programs often fail on special values

◼ These values often lie on boundary of
equivalence classes
◼ Test cases that have boundary values have high
yield
◼ These are also called extreme cases
◼ A Boundary Value test case is a set of input data
that lies on the edge of an equivalent class of
input/output

Testing 20
 BVA...

◼ For each equivalence class

◼ choose values on the edges of the class
◼ choose values just outside the edges
◼ E.g. if 0 <= x <= 1.0
◼ 0.0 , 1.0 are edges inside
◼ -0.1,1.1 are just outside
◼ E.g. a bounded list - have a null list , a maximum
value list
◼ Consider outputs also and have test cases
generate outputs on the boundary

Testing 21
 BVA…
◼ In Boundary Value Analysis we determine the
value of variables that should be used
◼ If input is a defined range, then there are 6
boundary values plus 1 normal value (total: 7)
◼ min = 1, min, min+1, max – 1, max, max+1
(boundary values
◼ .5 (normal value)
◼ If multiple inputs, how to combine them into
test cases; two strategies possible
◼ Try all possible combinations of Boundary Values of
different variables, with n variables this will have 7n
test cases!
◼ Select Boundary Value for one variable; have other
variables at normal values + 1 of all normal values
Testing 22
Boundary Value Analysis..
test cases for two variables – X and Y (13 test cases)

Testing 23
 Cause Effect graphing

◼ Equivalence classes and boundary value analysis

consider each input separately
◼ To handle multiple inputs, different combinations
of equivalent classes of inputs can be tried
◼ Number of combinations can be large – if n
different input conditions such that each
condition is valid/invalid, total: 2n
◼ Cause effect graphing helps in selecting
combinations as input conditions

Testing 24
 Cause Effect graphing

◼ Identify causes and effects in the system

◼ Cause: distinct input condition which can be true or
false
◼ Effect: distinct output condition (T/F)
◼ Identify which causes can produce which
effects; can combine causes
◼ Causes/effects are nodes in the graph and arcs
are drawn to capture dependency; Boolean
operators and/or are allowed

Testing 25
 Cause Effect graphing

◼ From the Cause Effect graph, can make a

decision table
◼ Lists combination of conditions that set different
effects
◼ Together they check for various effects
◼ Decision table can be used for forming the test
cases

Testing 26
 Cause Effect graphing: Example
◼ A bank database which allows two commands
◼ Credit acc# amt
◼ Debit acc# amt
◼ Requirements
◼ If credit and acc# valid, then credit
◼ If debit and acc# valid and amt less than balance,
then debit
◼ Invalid command - message

Testing 27
 Example…

◼ Causes
◼ C1: command is credit
◼ C2: command is debit
◼ C3: acc# is valid
◼ C4: amt is valid
◼ Effects
◼ Print “Invalid command”
◼ Print “Invalid acct#”
◼ Print “Debit amt not valid”
◼ Debit account
◼ Credit account

Testing 28
 Example…

C1: command is E1: Print “Invalid

credit command”

C2: command is E2: Print “Invalid

debit acct#”

C3: acc# E3: Print “Debit amt not

is valid valid”

C4: amt is E5: Credit

valid account

E4 :Debit
Legend: account
v: or
^: and
Testing 29
Example…
# 1 2 3 4 5

C1 0 1 x x x
C2 0 x 1 1 x
C3 x 0 1 1 1
C4 x x 0 1 1
E1 1
E2 1
E3 1
E4 1
E5 1

1.Set Effect to 1
2.Set Cause that enables that effec
true, false, don’t care
Testing 30
 Pair-wise testing
◼ Often many parameters determine the behavior
of a software system
◼ The parameters may be inputs or settings, and
take different values (or different value ranges)
◼ Many defects involve one condition (single-mode
fault), e.g. software not being able to print on
some type of printer
◼ Single mode faults can be detected by testing for
different values of different parameters
◼ If n parameters and each can take m values, we can
test for one different value for each parameter in
each test case
◼ Total test cases: m
Testing 31
 Pair-wise testing…

◼ All faults are not single-mode and software may

fail at some combinations
◼ E.g. telephone billing software does not compute
correct bill for night time calling (one parameter) to a
particular country (another parameter)
◼ E.g. ticketing system fails to book a business class
ticket (a parameter) for a child (a parameter)
◼ Multi-modal faults can be revealed by testing
different combination of parameter values
◼ This is called combinatorial testing

Testing 32
 Pair-wise testing…
◼ Full combinatorial testing not feasible
◼ For n parameters each with m values, total
combinations are nm
◼ For 5 parameters, 5 values each (total: 3125), if one
test is 5 minutes, total time > 1 month!
◼ Research suggests that most such faults are
revealed by interaction of a pair of values
◼ I.e. most faults tend to be double-mode
◼ For double mode, we need to exercise each pair
– called pair-wise testing

Testing 33
 Pair-wise testing…

◼ In pair-wise, all pairs of values have to be

exercised in testing
◼ If n parameters with m values each, between
any 2 parameters we have m*m pairs
◼ 1st parameter will have m*m with n-1 others
◼ 2nd parameter will have m*m pairs with n-2
◼ 3rd parameter will have m*m pairs with n-3, etc.
◼ Total number of pairs are m*m*n*(n-1)/2

Testing 34
 Pair-wise testing…

◼ A test case consists of some setting of the n

parameters
◼ Smallest set of test cases when each pair is
covered once only
◼ A test case can cover a maximum of (n-1)+(n-
2)+…=n(n-1)/2 pairs
◼ In the best case when each pair is covered
exactly once, we will have m2 different test
cases providing the full pair-wise coverage

Testing 35
 Pair-wise testing…

◼ Generating the smallest set of test cases that

will provide pair-wise coverage is non-trivial
◼ Efficient algorithms exist; efficiently generating
these test cases can reduce testing effort
considerably
◼ In an example with 13 parameters each with 3 values
pair-wise coverage can be done with 15 test cases
◼ Pair-wise testing is a practical approach that is
widely used in industry

Testing 36
 Pair-wise testing, Example

◼ A software product for multiple platforms and

uses browser as the interface, and is to work
with different Operating Systems
◼ We have these parameters and values
◼ Operating System (parameter A): Windows, Solaris,
Linux
◼ Memory size (B): 128M, 256M, 512M
◼ Browser (C): IE, Netscape, Mozilla
◼ Total number of pair wise combinations: 27
◼ Number of cases can be less

Testing 37
 Pair-wise testing…

Test case Pairs covered

a1, b1, c1 (a1,b1) (a1, c1) (b1,c1)

a1, b2, c2 (a1,b2) (a1,c2) (b2,c2)
a1, b3, c3 (a1,b3) (a1,c3) (b3,c3)
a2, b1, c2 (a2,b1) (a2,c2) (b1,c2)
a2, b2, c3 (a2,b2) (a2,c3) (b2,c3)
a2, b3, c1 (a2,b3) (a2,c1) (b3,c1)
a3, b1, c3 (a3,b1) (a3,c3) (b1,c3)
a3, b2, c1 (a3,b2) (a3,c1) (b2,c1)
a3, b3, c2 (a3,b3) (a3,c2) (b3,c2)

Testing 38
Stop

Testing 39
 Special cases

◼ Programs often fail on special cases

◼ These depend on nature of inputs, types of data
structures,etc.
◼ No good rules to identify them
◼ One way is to guess when the software might fail
and create those test cases
◼ Also called error guessing
◼ Play the sadist & hit where it might hurt

Testing 40
 Error Guessing
◼ Use experience and judgement to guess
situations where a programmer might make
mistakes
◼ Special cases can arise due to assumptions
about inputs, user, operating environment,
business, etc.
◼ E.g. A program to count frequency of words
◼ file empty, file non existent, file only has blanks,
contains only one word, all words are same, multiple
consecutive blank lines, multiple blanks between
words, blanks at the start, words in sorted order,
blanks at end of file, etc.
◼ Perhaps the most widely used in practice
Testing 41
 State-based Testing

◼ Some systems are state-less: for same inputs,

same behavior is exhibited
◼ Many systems’ behavior depends on the state of
the system i.e. for the same input the behavior
could be different
◼ I.e. behavior and output depend on the input as
well as the system state
◼ System state – represents the cumulative impact
of all past inputs
◼ State-based testing is for such systems

Testing 42
 State-based Testing…

◼ A system can be modeled as a state machine

◼ The state space may be too large (is a cross
product of all domains of variables)
◼ The state space can be partitioned in a few
states, each representing a logical state of
interest of the system
◼ State model is generally built from such states

Testing 43
 State-based Testing…

◼ A state model has four components

◼ States: Logical states representing cumulative impact
of past inputs to system
◼ Transitions: How state changes in response to some
events
◼ Events: Inputs to the system
◼ Actions: The outputs for the events

Transition
Input
State

Testing 44
 State-based Testing…

◼ State model shows what transitions occur and

what actions are performed
◼ Often state model is built from the specifications
or requirements
◼ The key challenge is to identify states from the
specifications/requirements which capture the
key properties but is small enough for modeling

Testing 45
 State-based Testing…

◼ State model can be created from the

specifications or the design
◼ For objects, state models are often built during
the design process
◼ Test cases can be selected from the state model
and later used to test an implementation
◼ Many criteria possible for test cases

Testing 46
 State-based Testing criteria

◼ All transaction coverage (AT): test case set T

must ensure that every transition is exercised
◼ All transitions pair coverage (ATP). T must
execute all pairs of adjacent transitions
(incoming and outgoing transition in a state)
◼ Transition tree coverage (TT). T must execute all
simple paths (i.e. a path from start to a state it
has already visited or it reaches the end)

Testing 47
 State-based testing…

◼ State Based testing focuses on testing the states

and transitions to/from them
◼ Different system scenarios get tested; some
easy to overlook otherwise
◼ State model is often done after design
information is available
◼ Hence it is sometimes called grey box testing
(not pure black box)

Testing 48
 White box testing

◼ Black box testing focuses only on functionality

◼ What the program does; not how it is implemented
◼ White box testing focuses on implementation
◼ Aim is to exercise different program structures with
the intent of uncovering errors
◼ Is also called structural testing
◼ Various criteria exist for test case design
◼ Test cases have to be selected to satisfy
coverage criteria

Testing 49
 Types of structural testing

◼ Control flow based criteria

◼ looks at the coverage of the control flow graph
◼ Data flow based testing
◼ looks at the coverage in the definition-use graph
◼ Mutation testing
◼ looks at various mutants of the program
◼ We will discuss control flow based and data flow
based criteria

Testing 50
 Control flow based criteria

◼ Considers the program as control flow graph

◼ Nodes represent code blocks – i.e. set of statements
always executed together
◼ An edge (i,j) represents a possible transfer of control
from i to j
◼ Assume a start node and an end node
◼ A path is a sequence of nodes from start to end

Testing 51
 Statement Coverage Criterion

◼ Criterion: Each statement is executed at least

once during testing
◼ I.e. set of paths executed during testing
should include all nodes
◼ Limitation: does not require a decision to
evaluate to false if no else clause
◼ E.g. : abs (x) : if ( x>=0) x = -x; return(x)
◼ The set of test cases {x = 0} achieves 100%
statement coverage, but error not detected
◼ Guaranteeing 100% coverage not always
possible due to possibility of unreachable
nodes
Testing 52
 Branch coverage

◼ Criterion: Each edge should be traversed at

least once during testing
◼ i.e. each decision must evaluate to both true
and false during testing
◼ Branch coverage implies statement coverage
◼ If multiple conditions in a decision, then all
conditions need not be evaluated to T and F

Testing 53
 Control flow based…

◼ There are other criteria too - path coverage,

predicate coverage, cyclomatic complexity
based, ...
◼ None is sufficient to detect all types of defects
(e.g. a program missing some paths cannot be
detected)
◼ They provide some quantitative handle on the
breadth of testing
◼ More used to evaluate the level of testing rather
than selecting test cases

Testing 54
 Data flow-based testing

◼ A def-use graph is constructed from the control

flow graph
◼ A statement in the control flow graph (in which
each statement is a node) can be of these types
◼ Def: represents definition of a var (i.e. when var is on
the left hand side)
◼ C-use: computational use of a var
◼ P-use: var used in a predicate for control transfer

Testing 55
 Data flow based…

◼ A def-use graph is constructed by associating

vars with nodes and edges in the control flow
graph
◼ For a node i, def(i) is the set of vars for which there
is a global def in i
◼ For a node i, C-use(i) is the set of vars for which
there is a global c-use in i
◼ For an edge, p-use(i,j) is set of vars for which there is
a p-use for the edge (i,j)
◼ Def clear path from i to j with regard to x: if no
def of x in the nodes in the path

Testing 56
 Data flow based criteria

◼ all-defs: for every node i, and every x in def(i)

there is a def-clear path
◼ For def of every var, one of its uses (p-use or c-use)
must be tested
◼ all-p-uses: all p-uses of all the definitions should
be tested
◼ All p-uses of all the defs must be tested
◼ Some-c-uses, all-c-uses, some-p-uses are some
other criteria

Testing 57
Relationship between diff criteria

Testing 58
 Tool support and test case selection

◼ Two major issues for using these criteria

◼ How to determine the coverage
◼ How to select test cases to ensure coverage
◼ For determining coverage - tools are essential
◼ Tools also tell which branches and statements
are not executed
◼ Test case selection is mostly manual - test plan is
to be augmented based on coverage data

Testing 59
 In a Project

◼ Both functional and structural should be used

◼ Test plans are usually determined using
functional methods; during testing, for further
rounds, based on the coverage, more test cases
can be added
◼ Structural testing is useful at lower levels only;
at higher levels ensuring coverage is difficult
◼ Hence, a combination of functional and
structural at unit testing
◼ Functional testing (but monitoring of coverage)
at higher levels
Testing 60
 Comparison

Code Review Structural Functional

Testing Testing
Computational M H M
Logic M H M
I/O H M H
Data handling H L H
Interface H H M
Data defn. M L M
Database H M M

Testing 61
Testing Process

Testing 62
 Testing

◼ Testing only reveals the presence of defects

◼ Does not identify nature and location of defects
◼ Identifying & removing the defect => role of
debugging and rework
◼ Preparing test cases, performing testing,
defects identification & removal all consume
effort
◼ Overall testing becomes very expensive : 30-
50% development cost

Testing 63
 Incremental Testing
◼ Goals of testing: detect as many defects as
possible, and keep the cost low
◼ Both frequently conflict - increasing testing can
catch more defects, but cost also goes up
◼ Incremental testing - add untested parts
incrementally to tested portion
◼ For achieving goals, incremental testing
essential
◼ helps catch more defects
◼ helps in identification and removal
◼ Testing of large systems is always incremental
Testing 64
 Integration and Testing

◼ Incremental testing requires incremental

‘building’ I.e. incrementally integrate parts to
form system
◼ Integration & testing are related
◼ During coding, different modules are coded
separately
◼ Integration - the order in which they should be
tested and combined
◼ Integration is driven mostly by testing needs

Testing 65
 Top-down and Bottom-up

◼ System : Hierarchy of modules

◼ Modules coded separately
◼ Integration can start from bottom or top
◼ Bottom-up requires test drivers
◼ Top-down requires stubs
◼ Both may be used, e.g. for user interfaces top-
down; for services bottom-up
◼ Drivers and stubs are code pieces written only
for testing

Testing 66
 Levels of Testing

◼ The code contains requirement defects, design

defects, and coding defects
◼ Nature of defects is different for different
injection stages
◼ One type of testing will be unable to detect the
different types of defects
◼ Different levels of testing are used to uncover
these defects

Testing 67
Levels of Testing…

User needs Acceptance testing

Requirement System testing

specification

Design Integration testing

code Unit testing

Testing 68
 Unit Testing

◼ Different modules tested separately

◼ Focus: defects injected during coding
◼ Essentially a code verification technique,
covered in previous chapter
◼ Unit Testing is closely associated with coding
◼ Frequently the programmer does Unit Testing;
coding phase sometimes called “coding and unit
testing”

Testing 69
 Integration Testing

◼ Focuses on interaction of modules in a

subsystem
◼ Unit tested modules combined to form
subsystems
◼ Test cases to “exercise” the interaction of
modules in different ways
◼ May be skipped if the system is not too large

Testing 70
 System Testing

◼ Entire software system is tested

◼ Focus: does the software implement the
requirements?
◼ Validation exercise for the system with respect
to the requirements
◼ Generally the final testing stage before the
software is delivered
◼ May be done by independent people
◼ Defects removed by developers
◼ Most time consuming test phase
Testing 71
 Acceptance Testing

◼ Focus: Does the software satisfy user needs?

◼ Generally done by end users/customer in
customer environment, with real data
◼ The software is deployed only after successful
Acceptance Testing
◼ Any defects found are removed by developers
◼ Acceptance test plan is based on the acceptance
test criteria in the SRS

Testing 72
 Other forms of testing

◼ Performance testing
◼ Tools needed to “measure” performance
◼ Stress testing
◼ load the system to peak, load generation tools
needed
◼ Regression testing
◼ Test that previous functionality works alright
◼ Important when changes are made
◼ Previous test records are needed for comparisons
◼ Prioritization of test cases needed when complete
test suite cannot be executed for a change

Testing 73
 Test Plan

◼ Testing usually starts with test plan and ends

with acceptance testing
◼ Test plan is a general document that defines the
scope and approach for testing for the whole
project
◼ Inputs are SRS, project plan, design
◼ Test plan identifies what levels of testing will be
done, what units will be tested, etc in the
project

Testing 74
 Test Plan…

◼ Test plan usually contains

◼ Test unit specifications: what units need to be tested
separately
◼ Features to be tested: these may include
functionality, performance, usability,…
◼ Approach: criteria to be used, when to stop, how to
evaluate, etc
◼ Test deliverables
◼ Schedule and task allocation
◼ Example Test Plan

Testing 75
 Test case specifications

◼ Test plan focuses on approach; does not deal

with details of testing a unit
◼ Test case specification has to be done separately
for each unit
◼ Based on the plan (approach, features,..) test
cases are determined for a unit
◼ Expected outcome also needs to be specified for
each test case

Testing 76
 Test case specifications…

◼ Together the set of test cases should detect

most of the defects
◼ Would like the set of test cases to detect any
defect, if it exists
◼ Would also like set of test cases to be small -
each test case consumes effort
◼ Determining a reasonable set of test cases is
the most challenging task of testing

Testing 77
 Test case specifications…
◼ The effectiveness and cost of testing depends
on the set of test cases
◼ Q: How to determine if a set of test cases is
good? I.e. the set will detect most of the
defects, and a smaller set cannot catch these
defects
◼ No easy way to determine goodness; usually the
set of test cases is reviewed by experts
◼ This requires test cases be specified before
testing – a key reason for having test case
specifications
◼ Test case specifications are essentially a table
Testing 78
Test case specifications…

Seq.No Condition Test Data

Expected successful
to be tested result

Testing 79
 Test case specifications…

◼ So for each testing, test case specifications are

developed, reviewed, and executed
◼ Preparing test case specifications is challenging
and time consuming
◼ Test case criteria can be used
◼ Special cases and scenarios may be used
◼ Once specified, the execution and checking of
outputs may be automated through scripts
◼ Desired if repeated testing is needed
◼ Regularly done in large projects

Testing 80
 Test case execution and analysis

◼ Executing test cases may require drivers or stubs

to be written; some tests can be automatic, others
manual
◼ A separate test procedure document may be prepared
◼ Test summary report is often an output – gives a
summary of test cases executed, effort, defects
found, etc
◼ Monitoring of testing effort is important to ensure
that sufficient time is spent
◼ Computer time also is an indicator of how testing
is proceeding

Testing 81
 Defect logging and tracking

◼ A large software system may have thousands of

defects, found by many different people
◼ Often person who fixes the defect (usually the
coder) is different from the person who finds the
defect
◼ Due to large scope, reporting and fixing of
defects cannot be done informally
◼ Defects found are usually logged in a defect
tracking system and then tracked to closure
◼ Defect logging and tracking is one of the best
practices in industry

Testing 82
 Defect logging…

◼ A defect in a software project has a life cycle of

its own, like
◼ Found by someone, sometime and logged along with
information about it (submitted)
◼ Job of fixing is assigned; person debugs and then
fixes (fixed)
◼ The manager or the submitter verifies that the defect
is indeed fixed (closed)
◼ More elaborate life cycles possible

Testing 83
Defect logging…

Testing 84
 Defect logging…

◼ During the life cycle, information about defect is

logged at different stages to help debug as well
as analysis
◼ Defects generally categorized into a few types,
and type of defects is recorded
◼ Orthogonal Defect Classification (ODC) is one
classification with categories
◼ Functional, interface, assignment, timing, documentation,
algorithm
◼ Some standard industry categories
◼ Logic, standards, user interface, component interface,
performance, documentation

Testing 85
 Defect logging…

◼ Severity of defects in terms of its impact on

software is also recorded
◼ Severity useful for prioritization of fixing
◼ One categorization
◼ Critical: Show stopper
◼ Major: Has a large impact
◼ Minor: An isolated defect
◼ Cosmetic: No impact on functionality
◼ See sample peer review form
◼ Peer Review Form

Testing 86
 Defect logging and tracking…

◼ Ideally, all defects should be closed

◼ Sometimes, organizations release software with
known defects (hopefully of lower severity only)
◼ Organizations have standards for when a
product may be released
◼ Defect log may be used to track the trend of
how defect arrival and fixing is happening

Testing 87
Defect arrival and closure trend

Testing 88
 Defect analysis for prevention

◼ Quality control focuses on removing defects

◼ Goal of defect prevention is to reduce the defect
injection rate in future
◼ Defect Prevention done by analyzing defect log,
identifying causes and then remove them
◼ Is an advanced practice, done only in mature
organizations
◼ Finally results in actions to be undertaken by
individuals to reduce defects in future

Testing 89
 Metrics - Defect removal efficiency

◼ Basic objective of testing is to identify defects

present in the programs
◼ Testing is good only if it succeeds in this goal
◼ Defect removal efficiency of a Quality Control
activity = % of present defects detected by that
Quality Control activity
◼ High Defect Removal Efficiency of a quality
control activity means most defects present at
the time will be removed

Testing 90
 Defect removal efficiency …

◼ Defect Removal Efficiency for a project can be evaluated

only when all defects are know, including delivered
defects
◼ Delivered defects are approximated as the number of
defects found in some duration after delivery
◼ The injection stage of a defect is the stage in which it
was introduced in the software, and detection stage is
when it was detected
◼ These stages are typically logged for defects
◼ With injection and detection stages of all defects, Defect
Removal Efficiency for a Quality Control activity can be
computed

Testing 91
 Defect Removal Efficiency …

◼ Defect Removal Efficiencies of different Quality

Control activities are a process property -
determined from past data
◼ Past Defect Removal Efficiency can be used as
expected value for this project
◼ Process followed by the project must be
improved for better Defect Removal Efficiency

Testing 92
 Metrics – Reliability Estimation

◼ High reliability is an important goal being

achieved by testing
◼ Reliability is usually quantified as a probability or
a failure rate
◼ For a system it can be measured by counting
failures over a period of time
◼ Measurement often not possible for software as
due to fixes reliability changes, and with one-off,
not possible to measure

Testing 93
 Reliability Estimation…

◼ Software reliability estimation models are used

to model the failure followed by fix model of
software
◼ Data about failures and their times during the
last stages of testing is used by these model
◼ These models then use this data and some
statistical techniques to predict the reliability of
the software
◼ A simple reliability model is given in the book

Testing 94
 Summary

◼ Testing plays a critical role in removing defects,

and in generating confidence
◼ Testing should be such that it catches most
defects present, i.e. a high Defect Removal
Efficiency
◼ Multiple levels of testing needed for this
◼ Incremental testing also helps
◼ At each testing, test cases should be specified,
reviewed, and then executed

Testing 95
 Summary …

◼ Deciding test cases during planning is the most

important aspect of testing
◼ Two approaches – black box and white box
◼ Black box testing - test cases derived from
specifications.
◼ Equivalence class partitioning, boundary value, cause
effect graphing, error guessing
◼ White box - aim is to cover code structures
◼ statement coverage, branch coverage

Testing 96
 Summary…

◼ In a project both used at lower levels

◼ Test cases initially driven by functionality
◼ Coverage measured, test cases enhanced using
coverage data
◼ At higher levels, mostly functional testing done;
coverage monitored to evaluate the quality of
testing
◼ Defect data is logged, and defects are tracked to
closure
◼ The defect data can be used to estimate
reliability, Defect Removal Efficiency

Testing 97