Module 44

Fundamentals Module 44: Software Engineering

Verification &
Validation Software Testing
Black Box Testing
White Box Testing

Development
Testing
Regression
Instructors: Abir Das and Jibesh Patra
System Department of Computer Science and Engineering
Test Plans Indian Institute of Technology, Kharagpur
LMS
QES {abir, jibesh}@cse.iitkgp.ac.in

Slides taken from NPTEL course on Object-Oriented Analysis & Design

by Prof. Partha Pratim Das

CS20202: Software Engineering 1

 Table of Contents

Module 44

1 Fundamentals
Fundamentals

Verification &
Validation
2 Verification & Validation
Black Box Testing Black Box Testing
White Box Testing White Box Testing
Development
Testing
Regression 3 Development
System Testing
Test Plans Regression
LMS
QES
System

4 Test Plans
LMS
QES

CS20202: Software Engineering 2

 Why Test?

Module 44 • Ariane 5 Flight 501

Fundamentals

Verification &
Validation
Black Box Testing
White Box Testing

Development
Testing
Regression
System
◦ Un-manned satellite-launching rocket in 1996
Test Plans ◦ Self-destructed 37 seconds after launch
LMS
QES
◦ Conversion from 64-bit floating point to 16-bit signed
integer value had caused an exception (re-used from
Ariane 4)
▷ The floating point number was larger than 32767
▷ Efficiency considerations had led to the disabling of
the exception handler
Source:
CS20202:11Software
of the most costly software errors in history
Engineering 3
 Why Test?

Module 44 • NASA’s Mars Climate Orbiter

◦ Mission to Mars in 1998, $125 Million
◦ Lost in space
Fundamentals
◦ Simple conversion from English units to metric failed
Verification &
Validation • EDS Child Support System in 2004
Black Box Testing
White Box Testing
◦ Overpay 1.9 million people
Development ◦ Underpay another 700,000
Testing
Regression
◦ US $7 billion in uncollected child support payments
System
◦ Backlog of 239,000 cases
Test Plans
LMS
◦ 36,000 new cases “stuck” in the system
QES
◦ Cost the UK taxpayers over US $1 billion
◦ Incompatible software integration
• Heathrow Terminal 5 Opening
◦ Baggage handling tested for 12K test pieces of luggage
◦ Missed to test for removal of baggage. In 10 days some
42,000 bags failed to travel with their owners, and over
CS20202: Software Engineering 4
 Why Test?

Module 44 • The Morris Worm

◦ Developed by a Cornell University student for a harmless experiment
◦ Spread wildly and crashing thousands of computers in 1988 because of a
Fundamentals coding error
Verification &
◦ It was the first widespread worm attack on the fledgling Internet
Validation ◦ The graduate student, Robert Tappan Morris, was convicted of a
Black Box Testing criminal hacking offense and fined $10,000
White Box Testing ◦ Costs for cleaning up the mess may have gone as high as $100 Million
Development ◦ Morris, who co-founded the startup incubator Y Combinator, is now a
Testing professor at the Massachusetts Institute of Technology
Regression
System
◦ A disk with the worm’s source code is now housed at the University of
Boston
Test Plans
LMS
QES

CS20202: Software Engineering 5

 Why Test?

Module 44 • Boeing Crash

◦ On March 10, 2019, Ethiopian Airlines Flight 302 crashed just minutes
after takeoff. All 157 people on board the flight died
Fundamentals ◦ On October of 2018, Lion Air Flight 610 also crashed minutes after
taking off
Verification &
Validation ◦ Both flights involved Boeing’s 737 MAX jet
Black Box Testing ◦ The software overpowered all other flight functions trying to mediate the
White Box Testing nose lift
Development ◦ Many pilots did not know this system existed - they were not re-trained
Testing on 737 MAX Jet
Regression
System Source: Boeing Software Scandal Highlights Need for Full Lifecycle Testing
Test Plans
LMS
• Airbus Crash
QES ◦ On May 9, 2015, the Airbus A400M crashed near Seville after a failed
emergency landing during its first flight
◦ Electronic Control Units (ECU) on board malfunctioned
Source: Airbus A400M plane crash linked to software fault

CS20202: Software Engineering 6

 Testing a Program

Module 44
• Input test data to the program
Fundamentals
• Observe the output
Verification & • Check if the program behaved as expected
Validation
Black Box Testing • If the program does not behave as expected:
White Box Testing

Development ◦ Note the conditions under which it failed

Testing
Regression
◦ Debug and correct
System

Test Plans
LMS
QES

CS20202: Software Engineering 7

 What’s So Hard About Testing?

Module 44
• Consider int proc1(int x, int y)
Fundamentals
• Assuming a 64 bit computer
Verification & ◦ Input space = 2128
Validation
Black Box Testing • Assuming it takes 10secs to key-in an integer pair
White Box Testing

Development
◦ It would take about a billion years to enter all possible
Testing values!
Regression
System ◦ Automatic testing has its own problems!
Test Plans
LMS
QES

CS20202: Software Engineering 8

 Testing Facts

Module 44
• Consumes largest effort among all phases
◦ Largest manpower among all other development roles
Fundamentals

Verification &
◦ Implies more job opportunities
Validation
Black Box Testing
• About 50% development effort
White Box Testing
◦ But 10% of development time?
Development
Testing ◦ How?
Regression
System • Testing is getting more complex and sophisticated every
Test Plans year
LMS
QES
◦ Larger and more complex programs
◦ Newer programming paradigms

CS20202: Software Engineering 9

 Overview of Testing

Module 44
• Testing Activities
◦ Test Suite Design
Fundamentals

Verification &
◦ Run test cases and observe results to detect failures.
Validation ◦ Debug to locate errors
Black Box Testing
White Box Testing ◦ Correct errors
Development
Testing
• Error, Faults, and Failures
Regression
System
◦ A failure is a manifestation of an error (also defect or
Test Plans bug)
LMS
QES
◦ Mere presence of an error may not lead to a failure

CS20202: Software Engineering 10

 Pesticide Effect

Module 44
• Errors that escape a fault detection technique:
◦ Can not be detected by further applications of that
Fundamentals

Verification &
technique
Validation
Black Box Testing
White Box Testing

Development
Testing
Regression
System

Test Plans
LMS
QES

• Assume we use 4 fault detection techniques and 1000 bugs:

◦ Each detects only 70% bugs
◦ How many bugs would remain?
◦ 1000 ∗ (0.3)4 = 81 bugs
CS20202: Software Engineering 11
 Fault Model

Module 44
• Types of faults possible in a program
• Some types can be ruled out
◦ Concurrency related-problems in a sequential program
Fundamentals
◦ Consider a singleton in multi-thread
Verification & #include <iostream>
Validation using namespace std;
Black Box Testing class Printer { /* THIS IS A SINGLETON PRINTER -- ONLY ONE INSTANCE */
White Box Testing bool blackAndWhite_, bothSided_;
Printer(bool bw = false, bool bs = false) : blackAndWhite_(bw), bothSided_(bs)
Development
{ cout << "Printer constructed" << endl; }
Testing
static Printer *myPrinter_; // Pointer to the Singleton Printer
Regression public:
System ~Printer() { cout << "Printer destructed" << endl; }
static const Printer& printer(bool bw = false, bool bs = false) {
Test Plans
if (!myPrinter_) // What happens on multi-thread?
LMS
myPrinter_ = new Printer(bw, bs);
QES
return *myPrinter_;
}
void print(int nP) const { cout << "Printing " << nP << " pages" << endl; }
};
Printer *Printer::myPrinter_ = 0;

int main() {
Printer::printer().print(10);
Printer::printer().print(20);
delete &Printer::printer();
return 0;
}
CS20202: Software Engineering 12
 Fault Model

Module 44 • Fault Model of an OO Program

Fundamentals

Verification &
Validation
Black Box Testing
White Box Testing

Development
Testing
Regression
System
• Hardware Fault-Model
Test Plans
LMS
◦ Simple:
QES
▷ Stuck-at 0
▷ Stuck-at 1
▷ Open circuit
▷ Short circuit
◦ Simple ways to test the presence of each
◦ Hardware testing is fault-based testing
CS20202: Software Engineering 13
 Test Cases and Test Suites

Module 44 • Each test case typically tries to establish correct working of

some functionality:
Fundamentals ◦ Executes (covers) some program elements
Verification & ◦ For restricted types of faults, fault-based testing exists
Validation
Black Box Testing
White Box Testing
• Test a software using a set of carefully designed test cases:
Development ◦ The set of all test cases is called the test suite
Testing
Regression • A test case is a triplet [I,S,O]
System

Test Plans
◦ I is the data to be input to the system
LMS ◦ S is the state of the system at which the data will be
QES
input
◦ O is the expected output of the system (called Golden)

CS20202: Software Engineering 14

 Verification versus Validation
• Verification is the process of determining
Module 44 ◦ Whether output of one phase conforms to its previous phase
◦ If we are building the system correctly
◦ Verification is concerned with phase containment of errors
• Validation is the process of determining
Fundamentals ◦ Whether a fully developed system conforms to its SRS document
◦ If we are building the correct system
Verification &
Validation
◦ Whereas the aim of validation is that the final product be error free

Black Box Testing

White Box Testing

Development
Testing
Regression
System

Test Plans
LMS
QES

Source:
CS20202: Software
Software Testing
Engineering 15
 Design of Test Cases

Module 44 • Exhaustive testing of any non-trivial system is impractical

◦ Input data domain is extremely large
Fundamentals
• Design an optimal test suite
Verification & ◦ Of reasonable size and
Validation
Black Box Testing
◦ Uncovers as many errors as possible
White Box Testing
• If test cases are selected randomly
Development
Testing
◦ Many test cases would not contribute to the significance
Regression
System
of the test suite
Test Plans ◦ Would not detect errors not already being detected by
LMS
QES
other test cases in the suite
• Number of test cases in a randomly selected test suite
◦ Not an indication of effectiveness of testing
• Testing a system using a large number of randomly selected
test cases
◦ Does not mean that many errors in the system will be
uncovered
CS20202: Software Engineering 16
 Design of Test Cases

Module 44
• Consider following example
◦ Find the maximum of two integers x and y
Fundamentals

Verification & • The code has a simple programming error

Validation
Black Box Testing if (x>y)
White Box Testing
max = x;
Development
Testing else
Regression
System max = x;
Test Plans
LMS
QES • Test suite {(x=3,y=2); (x=2,y=3)} can detect the error
• A larger test suite {(x=3,y=2); (x=4,y=3); (x=5,y=1)}
does not detect the error

CS20202: Software Engineering 17

 Design of Test Cases
• Systematic approaches are required to design an optimal test suite
Module 44
◦ Each test case in the suite should detect different errors

• There are essentially three main approaches to design test cases

Fundamentals

Verification &
Validation
Black Box Testing
White Box Testing

Development
Testing
◦ Black-box testing (Zero Knowledge)
Regression
System

Test Plans
LMS
QES

◦ White-box testing (Full Knowledge)

◦ Grey-box testing (Some Knowledge)

CS20202: Software Engineering 18
 Why Both BB and WB Testing?

Module 44
Black Box Testing
White Box Testing
• Impossible to write a test
• Does not address the
Fundamentals case for every possible set
question of whether or
Verification & of inputs and outputs
Validation not a program matches
Black Box Testing
White Box Testing
• Some code parts may the specification
Development not be reachable
Testing
• Does not tell you if all
Regression
System
of the functionality has
Test Plans been implemented
LMS • Does not tell if ex-
QES • Does not discover miss-
tra functionality has been
ing program logic
implemented.

CS20202: Software Engineering 19

 Black-Box Testing

Module 44
• Black-box testing is a method of software testing that
examines the functionality of an application without peering
Fundamentals
into its internal structures or workings
Verification &
Validation • This method of test can be applied virtually to every level
Black Box Testing
White Box Testing of software testing
Development ◦ unit
Testing
Regression ◦ integration
System
◦ system and
Test Plans
LMS ◦ acceptance
QES
• Test cases are designed using only functional specification
of the software
◦ Without any knowledge of the internal structure of the
software
• For this reason, black-box testing is also known as
functional testing or specification-based testing
CS20202: Software Engineering 20
 White-Box Testing

Module 44
• White-box testing is a method of software testing that tests
internal structures or workings of an application, as opposed
Fundamentals
to its functionality
Verification &
Validation
Black Box Testing
• In white-box testing an internal perspective of the system,
White Box Testing as well as programming skills, are used to design test cases
Development
Testing
• Designing white-box test cases
Regression
System ◦ Requires knowledge about the internal structure of
Test Plans software
LMS
QES • White-box testing is also called structural testing

CS20202: Software Engineering 21

 Grey-Box Testing

Module 44
• Grey-box testing is a combination of white-box testing and
black-box testing
Fundamentals

Verification &
• The aim of this testing is to search for the defects if any
Validation
Black Box Testing
due to improper structure or improper usage of applications
White Box Testing

Development
Testing
Regression
System

Test Plans
LMS
QES

CS20202: Software Engineering 22

 Black-Box Testing Strategies

Module 44
There are several significant approaches to design black box
test cases including
• Equivalence class partitioning
Fundamentals • Boundary value analysis
Verification & • State Transition Testing
Validation
Black Box Testing • Decision Table Testing
White Box Testing
• Graph-Based Testing
Development
Testing
• Error Guessing Technique
Regression
System Other approaches include:
Test Plans • Fuzzing Technique
LMS
QES
• All Pair Testing
• Orthogonal Array Testing
• and so on
Source: Black Box Testing Techniques with Examples

CS20202: Software Engineering 23

 Black-Box Testing:
Equivalence Class Partitioning
Module 44
• Input values to a program are partitioned into equivalence
classes
Fundamentals

Verification &
• Partitioning is done such that
Validation
Black Box Testing
◦ Program behaves in similar ways to every input value
White Box Testing
belonging to an equivalence class
Development
Testing
◦ Test the code with just one representative value from
Regression
System
each equivalence class – As good as testing using any
Test Plans other values from the equivalence classes
LMS
QES

CS20202: Software Engineering 24

 Black-Box Testing:
Equivalence Class Partitioning
Module 44
How do you determine the equivalence classes?
• Examine the input data – Few general guidelines for
determining the equivalence classes can be given
Fundamentals
◦ If the input data is specified by a range of values
Verification &
Validation ▷ For example, numbers between 1 to 5000
Black Box Testing
White Box Testing
▷ One valid and two invalid equivalence classes are
Development defined
Testing
Regression
◦ If input is an enumerated set of values
System
▷ For example, { a, b, c }
Test Plans
LMS
▷ One equivalence class for valid input values
QES
▷ Another equivalence class for invalid input values
should be defined
◦ A program reads an input value in the range of 1 and
5000
▷ Computes the square root of the input number
▷ One valid and two invalid equivalence classes are
defined – The set of negative integers, Set of
CS20202: Software Engineering 25
 Black-Box Testing:
Equivalence Class Partitioning
• Max program reads two non-negative integers and spits the larger one
Module 44
Equivalence Class Condition Test Case
EC 1 (Greater) x >y (5, 2)
EC 2 (Smaller) x <y (3, 7)
Fundamentals EC 3 (Equal) x =y (4, 4)
Verification &
Validation
• QES program reads (a, b, c) and solves: ax 2 + bx + c = 0
Black Box Testing Equivalence Class Condition Test Case
White Box Testing
Infinite roots a=b=c=0 (0,0,0)
Development No root a = b = 0; c ̸= 0 (0,0,2)
Testing
Single root a = 0, b ̸= 0 (0,2,-4)
Regression
System
Repeated roots a ̸= 0; b ∗ b − 4 ∗ a ∗ c = 0 (4,4,1)
Distinct roots a ̸= 0; b ∗ b − 4 ∗ a ∗ c > 0 (1,-5,6)
Test Plans
LMS
Complex roots a ̸= 0; b ∗ b − 4 ∗ a ∗ c < 0 (2,3,4)
QES

CS20202: Software Engineering 26

 Black-Box Testing:
Boundary Value Analysis (BVA)
Module 44 • Some typical programming errors occur
◦ At boundaries of equivalence classes
Fundamentals
◦ Might be purely due to psychological factors
Verification & • Programmers often fail to see
Validation
Black Box Testing ◦ Special processing required at the boundaries of
White Box Testing
equivalence classes
Development
Testing
Regression
System

Test Plans
LMS
QES

CS20202: Software Engineering 27

 Black-Box Testing:
Boundary Value Analysis (BVA)
Module 44 • Some typical programming errors occur
◦ At boundaries of equivalence classes
◦ Might be purely due to psychological factors
Fundamentals

Verification &
• Programmers often fail to see
Validation
Black Box Testing
◦ Special processing required at the boundaries of
White Box Testing
equivalence classes
Development
Testing • Programmers may improperly use < instead of ≤
Regression
System • Boundary value analysis
Test Plans
LMS
◦ Select test cases at the boundaries of equivalence classes
QES
• For a function that computes the square root of an integer
in the range of 1 and 5000
◦ Test cases must include the values: { 0, 1, 5000, 5001 }
• QES program reads (a, b, c) and solves: ax 2 + bx + c = 0
◦ a = 0 is a boundary. Check if this test works well
◦ b ∗ b − 4 ∗ a ∗ c = 0 is a boundary. Check for the test 28
CS20202: Software Engineering
 Black-Box Testing:
State Transition Testing
Module 44 • This technique usually considers the state, outputs, and
inputs of a system during a specific period
Fundamentals
• Based on the type of software that is tested, it checks for
Verification & the behavioral changes of a system in a particular state or
Validation
Black Box Testing
another state while maintaining the same inputs
White Box Testing
• The test cases for this technique are created by checking
Development
Testing the sequence of transitions and state or events among the
Regression
System
inputs
Test Plans • The whole set of test cases will have the traversal of the
LMS
QES expected output values and all states
Source: Black Box Testing Techniques with Examples

CS20202: Software Engineering 29

 Black-Box Testing:
State Transition Testing
• Example: We need to perform black box testing for a login screen which
Module 44
allows a maximum of three attempts before the login is locked. Assuming
that the user-id is correct, design the test suite.

Fundamentals

Verification &
Validation
Black Box Testing
White Box Testing

Development
Testing
Regression
System

Test Plans
LMS
QES
Testcase # Password for Trial State
Trial 1 Trial 2 Trial 3 Golden
(1) Correct X X Access
(2) Wrong Correct X Access
(3) Wrong Wrong Correct Access
(4) Wrong Wrong Wrong Locked
Source: Black Box Testing Techniques with Examples

CS20202: Software Engineering 30

 Black-Box Testing:
Decision Table Testing
• In some instances, the inputs combinations can become very complicated for
Module 44
tracking several possibilities
• Such complex situations rely on decision tables, as it offers the testers an
organized view about the inputs combination and the expected output
Fundamentals
• This technique is identical to the graph-based testing technique; the major
Verification & difference is using tables instead of diagrams or graphs
Validation
Black Box Testing • Example 1:
White Box Testing

Development
Testing
Regression
System

Test Plans
LMS
QES

• Example 2: States of various leaves in LMS Testplan

Source: Black Box Testing Techniques with Examples

CS20202: Software Engineering 31

 Black-Box Testing:
Graph-Based Testing
• This technique of Black box testing involves a graph drawing that depicts the
Module 44
link between the causes (inputs) and the effects (output), which trigger the
effects
• This testing utilizes different combinations of output and inputs. It is a
Fundamentals helpful technique to understand the software’s functional performance, as it
Verification & visualizes the flow of inputs and outputs in a lively fashion
Validation
Black Box Testing
• Example:
White Box Testing

Development
Testing
Regression
System

Test Plans
LMS
QES

Source: Black Box Testing Techniques with Examples

CS20202: Software Engineering 32

 Black-Box Testing:
Error Guessing Technique
Module 44 • This testing technique is capable of guessing the erroneous
output and inputs to help the tester fix it easily
Fundamentals
• It is solely based on judgment and perception of the earlier
Verification & end user experience.
Validation
Black Box Testing Source: Black Box Testing Techniques with Examples
White Box Testing

Development
Testing
Regression
System

Test Plans
LMS
QES

CS20202: Software Engineering 33

 Types of Black-Box Testing

Module 44 • Functional Testing:

This type of testing is useful for the testers in identifying
the functional requirements of a software or system
Fundamentals

Verification & • Regression Testing:

Validation
Black Box Testing
This testing type is performed after the system
White Box Testing maintenance procedure, upgrades or code fixes to know the
Development
Testing
impact of the new code over the earlier code
Regression
System
• Non-Functional Testing:
Test Plans This testing type is not connected with testing for any
LMS
QES
specific functionality but relates to non-functional
parameters like usability, scalability, and performance

CS20202: Software Engineering 34

 White-Box Testing Strategies

Module 44 • Coverage-based
◦ Design test cases to cover certain program elements
Fundamentals • Fault-based
Verification & ◦ Design test cases to expose some category of faults
Validation
Black Box Testing • There exist several popular white-box testing methodologies
White Box Testing

Development ◦ Coverage Based

Testing
Regression
▷ Statement / Line Coverage
System
▷ Function / Call Coverage
Test Plans
LMS
▷ Branch Coverage
QES
▷ Condition Coverage
▷ Path Coverage
▷ MC/DC Coverage
◦ Fault Based
▷ Mutation Testing
▷ Data Flow Testing
CS20202: Software Engineering 35
 White Box Testing:
Coverage-Based Testing vs Fault-Based Testing
Module 44
• Idea behind coverage-based testing
◦ Design test cases so that certain program elements are
Fundamentals

Verification &
executed (or covered)
Validation ◦ Example: statement coverage, path coverage, etc.
Black Box Testing
White Box Testing
• Idea behind fault-based testing
Development
Testing ◦ Design test cases that focus on discovering certain types
Regression
System
of faults
Test Plans ◦ Example: Mutation testing
LMS
QES

CS20202: Software Engineering 36

 Stronger, Weaker, and Complementary Testing

Module 44
• Stronger and Weaker Testing: Test cases are a super-set of a weaker testing
◦ A stronger testing covers at least all the elements of the elements covered
by a weaker testing
Fundamentals

Verification &
Validation
Black Box Testing
White Box Testing

Development
Testing
• Complimentary Testing
Regression
System

Test Plans
LMS
QES

• Stronger, Weaker & Complimentary Testing

CS20202: Software Engineering 37

 White Box Testing:
Statement / Line Coverage
• Statement coverage methodology
Module 44 ◦ Design test cases so that every statement in the program is executed at least once

• The principal idea

◦ Unless a statement is executed
Fundamentals ◦ We do not know if an error exists in that statement

Verification & • Observe that a statement behaves properly for one i/p
Validation ◦ No guarantee that it will behave correctly for all i/p values
Black Box Testing
White Box Testing
• Line Coverage
◦ Most tools (like gcov, lcov) actually compute the coverage for the source lines
Development ◦ So it multiple statements are written in a single line, the coverage data may be inaccurate. For
Testing example, there are two statements in the following line that will be counted as one only
Regression x = 5; y = 6;
System

Test Plans ◦ While the above may be okay, conditional statements should be placed in separate lines for proper
LMS
statement coverage. For example, for
QES if (x > y) max = x;
max = y;

we would never now if the statement max = x; has actually been executed or not
◦ Single statement in every source line is a common coding guideline

CS20202: Software Engineering 38

 White Box Testing:
Statement / Line Coverage
• Coverage measurement
Module 44
◦ Statement Coverage = #executed statements
#statements
∗ 100%
◦ Line Coverage = #executed lines
#lines
∗ 100%
Fundamentals
◦ Rationale: a fault in a statement can only be revealed by executing the faulty statement. Consider
Euclid’s GCD algorithm:
Verification & int f1(int x, int y) {
Validation while (x != y) {
Black Box Testing if (x>y)
White Box Testing x = x - y;
else y = y - x;
Development }
Testing return x;
Regression }
System

Test Plans • By choosing the test set { (x=3,y=3), (x=4,y=3), (x=3,y=4) }, all
LMS
statements are executed at least once
QES
• Note that { (x=4,y=3) } or { (x=3,y=4) } itself will cover all lines due the
the iterations of the while loop. However, it is customary to keep the
analysis simple (mostly through a single flow) and include all of them in the
test suite

CS20202: Software Engineering 39

 White Box Testing:
Function / Call Coverage
• Function Coverage methodology
Module 44 ◦ Design test cases so that every function in the program is called at least once
◦ This metric reports the call count of a function; it does not pay mind to the execution of the body
of the function
◦ It is thus generally useful as an initial assessment of a project’s coverage, but higher-order metrics
Fundamentals are generally required for more in-depth analysis

Verification & • Call Coverage methodology

Validation ◦ Design test cases so that every function call in the program is executed at least once
Black Box Testing ◦ In contrast to Function Coverage (which is about execution of the function itself), Call Coverage
mandates that each call found in the code is executed at least once
White Box Testing
◦ 100% Function Coverage therefore does not imply 100% Call Coverage
Development ◦ The reverse is also not necessarily true either - there may be functions that are not called
Testing anywhere in the code (unused functions)
Regression
System
• Coverage measurement
Test Plans
◦ Function Coverage = #function called at least once
#total functions
∗ 100%
LMS ◦ Call Coverage = #call sites for functions called
#total call sites of functions
∗ 100%
QES

CS20202: Software Engineering 40

 White Box Testing:
Branch Coverage
• Test cases are designed such that
Module 44
◦ Different branch conditions – are given true and false values in turn
• Branch testing guarantees statement coverage
Fundamentals
◦ A stronger testing compared to the statement coverage-based testing
◦ Why?
Verification &
Validation 1: cin >> x;
Black Box Testing 2: if (0 == x)
White Box Testing 3: x = x + 1;
Development 4: y = 5;
Testing
Regression
System
Note that, {(x = 0)} covers lines {1, 2, 3, 4} while {(x = 1)} covers
only lines {1, 2, 4}. So with {(x = 0)}, we get 100% statement
Test Plans
coverage. But then, did we check for the jump from line 2 to 4 for the
LMS
QES
false condition? This condition did not get tested. So we need {(x =
0), (x = 1)} for 100% branch coverage and it obviously leads to 100%
statement coverage.

How do we get 100% branch coverage for:

1: if (true)
2: x = x + 1;
3: y = 5;
CS20202: Software Engineering 41
 White Box Testing: Branch Coverage

Module 44 • Example:
0: int f1(int x, int y) {
1: while (x != y) {
2: if (x > y)
3: x = x - y;
Fundamentals 4: else y = y - x;
Verification & 5: }
Validation 6: return x;
7: }
Black Box Testing
White Box Testing

Branches: {1-2, 1-6, 2-3, 2-4, 3-5, 4-5, 5-1}

Development
Testing • Test cases for branch coverage can be
Regression
◦ (x=3,y=3): {1-6}: {1, 6}
System
◦ (x=4,y=3): {1-2, 2-3}: {1, 2, 3, 5}
Test Plans ◦ (x=3,y=4): {1-2, 2-4}: {1, 2, 4, 5}
LMS
QES
• Adequacy criterion: Each branch (in CFG) must be executed at least once
◦ Branch Coverage = #executed branches
#branches
∗ 100%

• Traversing all edges of a graph causes all nodes to be visited

◦ So test suites that satisfy the branch adequacy criterion for a program also satisfy the statement
adequacy criterion for the same program

• The converse is not true

◦ A statement-adequate (or node-adequate) test suite may not be branch-adequate (edge-adequate)

CS20202: Software Engineering 42

 White Box Testing:
Condition Coverage
Module 44
All Branches can still miss conditions
• Sample fault: missing operator (negation)
Fundamentals

Verification &
digit_high == 1 || digit_low == -1
Validation
Black Box Testing
White Box Testing
• Branch adequacy criterion can be satisfied by varying only
Development
Testing
digit low
Regression
System ◦ The faulty sub-expression might not be tested
Test Plans ◦ Even though we test both outcomes of the branch
LMS
QES

CS20202: Software Engineering 43

 White Box Testing:
Condition Coverage
Module 44
• Test cases are designed such that
◦ Each component of a composite conditional expression
Fundamentals

Verification & ▷ Given both true and false values

Validation
Black Box Testing
◦ Consider the conditional expression
White Box Testing
▷ ((c1.and.c2).or.c3)
Development
Testing ◦ Each of c1, c2, and c3 are exercised at least once
Regression
System
▷ That is, given true and false values
Test Plans
LMS • Basic condition testing
QES
◦ Adequacy criterion: each basic condition must be
executed at least once
• Coverage
◦ Condition Coverage = #truth values taken by all basic conditions
2∗#basic conditions
∗ 100%

CS20202: Software Engineering 44

 White Box Testing:
Branch Testing
Module 44
• Branch testing is the simplest condition testing strategy
◦ Compound conditions appearing in different branch
Fundamentals

Verification &
statements
Validation
Black Box Testing
▷ Are given true and false values
White Box Testing
◦ Condition testing
Development
Testing ▷ Stronger testing than branch testing
Regression
System ◦ Branch testing
Test Plans
LMS
▷ Stronger than statement coverage testing
QES

CS20202: Software Engineering 45

 White Box Testing:
Condition Coverage
Module 44
• Consider a boolean expression having n components
◦ For condition coverage we require 2n test cases
Fundamentals

Verification & • Condition coverage-based testing technique

Validation
Black Box Testing ◦ Practical only if n (the number of component
White Box Testing
conditions) is small
Development
Testing • Commonly known as Multiple Condition Coverage
Regression
System (MCC), Multicondition Coverage and Condition
Test Plans Combination Coverage
LMS
QES

CS20202: Software Engineering 46

 White Box Testing:
Modified Condition / Decision (MC/DC)
Module 44 • Motivation
◦ Effectively test important combinations of conditions, without
exponential blowup in test suite size
Fundamentals ◦ Important combinations means: Each basic condition shown to
independently affect the outcome of each decision
Verification &
Validation • Requires
Black Box Testing
White Box Testing
◦ For each basic condition C, two test cases obtained
◦ Values of all evaluated conditions except C are the same
Development
Testing
◦ Compound condition as a whole evaluates to true for one and false for
Regression
the other
System
• MC/DC stands for Modified Condition / Decision Coverage
Test Plans
LMS
• A kind of Predicate Coverage technique
QES ◦ Condition: Leaf level Boolean expression.
◦ Decision: Controls the program flow
• Main Idea
◦ Each condition must be shown to independently affect the outcome of a
decision, that is, the outcome of a decision changes as a result of
changing a single condition

CS20202: Software Engineering 47

 White Box Testing:
MC/DC in action: The Cup of Coffee Example
To make a cup of coffee, we would need ALL of the following: a kettle, a cup and coffee. If any of the
Module 44 components were missing, we would not be able to make our coffee. Or, to express this another way:

if (kettle && cup && coffee)

return cup_of_coffee;
else
Fundamentals
return false;
Verification &
Validation Or to illustrate it visually:
Black Box Testing
White Box Testing

Development
Testing
Regression
System

Test Plans
LMS
QES

• Tests 4 & 8 demonstrate that ‘kettle’ can independently affect the outcome
• Tests 6 & 8 demonstrate that ‘mug’ can independently affect the outcome
• Tests 7 & 8 demonstrate that ‘coffee’ can independently affect the outcome

Source: What is MC/DC?

CS20202: Software Engineering 48
 White Box Testing:
MC/DC in action: Flow Check
A sample C/C++ function with a decision composed of OR and AND expressions illustrates the difference
Module 44 between Modified Condition/Decision Coverage and a coverage of all possible combinations as required by
MCC:

bool isSilent(int *line1, int *line2)

{
Fundamentals
if ((!line1 || *line1 <= 0) && (!line2 || *line2 <= 0))
Verification & return true;
Validation else
return false;
Black Box Testing
}
White Box Testing

Development Or to illustrate it visually:

Testing
Regression Source: Modified Condition/Decision Coverage (MC/DC)
System

Test Plans
LMS
QES

CS20202: Software Engineering 49

 White Box Testing:
Modified Condition / Decision (MC/DC)
Cond = (((a || b) && c) || d) && e
Module 44 Condition Coverage Test Cases
Every condition in the decision has taken all possible outcomes at least once
# a b c d e Cond # a b c d e Cond
0: F F F F F F 16: T F F F F F
1: F F F F T F 17: T F F F T F
Fundamentals 2: F F F T F F 18: T F F T F F
3: F F F T T T 19: T F F T T T
Verification &
4: F F T F F F 20: T F T F F F
Validation
5: F F T F T F 21: T F T F T T
Black Box Testing
6: F F T T F F 22: T F T T F F
White Box Testing 7: F F T T T T 23: T F T T T T
8: F T F F F F 24: T T F F F F
Development
9: F T F F T F 25: T T F F T F
Testing
10: F T F T F F 26: T T F T F F
Regression
11: F T F T T T 27: T T F T T T
System
12: F T T F F F 28: T T T F F F
Test Plans 13: F T T F T T 29: T T T F T T
14: F T T T F F 30: T T T T F F
LMS
15: F T T T T T 31: T T T T T T
QES

MC/DC Coverage Test Cases

Every condition in the decision independently affects the
decision’s outcome
# a b c d e Cond Cases
1: T X T X T T 21, 23, 29, 31
2: F T T X T T 13, 15
3: T X F T T T 19, 27
4: T X T X F F 20, 22, 28, 30
5: T X F F X F 16, 17, 24, 25
6: F F X F X F 0, 1, 4, 5
CS20202: Software Engineering 50
 White Box Testing:
Modified Condition / Decision (MC/DC)
Module 44 • MC/DC is
◦ basic condition coverage (C)
◦ branch coverage (DC)
◦ plus one additional condition (M): every condition must independently affect the decision’s output
Fundamentals • It is subsumed by compound conditions and subsumes all other criteria
Verification & discussed so far
Validation ◦ stronger than statement and branch coverage
Black Box Testing
White Box Testing • A good balance of thoroughness and test size (and therefore widely used)
Development • MC/DC code coverage criterion is commonly used software testing. For
Testing example, DO-178C software development guidance in the aerospace industry
Regression
requires MC/DC for the most critical software level (DAL A).
System

Test Plans
• MC/DC vs. MCC
LMS ◦ MCC testing is characterized as number of tests = 2C . In coffee example we have 3 conditions
QES (kettle, cup and coffee) therefore tests = 23 = 8
◦ MC/DC requires significantly fewer tests (C + 1). In coffee example we have 3 conditions,
therefore 3 + 1 = 4
◦ In a real-world setting, most aerospace projects would include some decisions with 16 conditions
or more. So the reduction would be from 21 6 = 65,536 to 16 + 1 = 17. That is, 65,519/65,536
= 99.97%

Source: What is MC/DC?

CS20202: Software Engineering 51

 White Box Testing:
Different Types of Code Coverage
Module 44 Coverage Criteria SC DC MC/DC MCC
Every statement in the program has been invoked at least once X
Every point of entry and exit in the program has been invoked at X X X
least once
Fundamentals Every control statement (that is, branch-point) in the program has X X X
taken all possible outcomes (that is, branches) at least once
Verification & Every non-constant Boolean expression in the program has evalu- X X X
Validation ated to both a True and False result
Black Box Testing Every non-constant condition in a Boolean expression in the pro- X X
White Box Testing gram has evaluated to both a True and False result
Every non-constant condition in a Boolean expression in the pro- X X
Development gram has been shown to independently affect that expression’s out-
Testing come
Regression Every combination of condition outcomes within a decision has been X
System invoked at least once
Test Plans
• SC: Statement Coverage • DC: Decision Coverage
LMS
• MC/DC: Modified Condition / Decision Coverage • MCC: Multiple Condition Coverage
QES

Source: What is MC/DC?

CS20202: Software Engineering 52

 White Box Testing:
Path Coverage
Module 44 • Design test cases such that
◦ All linearly independent paths in the program are executed at least once
• Defined in terms of
Fundamentals ◦ Control flow graph (CFG) of a program
Verification &
Validation
• To understand the path coverage-based testing
Black Box Testing ◦ we need to learn how to draw control flow graph of a program
White Box Testing
• A control flow graph (CFG) describes
Development
Testing
◦ The sequence in which different instructions of a program get executed
Regression ◦ The way control flows through the program
System
• Number all statements of a program
Test Plans
LMS
• Numbered statements
QES ◦ Represent nodes of control flow graph
• An edge from one node to another node exists
◦ If execution of the statement representing the first node – Can result in
transfer of control to the other node

CS20202: Software Engineering 53

 White Box Testing:
Path Coverage: CFG
Module 44
int f1(int x,int y) {
1 while (x != y){
2 if (x>y) then
3 x=x-y;
Fundamentals 4 else y=y-x;
Verification & 5 }
Validation 6 return x; }
Black Box Testing
White Box Testing

Development
• A path through a program:
Testing
◦ A node and edge sequence from the starting node to a terminal node of the control flow graph
Regression
System
◦ There may be several terminal nodes for program

Test Plans • Any path through the program that introduces at least one new edge is a
LMS Linearly Independent Path (LIP).
QES
• A set of paths are linearly independent if none of them can be created by
combining the others in some way.
◦ It is straight forward to identify linearly independent paths of simple programs; but not so for
complicated programs

• LIP in the above example:

◦ 1,6
◦ 1,2,3,5,1,6
◦ 1,2,4,5,1,6

CS20202: Software Engineering 54

 White Box Testing:
Path Coverage: LIP
Module 44
public static boolean isPrime(int n) {
A int i = 2;
B while (i < n) {
C if (n % i == 0) {
Fundamentals D return false
}
Verification & E i++;
Validation }
Black Box Testing F return true;
White Box Testing }

Development
Testing CFG LIP
Regression
System

Test Plans
LMS
QES

Source: The ’Linearly Independent Paths’ Metric for Java

CS20202: Software Engineering 55

 White Box Testing:
Path Coverage: McCabe’s Cyclomatic Metric
Module 44 • An upper bound for the number of linearly independent paths of a program –
a practical way of determining the maximum number of LIP
• Given a control flow graph G , cyclomatic complexity V (G ):
Fundamentals ◦ V (G ) = E − N + 2
Verification & ◦ N is the number of nodes in G
Validation ◦ E is the number of edges in G
Black Box Testing
White Box Testing
• Alternately, inspect control flow graph to determine number of bounded
areas (any region enclosed by a nodes and edge sequence) in the graph V (G )
Development
Testing
= Total number of bounded areas + 1
Regression • Example: Cyclomatic complexity = 7 − 6 + 2 = 3 = 2 + 1
System

Test Plans
LMS
QES

CS20202: Software Engineering 56

 White Box Testing:
Path Coverage: McCabe’s Cyclomatic Metric
Module 44 • McCabe’s metric provides a quantitative measure of testing difficulty and the
ultimate reliability
• Intuitively, number of bounded areas increases with the number of decision
Fundamentals nodes and loops
Verification & • The first method of computing V (G ) is amenable to automation:
Validation
Black Box Testing
◦ You can write a program which determines the number of nodes and
White Box Testing
edges of a graph
Development
◦ Applies the formula to find V (G )
Testing • The cyclomatic complexity of a program provides:
Regression
System
◦ A lower bound on the number of test cases to be designed
◦ To guarantee coverage of all linearly independent paths
Test Plans
LMS • A measure of the number of independent paths in a program
QES
• Provides a lower bound
◦ for the number of test cases for path coverage
• Knowing the number of test cases required
◦ Does not make it any easier to derive the test cases
◦ Only gives an indication of the minimum number of test cases required

CS20202: Software Engineering 57

 White Box Testing:
Path Coverage: Practical Path Testing
Module 44 • Tester proposes initial set of test data using her experience & judgement
• A dynamic program analyzer is used to measures which parts of the program
have been tested
Fundamentals • Result used to determine when to stop testing
Verification & • Derivation of Test Cases
Validation
◦ Draw control flow graph.
Black Box Testing
◦ Determine V(G).
White Box Testing
◦ Determine the set of linearly independent paths.
Development ◦ Prepare test cases to force execution along each path
Testing
Regression
• Example: Number of independent paths: 3
System int f1(int x,int y) {
1 while (x != y){
Test Plans
2 if (x>y) then
LMS
3 x=x-y;
QES
4 else y=y-x;
5 }
6 return x; }

◦ 1,6: test case (x=1, y=1)

◦ 1,2,3,5,1,6: test case (x=1, y=2)
◦ 1,2,4,5,1,6: test case (x=2, y=1)

CS20202: Software Engineering 58

 White Box Testing:
An Application of Cyclomatic Complexity
Module 44 • Relationship exists between:
◦ McCabe’s metric
◦ The number of errors existing in the code,
Fundamentals ◦ The time required to find and correct the errors.
Verification & • Cyclomatic complexity of a program:
Validation ◦ Also indicates the psychological complexity of a program
Black Box Testing
White Box Testing
◦ Difficulty level of understanding the program
Development
• From maintenance perspective,
Testing ◦ Limit cyclomatic complexity of modules To some reasonable value.
Regression
System • Good software development organizations:
Test Plans ◦ Restrict cyclomatic complexity of functions to a maximum of ten or so
LMS
QES

CS20202: Software Engineering 59

 White-Box Testing:
Coverage Testing Summary
Module 44

Fundamentals

Verification &
Validation
Black Box Testing
White Box Testing

Development
Testing
Regression
System

Test Plans
LMS
QES

CS20202: Software Engineering 60

 White Box Testing:
Mutation Testing
Module 44 • The software is first tested:
◦ Using an initial testing method based on white-box strategies we already discussed
• After the initial testing is complete
Fundamentals ◦ mutation testing is taken up

Verification & • The idea behind mutation testing

Validation ◦ Make a few arbitrary small changes to a program at a time
Black Box Testing
White Box Testing
• Good software development organizations:
◦ Restrict cyclomatic complexity of functions to a maximum of ten or so
Development
• Insert faults into a program:
Testing
◦ Check whether the tests pick them up
Regression
◦ Either validate or invalidate the tests
System
◦ Example:
Test Plans 1: cin >> x;
LMS 2: if (0 == x)
QES 3: x = x + 1;
4: y = 5;

Note that, {(x = 0)} covers lines {1, 2, 3, 4} while {(x = 1)} covers only lines {1, 2, 4}.
So with {(x = 0)}, we get 100% statement coverage. But then, did we check for the jump from
line 2 to 4 for the false condition? This condition did not get tested. So we need {(x = 0), (x
= 1)} for 100% branch coverage and it obviously leads to 100% statement coverage.
How do we get 100% branch coverage for:
1: if (true)
2: x = x + 1;
3: y = 5;
CS20202: Software Engineering 61
 White Box Testing:
Mutation Testing
• Insert faults into a program:
Module 44
◦ Check whether the tests pick them up
◦ Either validate or invalidate the tests

• Each time the program is changed

Fundamentals ◦ it is called a mutated program
◦ the change is called a mutant
Verification &
Validation • A mutated program:
Black Box Testing ◦ Tested against the full test suite of the program
White Box Testing
• If there exists at least one test case in the test suite for which:
Development ◦ A mutant gives an incorrect result, then the mutant is said to be dead
Testing
Regression • If a mutant remains alive:
System ◦ even after all test cases have been exhausted, the test suite is enhanced to kill the mutant
Test Plans • The process of generation and killing of mutants
LMS
QES
◦ can be automated by pre-defining a set of primitive changes that can be applied to the program

• The primitive changes can be

◦ Deleting a statement
◦ Altering an arithmetic operator
◦ Changing the value of a constant
◦ Changing a data type, etc.

CS20202: Software Engineering 62

 White Box Testing:
Mutation Testing: Error Seeding
• How Many Errors are Still Remaining?
Module 44
• Seed the code with some known errors:
◦ Artificial errors are introduced into the program
◦ Check how many of the seeded errors are detected during testing
Fundamentals • Let
Verification & ◦ N be the total number of errors in the system and n of these errors be found by testing
Validation ◦ S be the total number of seeded errors and s of the seeded errors be found during testing
Black Box Testing ◦ N
n = s
S
White Box Testing ◦ N = S ∗ ns

Development
◦ Remaining defects:
S −s
Testing N−n =n∗
s
Regression
System
• Example:
Test Plans ◦ 100 errors were introduced
LMS ◦ 90 of these errors were found during testing
◦ 50 other errors were also found
QES
◦ Remaining errors:
100 − 90
50 ∗ =6
90

• The kind of seeded errors should match closely with existing errors
◦ However, it is difficult to predict the types of errors that exist

• Categories of remaining errors

◦ even after all test cases have been exhausted, the test suite is enhanced to kill the mutant

CS20202: Software Engineering 63

 White Box Testing:
Data Flow Testing
Data Flow Testing is a type of structural testing. It is a method that is used to
Module 44
find the test paths of a program according to the locations of definitions and uses
of variables in the program
• It is concerned with:
Fundamentals ◦ Statements where variables receive values
Verification &
◦ Statements where these values are used or referenced
Validation • To illustrate the approach of data flow testing, assume that each statement in
Black Box Testing
the program assigned a unique statement number. For a statement number S:
White Box Testing
◦ DEF(S) = {X | statement S contains a definition of X}
Development ◦ USE(S)= {X | statement S contains a use of X}
Testing ◦ Example: 1: a = b; DEF(1) = {a}, USE(1) = {b}
Regression ◦ Example: 2: a = a + b; DEF(2) = {a}, USE(2) = {a,b}
System
• If a statement is a loop or if condition then its DEF set is empty and USE set
Test Plans is based on the condition of statement s.
LMS
QES • Data Flow Testing uses the control flow graph to find the situations that can
interrupt the flow of the program.
• Reference or define anomalies in the flow of the data are detected at the time
of associations between values and variables. These anomalies are:
◦ A variable is defined but not used or referenced
◦ A variable is used but never defined
◦ A variable is defined twice before it is used

CS20202: Software Engineering 64

 White Box Testing:
Data Flow Testing
Module 44 • Advantages of Data Flow Testing
◦ To find a variable that is used but never defined
◦ To find a variable that is defined but never used
Fundamentals ◦ To find a variable that is defined multiple times before it is use
Verification &
◦ Deallocating a variable before it is used
Validation • Disadvantages of Data Flow Testing
Black Box Testing
White Box Testing
◦ Time consuming and costly process
◦ Requires knowledge of programming languages
Development
Testing
Regression
System

Test Plans
LMS
QES

CS20202: Software Engineering 65

 White Box Testing:
Data Flow Testing
• Example:
Module 44
1. read x, y;
2. if (x > y)
3. a = x + 1
else
Fundamentals 4. a = y - 1
5. print a;
Verification &
Validation
• CFG
Black Box Testing
White Box Testing

Development
Testing
Regression
System

Test Plans
LMS
QES

• Define/use of variables
Variable Defined at node Used at node
x 1 2, 3
y 1 2, 4
a 3, 4 5
Source: Data Flow Testing
CS20202: Software Engineering 66
 Data Object Categories

Module 44 • (d) Defined, Created, Initialized. An object (like variable) is defined when it:
◦ appears in a data declaration
◦ is assigned a new value
Fundamentals ◦ is a file that has been opened
Verification &
◦ is dynamically allocated
Validation ◦ ...
Black Box Testing • (k) Killed, Undefined, Released
White Box Testing

Development
• (u) Used:
Testing ◦ (c) Used in a calculation
Regression ◦ (p) Used in a predicate
System
◦ An object is used when it is part of a computation or a predicate
Test Plans ▷ A variable is used for a computation (c) when it appears on the RHS (sometimes even the
LMS LHS in case of array indices) of an assignment statement
QES ▷ A variable is used in a predicate (p) when it appears directly in that predicate

Source: Topics in Software Dynamic White-box Testing: Part 2: Data-flow Testing

CS20202: Software Engineering 67

 White Box Testing:
Data Flow-Based Testing: Definition and Use
Module 44

Fundamentals

Verification &
Validation
Black Box Testing
White Box Testing

Development
Testing
Regression
System

Test Plans
LMS
QES

• To find a variable that is used but never defined

• To find a variable that is defined but never used
• To find a variable that is defined multiple times before it is use
• Deallocating a variable before it is used
CS20202: Software Engineering 68
 Processes for Development

Module 44
Development involves a number of processes:
• Design & Coding (discussed in Modules 2 & 4)
Fundamentals • Debugging
Verification & • Issue / Bug Tracking
Validation
Black Box Testing
White Box Testing
• Testing
Development • Documentation
Testing
Regression • Release (discussed as a part of Maintenance)
System

Test Plans • Version Control (discussed as a part of Maintenance)

LMS
QES

CS20202: Software Engineering 69

 Testing Processes

Module 44
• The aim of testing is to identify all defects in a software product
• However, in practice even after thorough testing, one cannot guarantee that
the software is error-free
Fundamentals • The input data domain of most software products is very large
Verification &
◦ It is not practical to test the software exhaustively with each input
Validation • Testing however exposes many errors
Black Box Testing
White Box Testing ◦ Testing provides a practical way of reducing defects in a system
Development
◦ Increases the users’ confidence in the system
Testing • Testing
Regression
System
◦ Is a continual process
Test Plans
◦ Needs significant automation (especially to repeats tests already done
LMS
when new stuff is added). Usually achieved through
QES ▷ Regression Testing
◦ Has to happen at all phases and all levels of abstraction – both for
development and for maintenance
• Software products are tested at three levels
◦ Unit testing
◦ Integration testing
◦ System testing
CS20202: Software Engineering 70
 Regression Testing

Module 44
• Regression1 testing is re-running functional and non-functional tests to ensure
that previously developed and tested software still performs after a change
◦ If not, that would be called a regression
◦ The software Regresses
Fundamentals
• Regression testing
Verification &
Validation ◦ Is usually done through auotmated processes after each change to the
Black Box Testing system after each bug fix
White Box Testing ◦ Ensures that no new bug has been introduced due to the change or the bug
Development fix – the new system’s performance is at least as good as the old system
Testing ◦ Is always used during incremental system development
Regression
System • Regression test is:
Test Plans ◦ Required before every code check-in (no regression, that is)
LMS ◦ Used at every level for every kind of test
QES ◦ Most powerful tool for quality control for software development and
maintenance

1
a return to a former or less developed state
CS20202: Software Engineering 71
 Regression Testing:
Positive & Negative Test Cases
Module 44
Execution Outcome
Type of Test Successful Unsuccessful
Fundamentals
Positive Case PASS FAIL
Verification &
Validation (Success)
Black Box Testing
White Box Testing
Negative Case FAIL PASS
Development (Failure)
Testing
Regression
System Regression outcome is typically shown by:
Test Plans
LMS # of PASS cases
QES ∗ 100%
# of PASS + FAIL (total) cases

CS20202: Software Engineering 72

 How to do Regression Testing?
Approach
Module 44
• We need to first debug the code to identify the bugs
• Once the bugs are identified and fixed, then the regression testing is done by
selecting relevant test cases from the test suite that covers both modified and
affected parts of the code
Fundamentals

Verification &
• Regression Testing can be carried out using the following strategies:
Validation ◦ Retest All
Black Box Testing ▷ Needed for all version major / minor releases
White Box Testing
◦ Regression Test Selection
Development ▷ Test cases which have frequent defects
Testing ▷ Functionalities which are more visible to the users
Regression ▷ Test cases which verify core features of the product
System ▷ Test cases of Functionalities which has undergone more and recent changes
▷ All Integration Test Cases & all Complex Test Cases
Test Plans ▷ Boundary value test cases
LMS ▷ Samples of Successful test cases & Failure test cases
QES
◦ Prioritization of Test Cases
▷ Depending on business impact, critical & frequently used functionalities
▷ General prioritization: Beneficial on subsequent versions
▷ Version-specific prioritization: Beneficial for a particular version of the software
◦ Hybrid
▷ This technique is a hybrid of regression test selection and test case prioritization

Source: What is Regression Testing? Definition, Test Cases (Example)

CS20202: Software Engineering 73

 How to do Regression Testing?
Automation
Regression Testing relies on intense automation to reduce menial manual testing
Module 44 efforts and costs
• Scripting for Regression Testing
◦ Scripting is the biggest handle for Regression Automation
Fundamentals ◦ Unix: Shell, Bash, sed, awk etc.
◦ Windows: VBScript
Verification & ◦ Perl
Validation ◦ Python
Black Box Testing
White Box Testing
• Regression Testing Tools
◦ Selenium: Open Source portable framework for automating web applications (browser-based
Development regression). It provides a playback tool for authoring functional tests without the need to learn a
Testing test scripting language (Selenium IDE)
Regression ◦ Micro Focus Unified Functional Testing (UFT One) (formerly known as QuickTest Professional
System (QTP)): An tool to automate functional and regression test cases. UFT One Intelligent test
automation with embedded AI-based capabilities that accelerates testing across desktop, web,
Test Plans mobile, mainframe, composite and packaged enterprise-grade apps.
LMS ◦ IBM Rational Functional Tester (RFT): An automated functional testing and regression testing
QES tool. This software provides automated testing capabilities for functional, regression, GUI and
data-driven testing. It supports a range of applications, such as web-based, .Net, Java, Siebel, SAP,
terminal emulator-based applications, PowerBuilder, Ajax, Adobe Flex, Dojo Toolkit, GEF, Adobe
PDF documents, zSeries, iSeries and pSeries.

Source: What is Regression Testing? Definition, Test Cases (Example)

CS20202: Software Engineering 74

 Retesting and Regression Testing

Module 44
• Retesting means testing the functionality or bug again to ensure the code is
fixed
◦ If it is not fixed, defect needs to be re-opened
◦ If fixed, defect is closed
Fundamentals
• Regression testing means testing your software application when it undergoes
Verification & a code change to ensure that the new code has not affected other parts of the
Validation
Black Box Testing
software.
White Box Testing
Regression Testing Retesting
Development
Confirms that a recent program or code change has Confirms that the test cases that failed in the final
Testing
not adversely affected existing features execution are passing after the defects are fixed
Regression
Ensures that new code changes do not have any side Is done on the basis of the Defect fixes
System
effects to existing functionalities
Test Plans Defect verification is not in scope Defect verification is in scope
LMS
May be carried out parallel with Re-testing Must be carried out before regression testing
QES
May be automated Cannot automate the test cases for Retesting
Manual Testing is expensive
Generic testing Planned testing
Done for passed test cases Done only for failed test cases
Checks for unexpected side-effects Makes sure that the original fault has been corrected
Done for any mandatory modification or changes in an Executes a defect with the same data and the same
existing project environment with different inputs with a new build
Test cases for can be obtained from the functional Test cases for retesting cannot be obtained before
specification, user tutorials and manuals, and defect start testing
reports in regards to corrected problems

Source:
CS20202:Difference
Software Between Retesting and Regression Testing
Engineering 75
 Regression Test Suite

Module 44
• To repeat old tests and runs
• Test Case with Golden
Fundamentals ◦ Unit Tests
Verification &
Validation
◦ API / Application Tests
Black Box Testing ◦ Directed Tests, Corner Cases, Customer Tests
White Box Testing

Development
◦ Random & Huge Tests
Testing ◦ Performance Tests
Regression
System ▷ Time
Test Plans
LMS
▷ Resources: Low Memory Tests
QES
◦ Coverage Tests
◦ ...
• Folder Structure
◦ Uniformity names of test case files: Critical for
automation scripts
• Run script
CS20202: Software Engineering 76
 Unit Testing

Module 44
• During unit testing, modules are tested in isolation
◦ If all modules were to be tested together, it may not be easy to determine
which module has the error
Fundamentals

Verification &
• Unit testing reduces debugging effort several folds
Validation ◦ Programmers carry out unit testing immediately after they complete the
Black Box Testing coding of a module
White Box Testing

Development
• Unit testing drives development in Test-Driven Development (TDD)
Testing
Regression
System

Test Plans
LMS
QES

CS20202: Software Engineering 77

 TDD

Module 44

Fundamentals

Verification &
Validation
Black Box Testing
White Box Testing

Development
Testing
Regression
System

Test Plans
LMS
QES

CS20202: Software Engineering 78

 TDD

Module 44

Fundamentals

Verification &
Validation
Black Box Testing
White Box Testing

Development
Testing
Regression
System

Test Plans
LMS
QES

CS20202: Software Engineering 79

 Unit Testing

Module 44
• For a class attach a (test) unit (typically JUnit in Java or CPPUnit in C++))
that tests the class
Fundamentals
• D Language provides specific support for unit testing

Verification &
• Unit testing is a main feature of D
Validation • Unit testing in D
Black Box Testing
White Box Testing ◦ Unit tests can be added to a class - they are run upon program start-up
Development
◦ Aids in verifying that class implementations weren’t inadvertently broken
Testing
◦ Unit tests is a part of the code for a class
Regression ◦ Creating tests is a part of the development process
System

Test Plans
LMS
QES

CS20202: Software Engineering 80

 Integration Testing

Module 44
• After different modules of a system have been coded and unit tested:
◦ Modules are integrated in steps according to an integration plan
Fundamentals
◦ Partially integrated system is tested at each integration step
Verification &
• System Testing
Validation ◦ Validate a fully developed system against its requirements
Black Box Testing
White Box Testing
• Develop the integration plan by examining the structure chart:
Development
◦ Big bang approach
Testing ◦ Top-down approach
Regression ◦ Bottom-up approach
System
◦ Mixed approach
Test Plans
LMS
QES

CS20202: Software Engineering 81

 Integration Testing

Module 44
• Big Bang is the simplest integration testing approach
◦ All the modules are simply put together and tested
◦ This technique is used only for very small systems
◦ If an error is found:
Fundamentals ▷ It is very difficult to localize the error
Verification & ▷ The error may potentially belong to any of the modules being
Validation integrated
Black Box Testing
White Box Testing
◦ Debugging errors found during big bang integration testing are very
expensive to fix
Development
Testing
• Bottom-up Integration Testing Integrate and test the bottom level modules
Regression first. A disadvantage of bottom-up testing:
System
◦ when the system is made up of a large number of small subsystems
Test Plans ◦ This extreme case corresponds to the big bang approach.
LMS
QES
• Top-down Integration Testing Top-down integration testing starts with the
main routine
◦ and one or two subordinate routines in the system
◦ After the top-level ’skeleton’ has been tested
▷ immediate subordinate modules of the ’skeleton’ are combined with it
and tested
• Mixed Integration Testing Mixed (or sandwiched) integration testing
◦ Uses both top-down and bottom-up testing approaches
◦ Most common approach
CS20202: Software Engineering 82
 Integration Testing

Module 44
• In top-down approach
◦ testing waits till all top-level modules are coded and unit tested
• In bottom-up approach
Fundamentals ◦ testing can start only after bottom level modules are ready
Verification &
Validation
Black Box Testing
White Box Testing

Development
Testing
Regression
System

Test Plans
LMS
QES

CS20202: Software Engineering 83

 System Testing
System tests are designed to validate a fully developed system
Module 44
• To assure that it meets its requirements
• Functional requirements are validated through functional tests
Fundamentals • Non-functional requirements validated through performance tests
Testing Release Features
Verification &
Validation • All functionality has been implemented
Black Box Testing
Alpha Alpha
• Reasonable testing has been done – Regression passes with known exceptions
White Box Testing
• Testing is carried out by the test team within the company
Development • No preview expected from real users yet
Testing • All functionality has been thoroughly tested
Regression
System
• Extensive testing has been done – Regression passes with some (documented) ex-
ceptions
Test Plans Beta Beta • Releases made to friendly customers (or non-developer internal groups). Or beta
LMS programs launched
QES • Testing performed by a select group of friendly customers
• Must include real users of a system
• FCS: First Customer Shipment
• Company has gained fair confidence on the quality – Regression is clean with rare
(documented) exceptions
Acceptance FCS • Customer will use this release now till (Minor) bug fix releases are done or new
(Major) version is produced
• Testing performed by the customer to determine whether the system should be ac-
cepted or rejected

CS20202: Software Engineering 84

 Test Plans: LMS

Module 44

Fundamentals

Verification &
Validation
Black Box Testing
White Box Testing

Development
Testing
Regression
System

Test Plans
LMS
QES

CS20202: Software Engineering 85

 Test Plans: QES: Code
00: unsigned int Solve(double a, double b, double c, double& r1, double& r2)
Module 44 01: {
02: unsigned int retVal = 0;
03: if (0 == a) {
04: if (0 == b) {
05: if (0 == c) { // Infinite solutions
Fundamentals 06: retVal = 5;
07: } else { // Inconsistent equation
Verification & 08: retVal = 0;
Validation 09: }
Black Box Testing 10: } else { // Linear equation
White Box Testing 11: retVal = 1;
12: r1 = -c/b;
Development 13: }
Testing 14: } else {
Regression 15: double disc = b*b - 4*a*c;
System 16: if (0 == disc) { // Repeated roots
17: retVal = 2;
Test Plans
18: r1 = r2 = -b/(2*a);
LMS
19: } else {
QES 20: if (disc > 0) { // Real distinct roots
21: retVal = 3;
22: r1 = (-b + sqrt(disc))/(2*a);
23: r2 = (-b - sqrt(disc))/(2*a);
24: } else { // Complex conjugate roots
25: retVal = 4;
26: r1 = (-b)/(2*a); r2 = (sqrt(-disc))/(2*a);
27: }
28: }
29: }
30:
31:
CS20202:return retVal;
Software Engineering 86
 Test Plans: QES: CFG
CFG of QES Reduced CFG of QES
Module 44

Fundamentals

Verification &
Validation
Black Box Testing
White Box Testing

Development
Testing
Regression
System

Test Plans
LMS
QES

CS20202: Software Engineering 87

 Test Plans: QES

Module 44

Fundamentals

Verification &
Validation
Black Box Testing
White Box Testing

Development
Testing
Regression
System

Test Plans
LMS
QES

CS20202: Software Engineering 88

 Test Plans: QES

Module 44

Fundamentals

Verification &
Validation
Black Box Testing
White Box Testing

Development
Testing
Regression
System

Test Plans
LMS
QES

CS20202: Software Engineering 89