Lecture 9

Software Testing
BITP 2213 Software Engineering
 Content

Verification and Validation

V & V Planning

Inspection

Software Testing

Designing Test Cases

BITP 2213 | Sem 1 20222023 2
 Verification and Validation
• V&V – a system engineering discipline employing a
rigorous methodology for evaluating and assessing
the correctness and quality of software throughout
the software life cycle.
• V&V and QA are not the same, but compliment each
other.
• V&V usually focuses on ensuring the requirements
are being met, the overall project is focused on the
correct objectives, and risk is being managed

BITP 2213 | Sem 1 20222023 3

V and V
Verification Validation

“Are we building "Are we building

the the
product right? ” right product?”.

The software The software

should conform should do what
to its the user really
specification. requires.

BITP 2213 | Sem 1 20222023 4

 V n V Process
• Applies to all stages in the selected software
development model

• Why V n V?
– To discover defects in the deliverables / software artifacts
– To assess system usability and usefulness before
operational
– To establish confidence the software fit to be use
– To assess degree of stakeholder acceptance

BITP 2213 | Sem 1 20222023 5

 V n V Planning
• Planning should start early in the development process
• Balance between static and dynamic V & V approach
• Draw up standard and procedure for V n V
• Establish inspection checklist
• Define software test plan
– Establish standard for testing process
– Allocate resources and estimate time

BITP 2213 | Sem 1 20222023 6

 How to perform V n V?
• Software Inspection (Static) - Concerned with analysis of
the static system representation to discover problems. May be
supplement by tool-based document and code analysis
– Analyze and check software artifacts
– Documentations, diagrams, source codes
– Executed without system existence

• Software Testing (Dynamic) - Concerned with exercising and

observing product behaviour. The system is executed with test
data and its operational behaviour is observed
– Running the software with the test data
– Examine the software behavior

BITP 2213 | Sem 1 20222023 7

 Software Inspection
• Purpose : To find errors, omissions and anomalies
• May be applied to any representation of the system :

Documents Models Source codes Diagrams

• Does not require system execution

• An effective technique to discover program errors

BITP 2213 | Sem 1 20222023 8

 Inspection Effectiveness
• Proven to discover more defects than testing
– DOES NOT GUARANTEE ERROR FREE
• Many different defects may be discovered in a single
inspection. Errors in testing might masks another.
– Several executions are required
• Reviewer reusing domain and programming knowledge and
experience
– likely to have seen the types of error that commonly arise
• Inspections can check conformance with a specification ->
verification, but not conformance with the customer’s real
requirements.
• Inspection and testing : Complements each other and not
opposing verification techniques.
BITP 2213 | Sem 1 20222023 9
 Program Inspection
• Formalised approach to document reviews
• Intended explicitly for defect detection (not
correction)
• Common defects :-
– Logical errors
– Anomalies in the code that might indicate an erroneous
condition (e.g. an uninitialized variable)
– Non-compliance with standards.

BITP 2213 | Sem 1 20222023 10

 The Inspection Process

Planning

Overview Follow-up
Indi vidual
Rework
pr eparation
Inspection
meeting

BITP 2213 | Sem 1 20222023 11

 How To Perform Inspection?
• System overview presented to inspection team.
• Code and associated documents are
distributed to inspection team in advance.
• Inspection takes place and discovered errors
are noted.
• Modifications are made to repair discovered
errors.
• Re-inspection may or may not be required

BITP 2213 | Sem 1 20222023 12

 Inspection Roles
• Author or owner • Reader
– The programmer or designer – Presents the code or document
responsible for producing the at an inspection meeting.
program or document. • Scribe
– Responsible for fixing defects – Records the results of the
discovered during the inspection meeting.
inspection process.
• Chairman or moderator
• Inspector
– Manages the process and
– Finds errors, omissions and facilitates the inspection.
inconsistencies in programs Reports process results to the
and documents. Chief moderator.
– May also identify broader
issues that are outside the • Chief moderator
scope of the inspection team. – Responsible for inspection process
improvements, checklist updating,
standards development etc.

BITP 2213 | Sem 1 20222023 13

 Inspection Checklist
• Checklist of common errors should be used to drive the
inspection.
• Error checklists are programming language dependent and
reflect the characteristic errors that are likely to arise in the
language.
• Examples:
– Initialisation
– Constant naming,
– Loop termination,
– array bounds, etc.

BITP 2213 | Sem 1 20222023 14

Fault class Inspection check
Data faults Are all program variables initialised before their values
are used?
Have all constants been named?
Should the lower bound of arrays be 0, 1, or something
else?
Should the upper bound of arrays be equal to the size of
the array or Size -1?
If character strings are used, is a delimiter explicitly
assigned?
Control faults For each conditional statement, is the condition correct?
Is each loop certain to terminate?
Are compound statements correctly bracketed?
In case statements, are all possible cases accounted for?
Input/output faults Are all input variables used?
Are all output variables assigned a value before they are
output?
Interface faults Do all function and procedure calls have the correct
number of parameters?
Do formal and actual parameter types match?
Are the parameters in the right order?
If components access shared memory, do they have the
same model of the shared memory structure?
Storage management If a linked structure is modified, have all links been
faults correctly reassigned?
If dynamic storage is used, has space been allocated
correctly?
Is space explicitly de-allocated after it is no longer
required?
Exception Have all possible error conditions been taken into
management faults account?
BITP 2213 | Sem 1 20222023 15
 Inspection Rate
• 500 statements/hour during overview.
• 125 source statement/hour during individual
preparation.
• 90-125 statements/hour can be inspected.
• Inspection is therefore an expensive process.
• Inspecting 500 lines costs about 40 man/hours effort
- about £2800 at UK rates.

BITP 2213 | Sem 1 20222023 16

 Software Test Plan

The testing Requirements

Tested items.
process. traceability.

Hardware and
Testing Test recording
software
schedule. procedures.
requirements.

Constraints

BITP 2213 | Sem 1 20222023 17

 Software Test Plan
• Testing Process - A description of the major phases of the system testing process. This may
be broken down into the testing of individual sub-systems, the testing of external system
interfaces, etc.
• Requirements traceability - Users are most interested in the system meeting its
requirements and testing should be planned so that all requirements are individually tested.
• Tested items - The products of the software process that are to be tested should be specified.
• Testing schedule - An overall testing schedule and resource allocation.
• Test recording procedures - It is not enough simply to run tests; the results of the tests must
be systematically recorded. It must be possible to audit the testing process to check that it
has been carried out correctly.
• Hardware and software requirements - This section should set out the software tools
required and estimated hardware utilization.
• Constraints - Constraints affecting the testing process such as staff shortages should be
anticipated in this section.
• System tests - This section, which may be completely separate from the test plan, defines the
test cases that should be applied to the system. These tests are derived from the system
requirements specification.

BITP 2213 | Sem 1 20222023 18

 What is Static Analysis?
• Static = without program execution.
• Static analysis can be done manually, like inspections, or static
analysis can be automated by using software tools -> ASA.
• Because static analysis does not require execution of the
code, analysis for errors and vulnerabilities can be done
throughout the software development process, and analysis
can be conducted across all code paths.
• With dynamic analysis practices like testing, the line coverage
is determined by the lines of code executed by the suite of
test cases.

BITP 2213 | Sem 1 20222023 19

 Why Static Analysis?
• Static analysis : Analysing computer software
without executing the program.
• Particularly valuable when a language such as C is
used which has weak typing and hence many errors
are undetected by the compiler,
• Less cost-effective for languages like Java that have
strong type checking and can therefore detect many
errors during compilation.

BITP 2213 | Sem 1 20222023 20

 Stages of Static Analysis

Control flow analysis Data use analysis Interface analysis

• Checks for loops with multiple • Detects uninitialized variables, • Checks the consistency of
exit or entry points, finds variables written twice without routine and procedure
unreachable code, etc. an intervening assignment, declarations and their use
variables which are declared
but never used, etc.

Information flow analysis Path analysis

• Identifies the dependencies of • Identifies paths through the
output variables. program and sets out the
• Does not detect anomalies statements executed in that
itself but highlights information path.
for code inspection or review

BITP 2213 | Sem 1 20222023 21

 Verification for Formal Methods
• Formal methods can be used when a mathematical
specification of the system is produced.
• They are the ultimate static verification technique.
• They involve detailed mathematical analysis of the
specification and may develop formal arguments that
a program conforms to its mathematical specification

BITP 2213 | Sem 1 20222023 22

 Software Testing
“The process of analyzing a software item to detect the
differences between existing and required conditions
(i.e., bugs) and to evaluate the features of the software
item”

BITP 2213 | Sem 1 20222023 23

 Software Testing
Views of the Test Objects

• Closed box or black box: functionality of the

test objects
• Clear box or white box: structure of the test
objects
• Static white-box testing is the process of
carefully and methodically reviewing the
software design, architecture, or code for bugs
without executing it.

BITP 2213 | Sem 1 20222023 24

 Development Testing
• Development testing includes all activities that
are carried out by the team developing the
system.
• Use programmer/ tester pairs.
• There are six basic types of testing: unit,
integration, function/system, acceptance,
regression, and beta.
• White-box testing is used for three of these six
types.
BITP 2213 | Sem 1 20222023 25
 Unit Testing
The Best Team Size for Inspections

• The preparation rate, not the team size,

determines inspection effectiveness.
• The team’s effectiveness and efficiency
depend on their familiarity with their product.
• Testing of individual hardware or software
units or groups of related units (IEEE, 1990).
• A unit is a software component that cannot be
subdivided into other components (IEEE,
1990).
BITP 2213 | Sem 1 20222023 26
 Unit Testing
• Unit testing is important for ensuring the code
is solid before it is integrated with other code.
• Once the code is integrated into the code
base, the cause of an observed failure is more
difficult to find.
• Approximately 65% of all bugs can be caught
in unit testing (Beizer, 1990).

BITP 2213 | Sem 1 20222023 27

 Unit Testing
What is Unit Testing?
• To test each individual unit of code is to ensure that it performs its
intended functionality.

Who test the code?

• Software Engineer who has written code

Why unit test the code?

• Using some techniques which ensure that all logical paths of the code
are tested and
• Maximum number of errors can be detected by executing these tests

How to unit test the code ?

• Prepare the Unit Test Plan (UTP) at the end of the design phase

BITP 2213 | Sem 1 20222023 28

 Unit Testing
Code Review

• Code walkthrough
• Code inspection

BITP 2213 | Sem 1 20222023 29

 Integration Testing
• Integration testing, which is testing in which
software components, hardware components,
or both are combined and tested to evaluate
the interaction between them (IEEE, 1990).

BITP 2213 | Sem 1 20222023 30

 Regression testing
• Regression testing, which is selective retesting
of a system or component to verify that
modifications have not caused unintended
effects and that the system or component still
complies with its specified requirements (IEEE,
1990).
• White-box unit and integration test cases can
be saved and rerun as part of regression
testing
BITP 2213 | Sem 1 20222023 31
 Formal code reviews
• A formal code review is the process under
which static white-box testing is performed.
– Can be a simple one-on-one meeting or a detailed
rigorous code inspection.
– May be organized by the programming or the
testing team.

BITP 2213 | Sem 1 20222023 32

 Essential elements of
a formal code review
• Identify problems:
– Find problems with the software such as missing items, mistakes, etc.
• Follow rules:
– Amount of code to be reviewed, how much time will be spent, etc.
• Prepare:
– Each participant should prepare in order to contribute to the review.
• Write a report:
– Summarize the results of the review, make report available to the
development team.

BITP 2213 | Sem 1 20222023 33

 Informal code inspections
• Peer reviews:
– An informal small group of programmers and/or testers
act as reviewers.
– Participants should follow the 4 essential elements
even through the review is informal.
• Walkthroughs:
– A more formal process in which the author of the code
formally presents the code to a small group of
programmers and/or testers.
– The author reads the code line by line explaining what
it does, reviewers listen and ask questions.
– Participants should follow the 4 essential elements.
BITP 2213 | Sem 1 20222023 34
 Formal code inspections
• Code presenter is not the author of the code.
• The other participants are the inspectors.
• There is a moderator to assure that the rules are
followed and the meeting runs smoothly.
• After the inspection a report is composed. The
programmer then makes changes and a re-
inspection occurs, if necessary.
• Formal code inspections are effective at finding
bugs in code and designs and are gaining in
popularity.

BITP 2213 | Sem 1 20222023 35

 Code review checklist:
Data reference errors
• Is an un-initialized variable referenced?
• Are array subscripts integer values and are they within the
array’s bounds?
• Are there off-by-one errors in indexing operations or
references to arrays?
• Is a variable used where a constant would work better?
• Is a variable assigned a value that’s of a different type than
the variable?
• Is memory allocated for referenced pointers?
• Are data structures that are referenced in different
functions defined identically?

BITP 2213 | Sem 1 20222023 36

 Code review checklist:
Data declaration errors
• Are the variables assigned the correct length, type,
storage class?
– E.g. should a variable be declared a string instead of an array of
characters?
• If a variable is initialized at its declaration, is it properly
initialized and consistent with its type?
• Are there any variable with similar names?
• Are there any variables declared that are never referenced
or just referenced once (should be a constant)?
• Are all variables explicitly declared within a specific
module?

BITP 2213 | Sem 1 20222023 37

Testing Organization Illustration

BITP 2213 | Sem 1 20222023 38

 Component Testing & System
Testing
• Component testing • System testing
– Testing of individual program – Testing of groups of
components; components integrated to
– Usually the responsibility of create a system or sub-
the component developer system;
(except sometimes for critical – The responsibility of an
systems); independent testing team;
– Tests are derived from the – Tests are based on a system
developer’s experience. specification.

BITP 2213 | Sem 1 20222023 39

 What is a Test Case?
• A Test Case consist of:
– A set of inputs,
– Execution preconditions, and
– Expected outcomes

• The purpose of Test Case

– Developed for a particular objective
• To exercise a particular program path or to
• Verify compliance with a specific requirement

BITP 2213 | Sem 1 20222023 40

 Software Testing Process

Test Test Test Test

cases data results reports

Design test Prepare test Run program Compare results

cases data with test data to test cases

• Test cases – specifications of the inputs to the test and the

expected output from the systems (test results).
• Test data – the input that have been devised to test a system.

BITP 2213 | Sem 1 20222023 41

 Types of Test Cases
Positive test case:
• Design in such a way that the program or module being tested
succeeds.
• A valid input is passed to get a valid result.

Negative test case:

• Design in such a way that the program or module being tested
gives appropriate error code on an invalid input.
• An invalid input or condition is created in negative test cases.
• Negative test cases test the robustness of the program

BITP 2213 | Sem 1 20222023 42

Identifying Test Cases

Boundary
Equivalence
Value
Partitioning
Analysis

Logic Random
Coverage Generation

BITP 2213 | Sem 1 20222023 43

 Boundary Value Analysis
• A boundary value is one which indicates the border (or the
limit) of a value
– highest payoff in terms of detecting bugs, as the most common errors occur at
the boundary level

• Boundary value test cases are those which test the function or
field using values that
– Lie on the boundary
– Just above the boundary
– Just Below the boundary

BITP 2213 | Sem 1 20222023 44

 Boundary Value Analysis
• For example if an input condition specifies that the range of values of the
input variable items must be from 0 to 100, the boundary values would be
-1, 0,1,99,100 and 101

Valid Range of Input Values

Lower Upper
Lower Limit Lower Upper Limit Upper
Limit -1 Limit + 1 Limit -1 Limit + 1

BITP 2213 | Sem 1 20222023 45

 Equivalence Partitioning
• Dividing all possible inputs into a set of classes,
– All inputs that fall into a given class are valid or all are invalid.
– Then selecting a few test cases from each class is sufficient.

• Example if an input condition : between 1 to 999

– Valid equivalence class : 1<=items<=999
– Invalid equivalence classes : items<1 and items>999

• Most of the equivalence partition test cases are positive test cases

• The values chosen for Equivalence partition should not be a

boundary value

BITP 2213 | Sem 1 20222023 46

 Logic Coverage
• Aim : To generate enough test cases so that an appropriately defined coverage
criterion is met

• Criterion:
– Every statement in the program must be executed at least once,
– Every branch in the program must be executed at least once, or
– Every path in the program must be executed at least once

• Example:
– The User Interface for searching the address book should be very friendly and should use the
following search criteria
– Can search the address book based on one or more of the following fields:
• E-mail Id of the person
• Name of the person
• Employee Number of the person
– Can accept one or more fields for search criteria
– Accepts partial values for E-mail and Name

BITP 2213 | Sem 1 20222023 47

 Testing Policies
• Only exhaustive testing (perform completely) can show a
program is free from defects.
• Testing policies define the approach to be used in selecting
system tests:
– All functions accessed through menus should be tested;
– Combinations of functions accessed through the same menu should
be tested;
– Where user input is required, all functions must be tested with correct
and incorrect input.

BITP 2213 | Sem 1 20222023 48

 System Testing
• System Testing will contain the following testing :
– Functional Testing.
– Performance Testing.

• Function Testing will test the implementation of

the business needs.

• Performance Testing will test the non-functional

requirements of the system like the speed, load
etc.

BITP 2213 | Sem 1 20222023 49

 System Testing
• Performance Testing involves
– Load Testing
• Testing with many users accessing the system at the same
time.
– Endurance Testing
• Testing for a long time for reliability.
– Stress Testing
• Testing to identify the number of users the system can
handle at a time before breaking down.
– Spike Testing
• The system is stressed suddenly for a short duration.

• System testing will test the Customer

Requirements

BITP 2213 | Sem 1 20222023 50

 Types of Testing in the V-Model
• The level of test is the primary focus of a system and
derives from the way a software system is designed
and built up.
• The main software testing types are:
– Component.
– Interface.
– System.
– Acceptance.
– Release.

BITP 2213 | Sem 1 20222023 51

V-Model

BITP 2213 | Sem 1 20222023 52

 Types of Testing in the V-Model
• Component test: Starting from the bottom the first test level is "Component Test",
sometimes called Unit Testing (white box). It involves checking that each feature
specified in the "Component Design" has been implemented in the component.
• Interface Testing is not focused on what the components are doing but on how they
communicate with each other as specified in "System Design". The tests are organized
to check all the interfaces, until all the components have been built and interfaced to
each other producing the whole system.
• System Test is not about checking the individual parts of the design, but about checking
the system as a whole -> one giant component. Can involve a number of specialist
types of test to see if all the functional and non-functional requirements have been
met. Systems Testing checks that the system that was specified has been delivered.
• Acceptance Testing checks the system against the "User Requirements". It is similar to
systems testing in that the whole system is checked but the important difference is the
change in focus: Acceptance Testing checks that the system delivers what was
requested.
• Release Testing is about seeing if the new or changed system will work in the existing
business environment. Mainly this means the technical environment.
BITP 2213 | Sem 1 20222023 53
 Defect Testing & Debugging
• The goal of defect testing is to discover defects in programs
• A successful defect test is a test which causes a program to
behave in an anomalous way
• Defect testing and debugging are distinct processes
• Defect testing is concerned with confirming the presence not
the absence of defects or errors.
• Debugging is concerned with locating and repairing these
errors
• Debugging involves formulating a hypothesis about program
behavior then testing these hypotheses to find the system
error

BITP 2213 | Sem 1 20222023 54

 Debugging Process

Test Test
Specification
results cases

Locate Design Repair Retest

error error repair error program

BITP 2213 | Sem 1 20222023 55

End of Lecture 9
Software Testing
BITP 2213 Software Engineering

Next on Lecture 10
Software Maintenance