Levels of Software Testing

Course Supervisor
Zaeem Tariq
Department of Computer Science, University of Karachi
Email: [email protected]
Levels of Software Testing
Unit Testing Outline
• Concept of Unit Testing
• Static Unit Testing
• Dynamic Unit Testing
• Mutation Testing
• Debugging
• Tools For Unit Testing

3
Concept of Unit Testing
• Static Unit Testing
• Code is examined over all possible behaviors that might arise during run
time
• Code of each unit is validated against requirements of the unit by
reviewing the code
• Dynamic Unit Testing
• A program unit is actually executed and its outcomes are observed
• One observe some representative program behavior, and reach
conclusion about the quality of the system
• Static unit testing is not an alternative to dynamic unit testing
• Static and Dynamic analysis are complementary in nature
• In practice, partial dynamic unit testing is performed concurrently
with static unit testing
•4 It is recommended that static unit testing be performed prior to the
 Static Unit Testing
• In static unit testing code is reviewed by applying techniques:
• Inspection: It is a step by step peer group review of a work product, with each
step checked against pre-determined criteria
• Walkthrough: It is review where the author leads the team through a manual or
simulated executed of the product using pre-defined scenarios
• The idea here is to examine source code in detail in a systematic
manner
• The objective of code review is to review the code, and not to evaluate
the author of the code
• Code review must be planned and managed in a professional manner
• The key to the success of code is to divide and conquer
• An examiner inspect small parts of the unit in isolation
• nothing is overlooked
5
• the correctness of all examined parts of the module implies the correctness of the whole
Dynamic Unit Testing
• The environment of a unit is emulated and tested in isolation
• The caller unit is known as test driver
• A test driver is a program that invokes the unit under test (UUT)
• It provides input data to unit under test and report the test result
• The emulation of the units called by the UUT are called stubs
• It is a dummy program
• The test driver and the stubs are together called scaffolding
• The low-level design document provides guidance for selection of input test data

6
Figure 3.2: Dynamic unit test environment
Dynamic Unit Testing
• Selection of test data is broadly based on the following techniques:
• Control flow testing
• Draw a control flow graph (CFG) from a program unit
• Select a few control flow testing criteria
• Identify a path in the CFG to satisfy the selection criteria
• Derive the path predicate expression from the selection paths
• By solving the path predicate expression for a path, one can generate the
data
• Data flow testing
• Draw a data flow graph (DFG) from a program unit and then follow the
procedure described in control flow testing.
• Domain testing
• Domain errors are defined and then test data are selected to catch those
faults
• Functional program testing
• Input/output domains are defined to compute the input values that will cause
7 the unit to produce expected output values
Mutation Testing
• Modify a program by introducing a
single small change to the code
• A modified program is called mutant
• A mutant is said to be killed when the
execution of test case cause it to fail.
The mutant is considered to be dead

8
 Mutation Testing
• Modify a program by introducing a single small change to the code
• A modified program is called mutant
• A mutant is said to be killed when the execution of test case cause it
to fail. The mutant is considered to be dead
• A mutant is an equivalent tot the given program if it always produce
the same output as the original program
• A mutant is called killable or stubborn, if the existing set of test cases
is insufficient to kill it
• A mutation score for a set of test cases is the percentage of
non-equivalent mutants killed by the test suite
• The test suite is said to be mutation-adequate if its mutation score is
100%

9
 Mutation testing
Consider the following program P • Test Case 1:
1. function mainFunc(argv) { • mainFunc(["node", "script.js", "1", "2", "3"]);
2. let r = 1; Output: Value of the rank is 3
3. for (let i = 2; i <= 3; i++) { • Test Case 2:
4. if (parseInt(argv[i]) > • mainFunc(["node", "script.js", "1", "2", "1"]);
parseInt(argv[r])) { Output: Value of the rank is 2
5. r = i; } • Test Case 3:
6. } • mainFunc(["node", "script.js", "3", "1", "2"]);
Output: Value of the rank is 1
7. console.log(`Value of the rank is ${r}`);
8. }
Mutant 1:
Change: for (let i = 1; i <= 3; i++) Result: Fails Test Case 1 & 2
(killable).
Mutant 2:
Change: if (i > parseInt(argv[r])) r = i; Result: Fails Test Case 2
(killable).
Mutant 3:
10
Change: if (parseInt(argv[i]) >= parseInt(argv[r])) r = i; Result: Passes all test cases
(non-killable).
Mutation testing

• Total Killed Mutants: 3 (Mutants 1, 2, and 4)

• Total Non-Equivalent Mutants: 3 (Mutants 1, 2, and 4; excluding Mutant 3)

• Mutation Score Calculation = ( Number of Killed Mutants /

Total number of non-equivalent mutantsNumber of killed mutants ) * 100

• Mutation Score Calculation = (3 / 3) * 100

• Mutation Score Calculation = 100 %

11
Debugging
• The process of determining the cause of a failure is known as
debugging
• It is a time consuming and error-prone process
• Debugging involves a combination of systematic evaluation, intuition
and a little bit of luck
• The purpose is to isolate and determine its specific cause, given a
symptom of a problem
• There are three approaches to debugging
• Brute force
• Cause elimination
• Induction
• Deduction
• Backtracking

12
Tools For Unit Testing
• Code auditor
• This tool is used to check the quality of the software to ensure that it meets some
minimum coding standard
• Bound checker
• This tool can check for accidental writes into the instruction areas of memory, or to
other memory location outside the data storage area of the application
• Documenters
• These tools read the source code and automatically generate descriptions and
caller/callee tree diagram or data model from the source code
• Interactive debuggers
• These tools assist software developers in implementing different debugging
techniques
Examples: Breakpoint and Omniscient debuggers
13
 Tools for Unit Testing
• Memory leak detectors
• These tools test the allocation of memory to an application which request for
memory and fail to de-allocate memory
• Static code (path) analyzer
• These tool identify paths to test based on the structure of code such as McCabe’s
cyclomatic complexity measure
• Simulators and emulators
• These tools are used to replace the real software and hardware that are not
currently available. Both the kinds of tools are used for training, safety, and
economy purpose
• Traffic generators
• These produces streams of transactions or data packets.
• Version control
• A version control system provides functionalities to store a sequence of revisions of
the software and associated information files under development
14
 Integration Testing Outline
• The Concept of Integration Testing
• Different Types of Interfaces
• Different Types of Interface Errors
• System Integration Techniques: Incremental,
Top-down, Bottom-up, and Sandwich and Big-bang
• Software and Hardware Integration
• Off-the-self Component Integration
• Off-the-shelf Component Testing

15
The Concept of Integration Testing
• A software module is a self-contained element of a system
• Modules are individually tested commonly known as unit testing
• Next major task is to put the modules, i.e., pieces together to construct
the complete system
• Construction of a working system from the pieces is not a
straightforward task because of numerous interface errors
• The objective of system integration testing (SIT) is to build a “working”
version of the system
• Putting modules together in an incremental manner
• Ensuring that the additional modules work as expected without disturbing the
functionalities of the modules already put together
• Integration testing is said to be complete when
• The system is fully integrated togethr
• All the test cases have been executed
16
• All the severe and moderated defects found have been fixed
The Concept of Integration Testing
The major advantages of conducting SIT are as follows:
• Defects are detected early
• It is easier to fix defects detected earlier
• We get earlier feedback on the health and acceptability of the
individual modules and on the overall system

• Scheduling of defect fixes is flexible, and it can overlap with

development

17
Different Types of Interfaces
Three common paradigms for interfacing modules:

• Procedure call interface

• Shared memory interface

• Message passing interface

The problem arises when we “put modules together” because of interface errors

Interface errors
Interface errors are those that are associated with structures existing outside the local environment of
a module, but which the module uses

18
Different Types of Interface Errors
•Data Format Errors: Occur when the exchanged data does not conform to the
expected format, leading to misinterpretation or processing issues.

•Data Type Mismatch: Arise when the data type expected by one component differs
from the data type provided by another, causing runtime errors or incorrect
behavior.

•Protocol Mismatch: Happen when two systems communicate using incompatible

protocols, resulting in failed interactions or miscommunication.

•Versioning Issues: Occur when components are incompatible due to differences in

software or API versions, often leading to breaking changes not being accounted
for.

•Timing Issues: Arise from race conditions or incorrect assumptions about the order
19 of operations, causing components to interact in unintended ways.
Different Types of Interface Errors
•State Management Errors: Occur when components fail to maintain or share state
information correctly, leading to inconsistencies in user sessions or data
processing.

•Error Handling Issues: Arise when components do not adequately handle errors,
resulting in cascading failures or unexpected behavior in dependent systems.

•Security Issues: Occur when vulnerabilities exist in the interface, such as improper
authentication or lack of encryption, exposing data to potential threats.

•Configuration Errors: Result from incorrect settings between integrated systems,

like mismatched endpoint URLs or authentication keys, leading to connectivity
problems.

•Boundary Condition Errors: Arise when input values exceed or fall below expected
20 limits, causing components to crash or behave unpredictably.
System Integration Techniques
Common approaches to perform system integration testing
• Top-down
• Bottom-up
• Sandwich
• Big-bang

Pre-requisite
A module must be available to be integrated
A module is said to available for combining with other modules when the
module’s check-in request form is ready

21
 Top-down • Module A has been decomposed into
modules B, C, and D
• Modules B, D, E, F, and G are terminal
modules
• First integrate modules A and B using
stubs C` and D` (represented by grey
boxes)
• Next stub D` has been replaced with its
actual instance D
• Two kinds of tests are performed:
Figure 7.1: A module hierarchy with
three levels and seven modules • Test the interface between A and D
• Regression tests to look for interface
defects between A and B in the presence
of module D

Figure 7.2: Top-down integration of

Figure 7.3: Top-down integration of
modules A and B
modules A, B and D
22
 Top-down
• Stub C` has been replaced with the actual
module C, and new stubs E`, F`, and G`
• Perform tests as follows:
• first, test the interface between A and C;
• second, test the combined modules A, B, and Figure 7.5: Top-down integration of
D in the presence of C modules A, B, C, D and E
• The rest of the process depicted in the right
hand side figures.

Figure 7.6: Top-down integration of

Figure 7.4: Top-down integration of modules A, B, C, D, E and F
modules A, B, D and C
23
 Bottom-up
• We design a test driver to integrate lowest-level
modules E, F, and G
• Return values generated by one module is
likely to be used in another module Figure 7.8: Bottom-up integration of
module E, F, and G
• The test driver mimics module C to integrate E,
F, and G in a limited way.
• The test driver is replaced with actual module ,
i.e., C.
• A new test driver is used
• At this moment, more modules such as B and
D are integrated
• The new test driver mimics the behavior of
module A
Figure 7.9: Bottom-up integration of
• Finally, the test driver is replaced with module module B, C, and D with F, F, and G
A24and further test are performed
Big-bang and Sandwich
Big-bang Approach
• First all the modules are individually tested
• Next all those modules are put together to construct the entire system which is
tested as a whole
Sandwich Approach
• In this approach a system is integrated using a mix of top-down, bottom-up, and
big-bang approaches
• A hierarchical system is viewed as consisting of three layers
• The bottom-up approach is applied to integrate the modules in the bottom-layer
• The top layer modules are integrated by using top-down approach
• The middle layer is integrated by using the big-bang approach after the top and the
bottom layers have been integrated

25
Software and Hardware Integration
• Integration is often done in an iterative manner

• A software image with a minimal number of core modules is loaded on a

prototype hardware

• A small number of tests are performed to ensure that all the desired
software modules are present in the build

• Next, additional tests are run to verify the essential functionalities

• The process of assembling the build, loading on the target hardware, and
testing the build continues until the entire product has been integrated

26
Off-the-self Component Integration
Organization occasionally purchase off-the-self (OTS) components
from vendors and integrate them with their own components
Useful set of components that assists in integrating actual
components:
• Wrapper: It is a piece of code that one builds to isolate the
underlying components from other components of the system
• Glue: A glue component provides the functionality to combine
different components
• Tailoring: Components tailoring refers to the ability to enhance
the functionality of a component
• Tailoring is done by adding some elements to a component to enrich it
with a functionality not provided by the vendor
• Tailoring does not involve modifying the source code of the component
27
Off-the-shelf Component Testing
OTS components produced by the vendor organizations are known
as commercial off-the-shelf (COTS) components
Three types of testing techniques are use to determine the suitability
of a COTS component:
• Black-box component testing: This is used to determine the
quality of the component
• System-level fault injection testing: This is used to determine
how well a system will tolerate a failing component
• Operational system testing: This kind of tests are used to
determine the tolerance of a software system when the COTS
component is functioning correctly
28
Outline of the Chapter
• Taxonomy of System Tests
• Basic Tests
• Functionality Tests
• Robustness Tests
• Interoperability Tests
• Performance Tests
• Scalability Tests
• Stress Tests
• Load and Stability Tests
• Regression Tests
• Documentation Tests
29
Taxonomy of System Tests

30 Figure 8.1: Types of system tests

Taxonomy of System Tests
• Basic tests provide an evidence that the system can be installed,
configured and be brought to an operational state
• Functionality tests provide comprehensive testing over the full range of
the requirements, within the capabilities of the system
• Robustness tests determine how well the system recovers from various
input errors and other failure situations
• Inter-operability tests determine whether the system can inter-operate
with other third party products
• Performance tests measure the performance characteristics of the
system, e.g., throughput and response time, under various conditions

31
Taxonomy of System Tests
• Scalability tests determine the scaling limits of the system, in terms of
user scaling, geographic scaling, and resource scaling
• Stress tests put a system under stress in order to determine the
limitations of a system and, when it fails, to determine the manner in
which the failure occurs
• Load and Stability tests provide evidence that the system remains
stable for a long period of time under full load
• Reliability tests measure the ability of the system to keep operating for
a long time without developing failures
• Regression tests determine that the system remains stable as it cycles
through the integration of other subsystems and through maintenance
tasks
• Documentation tests ensure that the system’s user guides are
accurate and usable
32
Functionality Tests

Figure 8.3: Types of functionality tests

33
Robustness Tests
Robustness means how much sensitive a system is to erroneous input and changes its
operational environment
Tests in this category are designed to verify how gracefully the system behaves in error
situations and in a changed operational environment

34
Figure 8.4: Types of robustness tests
Interoperability Tests
• Tests are designed to verify the ability of the system to inter-operate with
third party products

• Note: Read the 1xEV-DO example described in the book

• The re-configuration activities during interoperability tests is known as

configuration testing

• Another kind of inter-operability tests is called (backward) compatibility

tests
• Compatibility tests verify that the system works the same way across different
platforms, operating systems, data base management systems
• Backward compatibility tests verify that the current software build flawlessly works
with older version of platforms

35
Performance Tests
• Tests are designed to determine the performance of the actual system
compared to the expected one
• Tests are designed to verify response time, execution time, throughput,
resource utilization and traffic rate
• One needs to be clear about the specific data to be captured in order to
evaluate performance metrics.
• For example, if the objective is to evaluate the response time, then one
needs to capture
• End-to-end response time (as seen by external user)
• CPU time
• Network connection time
• Database access time
• Network connection time
• Waiting time

36
Scalability Tests
• Tests are designed to verify that the system can scale up to its
engineering limits
• Scaling tests are conducted to ensure that the system response
time remains the same, or increases by a small amount, as the
number of users are increased.
• There are three major causes of these limitations:
• data storage limitations
• network bandwidth limitations
• speed limit
• Extrapolation is often used to predict the limit of scalability
37
Stress Tests
• The goal of stress testing is to evaluate and determine the
behavior of a software component while the offered load is in
excess of its designed capacity
• The system is deliberately stressed by pushing it to and beyond
its specified limits
• It ensures that the system can perform acceptably under
worst-case conditions, under an expected peak load. If the limit
is exceeded and the system does fail, then the recovery
mechanism should be invoked
• Stress tests are targeted to bring out the problems associated
with one or more of the following:
• Memory leak
38
• Buffer allocation and memory carving
Load and Stability Tests
• Tests are designed to ensure that the system remains stable for a
long period of time under full load
• When a large number of users are introduced and applications that
run for months without restarting, a number of problems are likely to
occur:
• the system slows down
• the system encounters functionality problems
• the system crashes altogether
• Load and stability testing typically involves exercising the system with
virtual users and measuring the performance to verify whether the
system can support the anticipated load
• This kind of testing help one to understand the ways the system will
fare in real-life situations
39
Reliability Tests
• Reliability tests are designed to measure the ability of the system to remain
operational for long periods of time.
• The reliability of a system is typically expressed in terms of mean time to
failure (MTTF)
• The average of all the time intervals between successive failures is called
the MTTF
• After a failure is observed, the developers analyze and fix the defects,
which consumes some time – let us call this interval the repair time.
• The average of all the repair times is known as the mean time to repair
(MTTR)
• Now we can calculate a value called mean time between failure (MTBF) as
MTBF = MTTF + MTTR
40
Regression Tests
• In this category, new tests are not designed, instead, test cases are
selected from the existing pool and executed
• The main idea in regression testing is to verify that no defect has been
introduced into the unchanged portion of a system due to changes made
elsewhere in the system
• During system testing, many defects are revealed and the code is modified
to fix those defects
• One of four different scenarios can occur for each fix:
• The reported defect is fixed
• The reported defect could not be fixed inspite of making an effort
• The reported defect has been fixed, but something that used to work before has
been failing
• The reported defect could not be fixed inspite of an effort, and something that used
to work before has been failing

41
Documentation Tests
• Documentation testing means verifying the technical accuracy and
readability of the user manuals, tutorials and the on-line help

• Documentation testing is performed at three levels:

• Read test: In this test a documentation is reviewed for clarity, organization,
flow, and accuracy without executing the documented instructions on the
system

• Hands-on test: Exercise the on-line help and verify the error messages to
evaluate their accuracy and usefulness.

• Functional test: Follow the instructions embodied in the documentation to

verify that the system works as it has been documented.
42
Types of Acceptance Testing
• Acceptance testing is a formal testing conducted to determine
whether a system satisfies its acceptance criteria
• There are two categories of acceptance testing:
• User Acceptance Testing (UAT)
• It is conducted by the customer to ensure that system satisfies the contractual
acceptance criteria before being signed-off as meeting user needs.

• Business Acceptance Testing (BAT)

• It is undertaken within the development organization of the supplier to ensure that
the system will eventually pass the user acceptance testing.

43
Acceptance Criteria
• The acceptance criteria are defined on the basis of the following
attributes:
• Functional Correctness and • Maintainability and
Completeness Serviceability
• Accuracy • Robustness
• Data Integrity • Confidentiality and Availability
• Data Conversion • Scalability
• Backup and Recovery • Documentation
• Competitive Edge
• Usability
• Performance
• Start-up Time
• Stress
• Reliability and Availability
44
Selection of Acceptance Criteria
• The acceptance criteria discussed are too many and very general

• The customer needs to select a subset of the quality attributes

• The quality attributes are prioritize them to specific situation

• IBM used the quality attribute list CUPRIMDS for their products
– Capability, Usability, Performance, Reliability, Installation, Maintenance,
Documentation, and Service

• Ultimately, the acceptance criteria must be related to the business

goals of the customer’s organization
45
Acceptance Test Plan

46
Acceptance Test Execution
• The acceptance test cases are divided into two subgroups
• The first subgroup consists of basic test cases, and
• The second consists of test cases that are more complex to execute
• The acceptance tests are executed in two phases
• In the first phase, the test cases from the basic test group are executed
• If the test results are satisfactory then the second phase, in which the
complex test cases are executed, is taken up.
• In addition to the basic test cases, a subset of the system-level test
cases are executed by the acceptance test engineers to independently
confirm the test results
• Acceptance test execution activity includes the following detailed actions:
• The developers train the customer on the usage of the system
• The developers and the customer co-ordinate the fixing of any problem
discovered during acceptance testing
• The developers and the customer resolve the issues arising out of any
47 acceptance criteria discrepancy
Acceptance Test Execution
• The acceptance test engineer may create an Acceptance Criteria Change
(ACC) document to communicate the deficiency in the acceptance criteria
to the supplier
• A representative format of an ACC document is shown in Table 14.2.
• An ACC report is generally given to the supplier’s marketing department
through the on-site system test engineers

48
Acceptance Test Report
• The acceptance test activities are designed to reach at a conclusion:
• accept the system as delivered
• accept the system after the requested modifications have been made
• do not accept the system
• Usually some useful intermediate decisions are made before making the
final decision.
• A decision is made about the continuation of acceptance testing if the
results of the first phase of acceptance testing is not promising
• If the test results are unsatisfactory, changes be made to the system
before acceptance testing can proceed to the next phase
• During the execution of acceptance tests, the acceptance team prepares a
test report on a daily basis
• A template of the test report is given in Table 14.3
• At the end of the first and the second phases of acceptance testing an
49 acceptance test report is generated which is outlined in Table 14.4
Acceptance Test Report

50
Acceptance Test Report

51
Acceptance Testing in eXtreme Programming
• In XP framework the user stories are used as acceptance criteria
• The user stories are written by the customer as things that the
system needs to do for them
• Several acceptance tests are created to verify the user story has
been correctly implemented
• The customer is responsible for verifying the correctness of the
acceptance tests and reviewing the test results
• A story is incomplete until it passes its associated acceptance tests
• Ideally, acceptance tests should be automated, either using the unit
testing framework, before coding
• The acceptance tests take on the role of regression tests
52