Software Testing Foundations

Chapter

Chapter 3
3 Page 1
Certified Tester
Testing Throughout the Software
Life Cycle
Software Testing Foundations Certified Tester
MSTB-GTB
2017
Version 2.2.0
© Copyright MSTB-GTB
V 2.2.0 / 2017
 Disclaimer

• These slides are published under CC BY-NC-SA 4.0. This means besides
other things that they can be freely changed and used for non-commercial
purposes. By that we like to reach maximum flexibility for the CTFL
teachers at universities, colleges, and alike. However, we also have to
take into account the interests of the commercial trainings providers,
whose explicit request is not to have these slides freely available in the
Internet.
• Hence, we ask you to make use of the slides only within your lecture
series and exercises on CTFL related matter. Please also advise the
students that a free distribution of the slides would risk the GTB and/or
MSTB cooperation with universities, colleges, etc. – at least wrt. the
exchange of the slides.
• CC BY-NC-SA 4.0: https://creativecommons.org/licenses/by-nc-sa/4.0/

© Copyright MSTB-GTB
Chapter 3 Page 2 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
3 After this lecture, you should ...

• Know when testing throughout the software life cycle begins

• Know how development and test activities are related
• Know that software development models must be adapted to the context
of project and product characteristics
• Be able to compare component, integration, system and acceptance
testing
• Recall characteristics of good testing that are applicable to any life cycle
model
• Know how the testing of new product versions should look like
• Know typical indicators for maintenance testing
• Be able to describe the role of regression testing and impact analysis in
maintenance
• Be able to compare the different test types

© Copyright MSTB-GTB
Chapter 3 Page 3 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Learning Objectives for Testing Throughout the Software Life Cycle
3 (according to the Certified Tester Foundation Level Syllabus, Version 2011)

• 2.1 Software Development Models (K2)

– LO-2.1.1 Explain the relationship between development, test activities and
work products in the development life cycle, by giving examples using project
and product types(K2)
– LO-2.1.2 Recognize the fact that software development models must be
adapted to the context of project and product characteristics (K1)
– LO-2.1.3 Recall characteristics of good testing that are applicable to any life
cycle model (K1)

• 2.2 Test Levels (K2)

– LO-2.2.1 Compare the different levels of testing: major objectives, typical
objects of testing. Typical targets of testing (e.g. functional and structural) and
related work products, people who test, types of defects and failures to be
identified (K2)

© Copyright MSTB-GTB
Chapter 3 Page 4 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Learning Objectives for Testing Throughout the Software Life Cycle
3 (according to the Certified Tester Foundation Level Syllabus, Version 2011)

• 2.3 Test Types(K2)

– LO-2.3.1 Compare four software test types (functional, structural and change-related) by
example (K2)
– LO-2.3.2 Recognize that functional and structural tests occur at any test level (K1)
– LO-2.3.3 Identify and describe non-functional test types based on non-functional
requirements (K2)
– LO-2.3.4 Identify and describe test types based on the analysis of a software system‘s
structure or architecture (K2)
– LO-2.3.5 Describe the purpose of confirmation testing and regression testing (K2)

• 2.4 Maintenance Testing (K2)

– LO-2.4.1 Compare maintenance testing (testing an existing system) to testing a new
application with respect to test types, triggers for testing and amount of testing (K2)
– LO-2.4.2 Recognize indicators for maintenance testing (modification, migration and
retirement) (K1)
– LO-2.4.3 Describe the role of the regression testing and impact analysis in maintenance
(K2)

© Copyright MSTB-GTB
Chapter 3 Page 5 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 3
Chapter

Testing
Throughout
} Testing in Software Development Models
Test Levels
Testing of New Product Versions
the Software
Overview of Test Types
Life Cycle

© Copyright MSTB-GTB
Chapter 3 Page 6 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Big Picture
Test for Development Test
Development
data security levels levels W-Model
Test for
robustness
Maintenance General Iterative-incremental
test V-Model development model
Usability test

Test of compatibility Verification

Reliability test / data conversion and
Testing in software
Validation
development models
Stress test Non-Functional Drivers and
test Testing Tips for
throughout good testing stubs
Load test the lifecycle
Test
Overview of first
testing types Component
Volume test test
Test levels
Top
Performance test Test of new Integration down
product versions test
Acceptance Bottom
Functional test Structure-based up
test
test
Change-based Regression System Big
test Re-testing Ad-hoc
testing test bang

© Copyright MSTB-GTB
Chapter 3 Page 7 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 General V-Model (sequential development model)

Requirement Definition Acceptance Testing

Functional System Testing

System Design

Technical Integration Testing

System Design

Component
Component Testing
Specification

Programming

Legend
Test cases base on the respective
documents

© Copyright MSTB-GTB
Chapter 3 Page 8 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 General V-Model - Construction Phase (1/2)

• The constructive activities in the left branch of the V-Model

represent the construction phases, in which the software system is
described in increasingly more detail.
• Requirement Definition
The specifications and requirements of the customer or the later
system user are collected, specified and adopted. The purpose,
and desired features of the software system to be developed are
thereby determined.
• Functional System Design
The requirements are represented by the functions and dialog
sequences of the new system.

© Copyright MSTB-GTB
Chapter 3 Page 9 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 General V-Model - Construction Phase (2/2)

• Technical System Design

The technical implementation of the system is designed.
This includes, among others:
– Definition of interfaces to the system environment
– Decomposition of the system into more easily manageable
sub-systems (system architecture) that can be developed
independently if possible.
• Component Specification
For each sub-system the tasks, behaviour, internal structure, and
interfaces to other sub-systems are defined
• Programming
Implementation of every specific component (module, unit, class,
or similar) using a programming language

© Copyright MSTB-GTB
Chapter 3 Page 10 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 General V-Model - Test Levels

• As defects are found most easily on the same level of

abstraction at which they incurred, the right branch assigns each
specification or construction step to a corresponding level of testing:
• Component Testing
checks whether each individual software component satisfies the
requirements of its specification.
• Integration Testing
checks whether groups of components as in the technical system
design intended operate together.
• System Testing
checks whether the system as a whole meets the specified
requirements.
• Acceptance Testing
checks whether the system shows the contractually approved
performance from a customer perspective

© Copyright MSTB-GTB
Chapter 3 Page 11 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Validation

• Checking the development results against the original

requirements is called validation
• Definition: Confirmation by examination and through provision of
objective evidence that the requirements for a specific intended
use or application have been fulfilled [ISO 9000].
• The task of each stage of testing is to prove whether the
development results meet the requirements which are specified or
relevant for the respective level .
• When performing validation, the testers evaluate whether a
(partial) product can actually solve a specified task and therefore is
suitable for its intended purpose.
• Examines whether the product is useful in the context of the
intended product use.

© Copyright MSTB-GTB
Chapter 3 Page 12 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Verification
• In addition to validating testing, using the V-model
also requires verification testing.
• Definition: Confirmation by examination and through the provision
of objective evidence that specified requirements have been
fulfilled [ISO 9000].
• Verification, unlike validation, is based on a single development
phase and proves the correctness and completeness of the results
from a phase relative to its direct specification (phase input
documents).
• Examines whether the specifications have been implemented
correctly, regardless of the intended purpose or use of the product.
• In practice, every test contains both validation and verification
aspects. The relative amount of validation increases with every
level of testing.

© Copyright MSTB-GTB
Chapter 3 Page 13 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 V-Model: Summary

• Construction and test activities are separated but

equal (left side / right side).
• “V” illustrates the testing aspects verification and validation.
• There are different testing levels in which each level performs tests
“against” their corresponding development phase.
• V-Model gives the impression that testing starts relatively late after
implementation. This is wrong.
• The test steps in the right branch of the model are to be
understood as phases of test execution.
• The associated test preparation (test planning, test specification)
starts earlier and is performed parallel with the development steps
in the left branch (see W-model).

© Copyright MSTB-GTB
Chapter 3 Page 14 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Digression: W-Model – Development of the V-Model

Requirement Preparation Implementation

Debug
Definition Acceptance Testing Acceptance Testing

Functional System Preparation System Implementation

Debug
Design Testing System Testing

Technical System Preparation Implementation

Debug
Design Integration Testing Integration Testion

Component Preparation Implementation

Debug
Specification Component Testing Component Testing

Programming Modification

Legend:
Test, debug Source:
Review, Test cases,, Spillner, Roßner, Winter, Linz:
Documents Test frame modify, re-test Software Testing Practice – Test Management.
Rocky Nook, 2007

© Copyright MSTB-GTB
Chapter 3 Page 15 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Iterative-Incremental Development Models

• Development takes place in small

steps.
• The system is not created “in one Activity 1
Activity 1 Activity 2
...
go”, but in a planned series of Activity 2
versions and intermediate
deliveries (increments). Activity n

• Enhancements are made at each Activity n Activity ...

iteration. Activity ...

• Each iteration results in one

...
expanding (yet) incomplete
system. 1st Iteration 2nd Iteration

© Copyright MSTB-GTB
Chapter 3 Page 16 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Testing in Iterative-Incremental Development Models

• Testing needs to be adjusted to the development sequence.

• For each increment (intermediate delivery) reusable tests need to
be planned.
• In each iteration the existing tests can be reused.
• For new functionality, additional tests are needed.
• For each increment within an iteration the different testing levels
can be performed.
• Continuous integration testing and regression testing is necessary.
• Verification and validation can be conducted for each increment.

© Copyright MSTB-GTB
Chapter 3 Page 17 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Examples of Iterative-Incremental Development Models

• Prototyping
• Rational Unified Process (RUP)
• Agile Development Models
– Extreme Programming (XP)
– Dynamic Systems Development Method (DSDM)
– SCRUM
– ... Sources:

• RUP
Kruchten, P.: Der Rational Unified Process – Eine Einführung.
Addison-Wesley-Longman, 1999.
• XP
Beck, K.: Extreme Programming. Addison-Wesley, München, 2000
• DSDM
Stapleton, J. (Hrsg.): DSDM: Business Focused Development
(Agile Software Development Series). Addison-Wesley, 2002
• SCRUM
Beedle, M.; Schwaber, K.: Agile Software Development with
Scrum. Prentice Hall, 2001

© Copyright MSTB-GTB
Chapter 3 Page 18 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Tips for Good Testing

• Regardless of the chosen software development model, the

following aspects regarding testing have been proven to be useful:
– For every development activity there is a corresponding test
activity.
– Testing activities should start early in the development cycle.
Test analysis and design should begin parallel to the
corresponding stage of development.
– Include testers early in the review process of the development
documents.
– Software development models should not be used "out of the
box". They must be adapted to project and product
characteristics (e.g. number of applicable test levels, the
number and length of iterations, etc. need to be adapted per
project context)

© Copyright MSTB-GTB
Chapter 3 Page 19 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 3
Chapter

Testing
Throughout
} Testing in Software Development Models
Test Levels  Component Test
Test of New Product Versions
theSoftware Overview of Test Types
Life Cycle

© Copyright MSTB-GTB
Chapter 3 Page 20 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Big Picture
Test for Development Test
Development
data security levels levels W-Model
Test for
robustness
Maintenance General Iterative-incremental
test V-Model development model
Usability test

Test of compatibility Verification

Reliability test / data conversion and
Testing in software
Validation
development models
Stress test Non-Functional
test Testing Tips for Drivers and
throughout good testing stubs
Load test the lifecycle
Overview of Test
testing types Component first
Volume test test
Test levels
Top
Performance test Test of new Integration down
product versions test
Acceptance Bottom
Functional test Structure-based up
test
test
Change-based Regression System Big
test Re-testing Ad-hoc
testing test bang

© Copyright MSTB-GTB
Chapter 3 Page 21 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Remarks
• The individual testing levels component testing, integration testing, system
testing and acceptance testing are explained in the V-Model.
• The testing levels also occur in other development models. Their names
may, however, be different.
• Testing levels start with the testing of small units and then gradually move
up to integrate the whole system until it is complete and accepted by the
customer.
• Additional possible levels are:
– Component integration testing
is carried out after component testing to test the interaction of software
components.
– System integration testing
is carried out after the system testing to test the interaction between
different systems or between hardware and software.

© Copyright MSTB-GTB
Chapter 3 Page 22 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Test Levels
• The distinction between different levels of testing is more than just
a time-based subdivision of testing activities.
• Test levels are distinguished from each other by:
- different test objects
- different test basis
- different testing strategy
- application of different test methods
- using different testing tools
- different goals and results
- different responsibilities
- different specialized test personnel required
• If configuration data is part of the system, the test of these data
should also be taken into account in the test planning.

© Copyright MSTB-GTB
Chapter 3 Page 23 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Component Testing – Explanation

• Component testing is the first test level, following directly after the
programming phase
• The created software modules are subjected to a systematic test for
the first time.
• Depending on the programming language used, these smallest units
of software are called differently
- modules, units or
classes (in case of object oriented programming).
- the corresponding tests are called module, unit, and class testing.
• Abstracted from the programming language the terms components or
software modules are used.

© Copyright MSTB-GTB
Chapter 3 Page 24 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Component Testing – Test Objects

• Every single software component is examined in

isolation from other software components of the system.
• The isolation is meant to exclude external component influences
during the test.
• If the testing exposes a defect, the cause can be assigned clearly
to the tested component.
• The component that is tested can also be made up of several
modules which are assembled into a unit. The objective of
component testing is to test the internal component aspects but not
the interaction with neighbouring components.
• The test basis for component testing is primarily the specification of
the component (created at the design stage) and their program
code. In addition, all other documents which are related to the
tested component also could be considered as test basis.

© Copyright MSTB-GTB
Chapter 3 Page 25 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Component Testing – Test Harness (1/2)

• Test harness contains

- Driver:
Calls the service of
the test object Driver
Test Case1 Test Case 2 ... Test case n
and/or
- Stubs: Stub 1
Dummy simulation
Stub 2
of the service that Test Outputs

the test object Test Object

... Comparison
imports & Record

Stub k

Runtime Environment, Analysis

Tools, Monitors
Stubs

© Copyright MSTB-GTB
Chapter 3 Page 26 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Component Testing – Test Harness (2/2)

• At this test level, the tasks are closely related to development.

• To create the test environment, developer know-how is required.
• The program code of the test object - at least the code of the
interface - must be available and understood, so that the driver can
be programmed to call the test object correctly.
• Therefore, component tests are often carried out by developers.
• For these reasons, component testing is often referred to as
development testing.

© Copyright MSTB-GTB
Chapter 3 Page 27 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Component Testing – Test Objectives (1/4)

• The most important task of component testing is to ensure that the

test object implements the specified functionality correctly and
completely.
• Functionality is equivalent to the in/output-behaviour of the test
object.
• In order to check the correctness and completeness of
implementation, the component undergoes a series of tests, in
which each test case covers a certain in/output combination
(partial functionality).

© Copyright MSTB-GTB
Chapter 3 Page 28 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Component Testing – Test Objectives (2/4)

• Typical failures that are detected during functional component

testing are computation errors and missing or incorrect program
paths (e.g. incorrect handling of special cases).
• Test of robustness
– During the operation of the entire system, each software
component must work with a large number of neighbouring
components and exchange data with them.
– It is possible that the component might be used in a wrong
way, i.e., is addressed or used contrary to its specification. In
such cases, the incorrectly addressed component should not
fail immediately and bring the whole system down.
– The situation should be caught by the module (exception
handling) and the component must respond »rationally«.

© Copyright MSTB-GTB
Chapter 3 Page 29 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Component Testing – Test Objectives (3/4)

• In addition to functionality and robustness, all component

characteristics should be checked that affect the quality of the
component and which cannot be tested at higher testing levels
e.g., efficiency and maintainability.
• Efficiency
– Indicates how economically the component handles the
available computer resources. Sub criteria (such as memory
consumption in kilobytes, response time in milliseconds) can
be accurately measured in the test.

© Copyright MSTB-GTB
Chapter 3 Page 30 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Component Testing – Test Objectives (4/4)
• Maintainability
– includes all those properties of a program that will determine how
easy or difficult it is going to be to change or enhance the
program. The crucial factor is how much effort is needed to
understand the program and its context.
– The following testing aspects are in the foreground: code
structure, modularity, commenting the code, understandability,
currentness of documents etc.
• Maintainability cannot, of course, be verified by dynamic testing.
– Analysis of program code and specifications are necessary.
Central to this is the static test and, in particular, the (code)
review.
– Since the characteristics of each component are examined, such
static analysis is best carried out in the context of the component
tests.

© Copyright MSTB-GTB
Chapter 3 Page 31 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Component Testing – Test Strategy (1/2)

• With access to the program code, the component test can be

carried out using white-box test techniques (refer to Chapter 5).
• The tester can create test cases by taking advantage of his/her
knowledge of component-internal program structures, methods and
variables.
• The availability of the program code is also useful during the
execution of the test. With appropriate tools (e.g. debuggers), the
program variables can be observed during the test to infer correct
or incorrect behaviour of the components.

© Copyright MSTB-GTB
Chapter 3 Page 32 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Component Testing – Test Strategy (2/2)

• In practice, in many cases the component test is carried out as a

combination of black-box and white-box test (refer to Chapter 4)
i.e., the inner structure is not used to design the test cases.
– Real software systems consist of hundreds or thousands of
components. An examination of the code is probably only
feasible for selected components.
– Due to integration, the elementary program components are
often composed to make bigger units.
• Frequently, only the composed program components are visible to
the tester as testable units. These test objects may already be too
large to allow an understanding of or permit observation or
intervention at the code level with a reasonable amount of effort.
• Whether elementary or already composed programme components
are going to be tested is a matter to be addressed in the integration
and test planning.

© Copyright MSTB-GTB
Chapter 3 Page 33 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 "Test-first" Approach at Component Testing

• Principle:
First, create the test cases and automate the test execution. Only then
develop/program the test object (software to be tested).

• Iterative approach:
Test object is checked with test cases and improved until the tests
show no failures

• Test driven development:

A way of developing software where the test cases are developed,
and often automated, before the software is developed to run those
test cases.
Note: at first, tests are programmed (make it red), then the code is
programmed (make it green) and, finally, the code is refactored (make
it blue)

© Copyright MSTB-GTB
Chapter 3 Page 34 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 3
Chapter

Testing
Throughout
} Testing in Software Development Models
Test Levels  Integration Test
Test of New Product Versions
the Software
Overview of Test Types
Life Cycle

© Copyright MSTB-GTB
Chapter 3 Page 35 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Big Picture
Test for Development Test
Development
data security levels levels W-Model
Test for
robustness
Maintenance General Iterative-incremental
test V-Model development model
Usability test

Test of compatibility Verification

Reliability test / data conversion and
Testing in software
Validation
development models
Stress test Non-Functional
test Testing Tips for Drivers and
throughout good testing stubs
Load test the lifecycle
Overview of Test
testing types Component first
Volume test test
Test levels
Top
Performance test Test of new Integration down
product versions test
Acceptance Bottom
Functional test Structure-based up
test
test
Change-based Regression System Big
test Re-testing Ad-hoc
testing test bang

© Copyright MSTB-GTB
Chapter 3 Page 36 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Integration Testing – Explanation

• Second test level after component testing

• Assumption
– The assigned test objects (i.e. individual components) have
already been tested
– Defects found have been corrected
• Integration
– Developers, testers and special integration teams build
groups of components into larger modules and sub systems.
• Integration Testing
– Tests whether the interaction of all components works
properly.
• Objective
– To find defects at the interfaces and in the interaction between
integrated components.

© Copyright MSTB-GTB
Chapter 3 Page 37 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Integration Testing – Test Objects
• Individual components are assembled progressively
into larger units (integration) and undergo an integration test.
• Each sub-system can be the basis for further integration of larger units.
• Test objects of the integration testing can thus be successively
assembled units.
• In practice
– a software system is rarely developed on a greenfield, but an
existing system is modified, extended or linked with other
systems.
– many system components are standard products which are
purchased on the market. In component testing, those old or
standard components probably go unnoticed. In integration
testing, these system parts need to be taken into account and
their interaction with other parts needs to be controlled.

© Copyright MSTB-GTB
Chapter 3 Page 38 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Integration Testing – Test Harness

• During integration testing, drivers are required,

that provide the test objects with test data, accept the test results,
and record them.
• Any existing drivers from the component testing can normally be
reused.
• As interface calls and data traffic needs to be tested via the driver
interfaces, integration testing often needs additional diagnostic
tools called monitors to read and record the data passed between
the components.

© Copyright MSTB-GTB
Chapter 3 Page 39 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Integration Testing – Test Objectives (1/2)
• The objectives of integration testing are clear:
– detect defects in the interfaces.
– detect defects in the interaction between components.
– test non-functional characteristics (e.g. performance) if possible
• Problems can occur even when trying to integrate two components, when
they cannot be built
– because their interface formats are incompatible
– because some data is missing
– because the developers have split the system into different
components than specified.
• It can be difficult to find problems affecting the performance of the
interacting program parts.
• These are defects in the data exchange between the components which
can only be detected by a dynamic test.

© Copyright MSTB-GTB
Chapter 3 Page 40 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Integration Testing – Test Objectives (2/2)

• The following defects can be broadly distinguished:

– A component transmits nothing or syntactically incorrect data
to the receiving component, which causes it to malfunction or
crash (functional defect of a component, incompatible formats,
interfaces, protocol errors).
– The communication works correctly, but the components
involved interpret data passed differently (functional defects of
a component, conflicting or misinterpreted specifications).
– The data is transferred correctly, but at the wrong time (timing
problem) or in too short intervals of time (throughput or load
problem).
None of these types of defects may be found in component
testing because the failure only appear in the interaction
between two software units.

© Copyright MSTB-GTB
Chapter 3 Page 41 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Integration Testing without Component Testing?

 Is it possible not to perform component testing, so that

all test cases are executed after the integration?
• Of course this is possible and, unfortunately, this is often an
approach encountered in practice.
• Some serious disadvantages are connected with this approach:
– Most failures that will occur in such tests are caused by
functional defects of individual components. So an implicit
component test has to be executed in an unsuitable test
environment, which complicates the access to the single
component.
– As no suitable access to the single component is possible,
some failures cannot be provoked and therefore defects can
not be found.
– If a failure occurs in the test, it may be difficult or even
impossible to narrow down its cause.

© Copyright MSTB-GTB
Chapter 3 Page 42 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Integration Testing – Integration Strategy (1/6)

 In which order must the individual components be integrated so

that the necessary test operations can be carried out as simply and
quickly as possible?
• The various software components being developed in the project
are completed at different times which may be weeks or months
apart.
• No project manager or test manager can tolerate that testers are
idle until all components are ready and can be integrated together.

© Copyright MSTB-GTB
Chapter 3 Page 43 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Integration Testing – Integration Strategy (2/6)

The test manager can plan on the following basic strategies:

Top-down Integration
– The test sequence begins with the component that calls other
components, but (except the operating system) is not called
itself. The lower components are replaced by stubs.
Successively, the components of lower system layers are
integrated. Each tested higher layer serves as a driver.
Advantage
– No, or only simple drivers are needed, as higher and pre-
tested components build the major part of the environment.
Disadvantage
– Subordinated, not yet integrated components must be
replaced by stubs, which can be very costly.

© Copyright MSTB-GTB
Chapter 3 Page 44 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Integration Testing – Integration Strategy (3/6)

Bottom-up Integration
– The test begins with the elementary components of the
system that do not call other components (except the
functions of the operating system). Successively larger
subsystems are assembled from tested components,
followed by a test of this integration.
Advantage
– No stubs are needed.
Disadvantage
– Higher components must be simulated by drivers.

© Copyright MSTB-GTB
Chapter 3 Page 45 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Integration Testing – Integration Strategy (4/6)

Remarks on top-down and bottom-up strategies

– Both strategies can only be used in their pure form if the
system to be tested is programmed in a strictly hierarchical
manner. This is rarely found in practice.
– Therefore, in reality, we always choose a more or less
individual blend of the two integration strategies.
– The larger the number of components introduced in an
individual integration, the more difficult it becomes to
isolate defects and the greater the time required for
debugging.

© Copyright MSTB-GTB
Chapter 3 Page 46 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Integration Testing – Integration Strategy (5/6)

Ad-hoc Integration
– The components are integrated, for example in the
(random) order of their completion. Once a component has
completed its testing, it is checked whether it belongs to an
already existing and previously tested component, or it fits
into a semi-integrated subsystem. If so, both elements are
integrated and the integration test is executed between the
two.
Advantage
– Time saved, as each component is integrated in its proper
environment as early as possible.
Disadvantage
– Stubs as well as drivers are needed.

© Copyright MSTB-GTB
Chapter 3 Page 47 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Integration Testing – Integration Strategy (6/6)

Non incremental integration – big-bang-integration

– The integration waits until all software components have been
developed. Then, everything is thrown together at once. In the
worst case, upstream component tests are abandoned.

Disadvantages
– The waiting time for the big-bang to finish is a lost in test
execution time. Since testing often suffers from a lack of time
anyway, not one day of testing should be given away.
– All failures are concentrated on one build; it will be difficult or
impossible to get the system to work at all.
– The localization and correction of defects is difficult and time
consuming.

© Copyright MSTB-GTB
Chapter 3 Page 48 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Integration Testing – Integration Strategy
? Example (1)

Example hierarchy

B C D

E F G

© Copyright MSTB-GTB
Chapter 3 Page 49 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Integration Testing – Integration Strategy
? Example (2)

Big-Bang Integration

Test
A
Test
B

A
Test
Test
C
A,B,C,
B C D
D,E,F,
G E F G
Test
D

Test
E Test
F

Component test
Integration test
Time
© Copyright MSTB-GTB
Chapter 3 Page 50 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Integration Testing – Integration Strategy
? Example (3)

Top-Down Integration
Test
A
Test
B

Test
C Test A
Test
A,B,C,
A,B,
D,E,F, B C D
C,D
G
Test
D E F G

Test
E
Test
F
Test
G

Component test
Time Integration test

© Copyright MSTB-GTB
Chapter 3 Page 51 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Integration Testing – Integration Strategy
? Example (4)

Bottom-Up Integration

Test
A Test
Test B,E,F
B

Test
A
C Test
A,B,C,
D,E,F, B C D
G
Test
Test
E F G
D
D,G
Test
E
Test
F
Test
G
Component test
Time Integration test

© Copyright MSTB-GTB
Chapter 3 Page 52 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Integration Testing – Choosing a Strategy

• An optimal integration strategy (time and cost saving) depends on

conditions that must be analysed individually in each project:
• The system architecture
– determines which and on how many components the whole
system consist of and how they depend on one another.
• The project plan
 defines when single components are to be developed and
when they are ready for testing.
• The test plan/master test plan
– determines how intensive the system aspects must be tested
and at what specific test level.
• The test manager
– must establish an integration strategy from these conditions
which is suitable for the project. This is done in consultation
with the project manager.

© Copyright MSTB-GTB
Chapter 3 Page 53 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Addendum: Test Plan - Terms from the glossary

• Master test plan

A test plan that typically addresses multiple test levels.
• Level test plan
A test plan that typically addresses one test level.
• Test plan
A document describing the scope, approach, resources and
schedule of intended test activities. It identifies amongst others test
items, the features to be tested, the testing tasks, who will do each
task, degree of tester independence, the test environment, the test
design techniques and test measurement techniques to be used,
and the rationale for their choice, and any risks requiring
contingency planning. It is a record of the test planning process
[IEEE 829].

© Copyright MSTB-GTB
Chapter 3 Page 54 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 3
Chapter

Testing
Throughout
} Testing with Software Development Models
Test Levels  System Testing
Testing New Product Versions
the Software
Overview of Test Types
Life Cycle

© Copyright MSTB-GTB
Chapter 3 Page 55 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Big Picture
Test for Development Test
Development
data security levels levels W-Model
Test for
robustness
Maintenance General Iterative-incremental
test V-Model development model
Usability test

Test of compatibility Verification

Reliability test / data conversion and
Testing in software
Validation
development models
Stress test Non-Functional
test Testing Tips for Drivers and
throughout good testing stubs
Load test the lifecycle
Overview of Test
testing types Component first
Volume test test
Test levels
Top
Performance test Test of new Integration down
product versions test
Acceptance Bottom
Functional test Structure-based up
test
test System
test
Change-based Regression Big
test Re-testing Ad-hoc
testing bang

© Copyright MSTB-GTB
Chapter 3 Page 56 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Big Picture: System Testing

Testing Drivers and

throughout stubs
the lifecycle
Test
Component first
test
Test levels
Top
Integration down
test
Acceptance Bottom
test up
System
test
See non- Big
Ad-hoc
functional test bang

Requirements-based
test
Test of whole
system

Business process-based Test strategy

test Use-case-based  functional test
test

© Copyright MSTB-GTB
Chapter 3 Page 57 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 System Testing – Definition

• After completed integration testing, system testing (third level of

testing) may be required before the system is delivered.
• System testing verifies if specific product requirements are met.
• Reasons for system testing
– In the lower test levels the software was tested against
technical specifications, mostly from the developer‘s
viewpoint. System testing however examines the system from
the customer‘s perspective and its later user. The testers
validate if the requirements were fully and appropriately
implemented.
– Many functions and system properties result from the
interaction of all system components and are therefore only
observable and testable at the level of the overall system.

© Copyright MSTB-GTB
Chapter 3 Page 58 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 System Testing – Test Objects

• With completed integration testing the final assembled software

system is available.
• System testing is concerned with the behavior of a whole system.

© Copyright MSTB-GTB
Chapter 3 Page 59 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 System Testing – Test Harness (1/2)

• The test environment should correspond as much as

possible to the final production environment.
• Instead of drivers and stubs, the actual hard- and software
products used should be installed in the test environment
(hardware equipment, system software, driver software, network,
external systems etc.).

© Copyright MSTB-GTB
Chapter 3 Page 60 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 System Testing – Test Harness (2/2)

• To save costs, time and effort, the system testing for database-
based information systems is often performed in the customer’s
production environment instead of a separate test environment.
• This is undesirable for the following reasons:
– In system testing, failures will occur. This brings the risk that
the customer’s production environment will be affected.
Expensive failures and loss of data in the customer’s
productive system can be the result.
– The testers have little or no control over the parameters and
configurations of the production environment. Through the
simultaneous operation of other customer systems, the test
criteria might be subtly modified. The system testing that has
already been conducted is hard or even impossible to
reproduce.

© Copyright MSTB-GTB
Chapter 3 Page 61 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 System Testing – Testing Objective (1/2)
 How well does the completed system fulfill the
stated requirements?
• Two classes of requirements are considered separately:
– Functional requirements
– Non-functional requirements

• Functional requirements
– Specify the behavior which the system or system parts have
to perform.
– They describe “what” the system(part) has to perform. Their
implementation is required in order for the system to be
suitable for use.
– Characteristics of the quality attribute “functionality” according
to [ISO 9126] are: suitability, accuracy, interoperability,
security, compliance.

© Copyright MSTB-GTB
Chapter 3 Page 62 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 System Testing – Testing Objective (2/2)
 How well does the completed system fulfil the
stated requirements?

• Non-functional requirements
– Describe “how” well certain attributes of the system(part)
behavior should be performed.
– Its implementation strongly influences how satisfied the
customer or user will be with the product and how much they
like using it.
– Quality attributes according to [ISO 9126] are: reliability,
efficiency, usability. Indirectly, maintainability and portability
may also influence the customer‘s satisfaction.
– With some projects data is the main focus, e.g., with data
conversion projects and with applications like Data
Warehouses. Requirements relating to data quality must also
be considered in the system test.
© Copyright MSTB-GTB
Chapter 3 Page 63 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 System Testing – Testing Strategy
Functional Requirements (1/3)

• In the project phase the requirements were collected

and recorded in a requirements specification document
• Testing based on requirements
– The approved requirements specification document is used
as test basis.
– The system specification will also be verified by a review.
– For each requirement (functional and non-functional) test
cases are derived and recorded in the system specification.

© Copyright MSTB-GTB
Chapter 3 Page 64 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 System Testing – Testing Strategy
Functional Requirements (2/3)

Business process-based testing

– Used if a software system has the purpose of automating or
supporting the business process of a customer.
– A business process analysis (mostly created as part of the
requirements analysis) shows which business processes are
relevant, how often they are used and in which context they
appear (persons, companies, external systems etc.)
– Subsequently, test scenarios (sequences of test cases) are set
up on the basis of this analysis that represent typical business
situations.
– The priority of the test scenarios depends on the amount and
relevance of the particular business processes.
• Whereas testing based on the requirements focuses on individual
system functions, a business process-based strategy focuses on a
series of connected tests.

© Copyright MSTB-GTB
Chapter 3 Page 65 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 System Testing – Testing Strategy
Functional Requirements (3/3)

• Use Case Testing

– System test cases can be derived which describe how the
user handles the system and which actions are typically
carried out.
– Different user groups each possess their own user profiles.
Typical interactions or use cases with a realistic frequency
can be identified for these groups.
– From these typical interactions, test scenarios can be derived.
– On the basis of the frequency that various actions are
performed in using the software, the tester determines how
important the corresponding test scenario is and with which
priority it therefore has to be included in the test plan.

© Copyright MSTB-GTB
Chapter 3 Page 66 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 System Testing – Test Strategy
Non-functional Requirements (1/4)

• Non-functional requirements are just as important and eligible as

functional requirements, as they determine important quality
attributes which the system must posses.
• The following non-functional system properties should be taken
into consideration (normally in the system testing):
– Load Testing
• Measurements of the system behavior in relation to
increasing system load (e.g. Number of parallel users,
number of transactions).
– Performance Testing
• Measurements of processing speed and response time for
specific use cases, normally in relation to increasing load.

© Copyright MSTB-GTB
Chapter 3 Page 67 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 System Testing – Test Strategy
Non-functional Requirements (2/4)

• Volume testing
– Observation of system behavior in relation to the amount of
data transmitted (e.g. processing large volumes of data).
• Stress testing
– Observation of system behavior when overloaded.
• Testing (data) security
– Test of access authorisations to the system or data.
• Testing reliability
– During nonstop operation (e.g. amount of failures per hour
with given user profile).
• Testing robustness
– Against incorrect operation, programming errors, hardware
failure etc. Tests also include system recovery handling.

© Copyright MSTB-GTB
Chapter 3 Page 68 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 System Testing – Test Strategy
Non-functional Requirements (3/4)
• Testing compliance/conversion of data
– Verifying the compatibility of existing systems. Import/export of
database etc.
• Testing different configurations
– of the system, e.g., different versions of operating systems,
language, hardware platform etc.
• Testing usability
– Verifying the suitability of operation, understandability, and of
system output etc., in relation to the needs of the user.
• Testing documentation
– Testing conformity with the system behavior (e.g. user manual)
• Testing portability/maintainability
– Understandability and current content of development documents,
modular system structure etc.

© Copyright MSTB-GTB
Chapter 3 Page 69 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 System Testing – Test Strategy
Non-functional Requirements (4/4)

• When testing non-functional requirements, we may

face the problem that requirements are often incomplete or
imprecisely worded.
– Phrases such as “The system should be easy to operate” or
“should react fast” cannot be tested in this form.
– Additionaly, many non-functional requirements are taken for
granted so that they are not mentioned in the requirements
specification (implied requirements).
– Even though these requirements are not specified, they are
still relevant attributes which also need to be validated.

© Copyright MSTB-GTB
Chapter 3 Page 70 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 System Testing – Test Strategy
Quality Characteristics

Quality Characteristics*
according to [ISO 9126] * replaced by [ISO 25010]

Functionality Reliability Usability Efficiency Maintainability Portability

Suitability Maturity Understandability Time behavior Analyzability Adaptability

Accuracy Fault tolerance Learnability Usage behavior Changeability Installability
Interoperability Recovery Usability Stability Co-existence
Security Attractiveness Testability Replaceability
Compliance

Examples: Examples : Examples : Examples : Examples : Examples :

Functional testing Testing robustness Testing usability Load testing Testing Testing various
maintainability configurations
Testing (data) Testing recovery Verification of user Performance (code review,
security documentation testing architecture-review)
including online
Testing data help Volume testing Verification of
compatabiltiy / data documentation
conversion Stress testing
Static analysis

© Copyright MSTB-GTB
Chapter 3 Page 71 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 System Testing – Test Strategy
 Quality Characteristics
Quality Characteristics*
according to [ISO 25010]

Functional Performance
Reliability Usability Security Compatibility Portability Maintain-ability
suitability Efficiency

Learnability
Appropriateness
Functional Confidentiality Modularity
Maturity Tecognizability
completeness Time behavior
Integrity Installability Reusability
Availability Operability Co-Existence
Functional Resource
Non-repudiation Replaceability Analyzability
correctness Recoverability User error Interoperability utilization
protection Accountability Adaptability Modifiability
Functional Fault tolerance Capacity
appropriateness User interface Authenticity Testability
aesthetics
Accessibility

Examples: Examples: Examples: Examples: Examples: Examples: Examples:

Functional Testing Testing Testing Testing various configurations Testing Load testing
testing robustness usability (data) maintain- Performance
security ability testing
Volume testing
Stress testing
© Copyright MSTB-GTB
Chapter 3 Page 72 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 System Testing
 Requirements on the Quality of Data (1/2)

• The amount of data in companies increases continuously, e.g.,

finances, materials, products and addresses.
• Requirements on the quality of data are very high because falsified,
incorrect or missing data can cause a lot of damage.
• Data quality problems occuring at the time of input
– falsely captured data, due to no or incomplete verification
– unsuitable input interfaces (e.g. complex input forms)
– lack of awareness that leads to careless handling of data.
– no standards at the time of input
– no validation at the interfaces between systems
• Data quality problems occuring when previously independent systems
are integrated
– missing data architecture
– redundant, heterogeneous data
• Data quality problems occuring during migration
– migration errors (e.g. loss of data, semantic incompatibility, etc.)
• Missing responsibilities may also result in data quality problems

© Copyright MSTB-GTB
Chapter 3 Page 73 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 System Testing
 Requirements on the Quality of Data (2/2)

Currency Remarks
Volatility, Punctuality
Requirements on the quality of
data depend on the exact
Relevance context of the project and has
Completeness
Information to be set specifically for the
Does the detailing and demand met?
the amount of data fit project.
to the purpose?
Currency: Duration between
Quality of Data occurence of event in the real
world and the change in the
Correctness information system
Is the data syntactically Consistency
correct? Does the data
Volatility: Time period in
Is the data consistent? which a value remains valid
portray reality
(semantically correct)?
Punctuality: Moment of
delivering data from the
Reliability
source system
Transformation and
origin of data
traceable?

© Copyright MSTB-GTB
Chapter 3 Page 74 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Analysis of Data Quality

• Automated
– Detecting statistical information aided by tools that indicate a
problem and identify false data
– By using profiling-tools
• Manual/semi-automated
– Interdisciplinary teams consisting of IT-specialists and domain
experts evaluate the results of the automated analysis on the
basis of tool supported analysis
– They also create a list of business rules that can be used for
the data cleansing.

© Copyright MSTB-GTB
Chapter 3 Page 75 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Data Cleansing

• The criticality of the business processes and the cost-benefit-analysis

serve as basis for the decision which data will be cleansed.
• Automated
– Matching: various ways of noting names, addresses etc.
– Consolidation: consolidation of scattered information
– Standardization according to rules
– Comparison with external sources (e.g. post code)
– Parsing: breaking up composed input
– An approach based on rules presumes that defects occur due to
systematic process (e.g. by faulty template or faulty
transformation process). By examining patterns, the faulty data
can be identified and cleansed.
 Data Cleaning Tools
• Manual
– Many defects cannot be cleansed by automated tools. The data
has to be cleansed in an expensive manual process.
– This approach is limited to critical areas due to the high costs.

© Copyright MSTB-GTB
Chapter 3 Page 76 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Continuous Quality Assurance

• Organisational:
– Create responsibilities for data (Data Owner), who bear
responsibility for the quality of data
– Implementation of instruments for prevention (constructive)
and identification (analytical) of new quality problems

© Copyright MSTB-GTB
Chapter 3 Page 77 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 System Testing - Problems (1/3)

• Indistinct customer requirements

– If there are no requirements, any system behavior is initially
acceptable or cannot be assessed.
– The user or customer will obviously have a certain idea of
what they expect of “their” software system.
So requirements do exist.
– However, these are not to be found in written form but only
exist “in the minds” of some persons that are involved in the
project.
– The testers then have the challenging task to retrospectively
gather all this information about the desired expected
behavior.

© Copyright MSTB-GTB
Chapter 3 Page 78 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 System Testing - Problems (2/3)

• Neglected decisions
– If the testers identify the original requirements, they will realize
that in the minds of the various persons there are very
different opinions and expectations regarding one single
matter.
– No wonder, because during the project it has been neglected
to record the requirements in a written form, to agree upon
and approve them.
– Therefore the system test not only has to gather the
requirements, but also needs to seek clarification and obtain
decisions which have been omitted for many months. This
needs to be done when it is actually too late.
– This gathering of information is very time and cost intensive.
Test and acceptance of the system are delayed.

© Copyright MSTB-GTB
Chapter 3 Page 79 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 System Testing - Problems (3/3)

• Projects fail
– If requirements are not documented, the developers are
missing clear objectives. The probability that the implemented
system fulfills the implicit customer requirements is therefore
extremely low.
– Nobody can sincerely hope that even a partially suitable
system can be produced under these project conditions.
– In these kinds of projects, the system test can often only
“officially” confirm the failure of the project.

© Copyright MSTB-GTB
Chapter 3 Page 80 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 3
Chapter

Testing
Throughout
} Testing with Software Development Models
Test Levels  Acceptance Testing
Testing New Product Versions
the Software
Overview of Test Types
Life Cycle

© Copyright MSTB-GTB
Chapter 3 Page 81 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Big Picture
Test for Development Test
Development
data security levels levels W-Model
Test for
robustness
Maintenance General Iterative-incremental
test V-Model development model
Usability test

Test of compatibility Verification

Reliability test / data conversion and
Testing in software
Validation
development models
Stress test Non-Functional
test Testing Tips for Drivers and
throughout good testing stubs
Load test the lifecycle
Overview of Test
testing types Component first
Volume test test
Test levels
Top
Performance test Test of new Integration down
product versions test
Acceptance Bottom
Functional test Structure-based up
test
test System
test
Change-based Regression Big
test Re-testing Ad-hoc
testing bang

© Copyright MSTB-GTB
Chapter 3 Page 82 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Big Picture: Acceptance Testing

Testing Drivers and

throughout stubs
the lifecycle
Test
Component first
test
Test levels
Top
Integration down
test
Acceptance Bottom
testing up
System
test
Regulatory Big
Ad-hoc
acceptance test bang

Contractual Operational
acceptance test test

Test of user acceptance

Alpha testing,Beta testing

© Copyright MSTB-GTB
Chapter 3 Page 83 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Acceptance Testing - Definition
• Previous test levels are performed by the producer
or the designing project group, before the software is passed to the
customers/users.
• According to the general process model before the software is
deployed (, especially if an individual software specified for customer
has been designed,) the final test level performed is the acceptance
testing.
• The main focus here is the view and opinion of the client/user. Finding
defects is not the main focus in acceptance testing. The goal is to
establish confidence in the product and assess its fitness for the
intended use.
• Acceptance testing might be the only test that the customer can
comprehend or in which he is directly involved.
• Acceptance testing is a special form of system testing which is often
performed at the customer‘s site.

© Copyright MSTB-GTB
Chapter 3 Page 84 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Acceptance Testing – Contract and Regulation
Acceptance Testing

• On the basis of the results of the acceptance testing,

the customer decides if he/she views the ordered software system
as free from defects and feels that the design contract and the
demands set out in the contract are fulfilled.
• This can also be a more or less formal contract or project order that
exists between the operating department which placed the order
and the IT department of a company or an enterprise that carries out
the order.
• Contract acceptance testing is performed explicitly against a
contract's acceptance criteria. The test criteria are therefore the
acceptance criteria recorded in the designed contract that should be
clearly defined and unambiguous.
• Regulation acceptance testing is performed against any
regulations and standards that the system must adhere to.

© Copyright MSTB-GTB
Chapter 3 Page 85 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Acceptance Testing – Contract Testing (1/2)

• In practise software designers will, of course, arrange

in their own system testing a verification of the fulfillment of these
acceptance criteria and create corresponding test cases.
• For the acceptance testing it is sufficient to repeat the test cases
that are relevant for the acceptance according to the contract. In
that way it can be demonstrated to the customer that the
acceptance criteria in the contract have been fulfilled.
• In contrast to the system testing that takes place in the system
testing environment of the producer, acceptance testing is
performed in the acceptance environment of the customer.
• Because of the different test environments, it is possible for a test
case to fail in the acceptance test that has never caused any
problems in the system testing!

© Copyright MSTB-GTB
Chapter 3 Page 86 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Acceptance Testing – Contract Testing (2/2)

• The acceptance environment should correspond as much as

possibe to the future production environment.
• However, test execution in the production environment itself is to
be avoided, so as not to endanger the productive operation of
running software systems.
• For setting the appropriate acceptance testing or acceptance
criteria, the same methods can be used as for the determining the
test cases for the system test.
• Particularly the business cases for a typical time or accounting
period (e.g. end-of-month accounting) are to be taken into
consideration.

© Copyright MSTB-GTB
Chapter 3 Page 87 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Acceptance Testing – Operational testing

• Operational acceptance testing is mostly to performed

by the system administrators of the customer. Test performed
include:
– Testing of backup/restore procedures
– Disaster recovery
– User management
– Maintenance tasks
– Data load and migration tasks
– Periodic checks of security vulnerabilities.

© Copyright MSTB-GTB
Chapter 3 Page 88 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Acceptance Testing –
Site acceptance testing (1/2)

• Another aspect in the context of acceptance – as last

step of the validation – is site acceptance testing.
• This kind of testing is recommended when the customer and the user
of the system are different (groups of) persons, who usually have
totally different expectations regarding the new system.
• If a user group rejects the system, e.g., because it is perceived as
"complicated" it can lead to the failure of the whole system
implementation, even if the system‘s functions are well under control.
• For that reason, it is important to plan site acceptance testing for
each user group. These tests are usually organized by the customer,
who also chooses the test cases (based on business processes and
typical user scenarios).

© Copyright MSTB-GTB
Chapter 3 Page 89 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Acceptance Testing –
Site acceptance testing (2/2)

• If serious acceptance problems are detected during acceptance

testing, it is often too late for countermeasures that go beyond
cosmetic corrections (minor fixing).
• To prevent such disasters, it is reasonable to have prototypes
examined by selected representatives of the future users at early
stages of the project.

© Copyright MSTB-GTB
Chapter 3 Page 90 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Alpha and Beta Testing (1/3)

• If the ordered software is to be used in many different production

environments, it is very cost intensive or even impossible to
recreate all the possible test environments during the system
testing.
• In such cases, the software developer will organize alpha and beta
testing after the system testing.
• The objective of such testing is to recognize and, if needed,
remove any negative influences of not fully specified or unknown
production environments.

© Copyright MSTB-GTB
Chapter 3 Page 91 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Alpha and Beta Testing (2/3)

• Testing by means of representative customers

• For these tests, the developer delivers stable drafts of the software
to a selected group which is representative of the market or whose
production environment covers the various possible environments
well.
• These customers either perform test scenarios set out by the
developer or they use the draft system tentatively under realistic
conditions. Subsequently, they will report failures (or also general
comments and impressions about the new product) to the
developer who then makes amendments accordingly.
• Difference between alpha and beta testing:
– Alpha testing is performed at the developer‘s site
– Beta testing (also called field-testing) is performed at the
customer‘s site

© Copyright MSTB-GTB
Chapter 3 Page 92 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Alpha and Beta Testing (3/3)

• Alpha and beta testing must not replace in-house

system testing performed by the development organization (even if
some developers may view it like that).
• Only when the system testing has proved a software to be
sufficiently stable, can the customer be expected to test the
product in alpha and beta testing.

© Copyright MSTB-GTB
Chapter 3 Page 93 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 3
Chapter

Testing
Throughout
} Testing with Software Development Models
Test Levels  Acceptance Testing
Testing New Product Versions
the Software
Overview of Test Types
Life Cycle

© Copyright MSTB-GTB
Chapter 3 Page 94 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Big Picture
Test for Development Test
Development
data security levels levels W-Model
Test for
robustness
Maintenance General Iterative-incremental
test V-Model development model
Usability test

Test of compatibility Verification

Reliability test / data conversion and
Testing in software
Validation
development models
Stress test Non-Functional
test Testing Tips for Drivers and
throughout good testing stubs
Load test the lifecycle
Overview of Test
testing types Component first
Volume test test
Test levels
Top
Performance test Test of new Integration down
product versions test
Acceptance Bottom
Functional test Structure-based up
test
test System
test
Change-based Regression Big
test Re-testing Ad-hoc
testing bang

© Copyright MSTB-GTB
Chapter 3 Page 95 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Testing New Product Versions (Maintenance Testing)

• Until now, it has been assumed that a software development

project is completed after successful acceptance testing and
delivery of the new product. The reality is different.
• The initial delivery a software product is merely at the beginning of
its life cycle. Once deployed, the software is often in service for
years or even decades. The software product, its environment, its
data and its configuration will be corrected, modified and extended
many times during this period.
• Each time a new version of the original product is developed, it
needs to be tested.

© Copyright MSTB-GTB
Chapter 3 Page 96 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Software Maintenance (1/3)

• In contrast to »classic« industrial products, software maintenance

is not about keeping up the usability by regular maintenance or
reparing damages e.g., from wear. Software does not wear down.
• We are talking about software maintenance when a product is
adapted to changing requirements or when failures are removed
that had already been in the system.

© Copyright MSTB-GTB
Chapter 3 Page 97 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Software Maintenance (2/3)

Typical Reasons for Maintenance

Modification Migration Retirement of the Software

Planned enhancement What can be migrated? Shutdown/Retirement

changes  adaptive • Platform(s)
maintenance • Data Replacement of system
• Hardware
Change of environment  • DB-systems
adaptive maintenance • Webserver
• Authentication and
Defect Fixing  Register services
corrective maintenance • Network services
• File storage
Quality improvement  • Printing services
perfective maintenance • System surveillance and
management services
etc.

© Copyright MSTB-GTB
Chapter 3 Page 98 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Software Maintenance (3/3)

• Each software system therefore requires a certain amount of

correction and amendment after delivery.

– The fact that maintenance is definitely necessary, may not be

misused as an excuse for not doing sufficient component
testing, integration and system testing.

– “We will have to create new versions over and over again, so it
is not that serious if the testing is not that detailed and some
defects are overlooked.” This is the wrong attitude.

© Copyright MSTB-GTB
Chapter 3 Page 99 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Modification (1/3)
• Planned enhancement changes
– Apart from maintenance work caused by defects or failures there
is also modification and extension work that the project
management has planned from the beginning.
– This is part of the normal product enhancements.
– This includes, for example,
• Adjustments due to a planned modification of neighboring
system.
• Implementation of a planned functionality that could not be
delivered at time of initial deployment due to lack of time.
• Extensions required because of market changes.
• A software project is therefore in no way completed with the delivery
of the first product version.

© Copyright MSTB-GTB
Chapter 3 Page 100 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Modification (2/3)

• Defect fixing
– Defect from operating
– Emergency changes („Hot Fixes“)
• Changes of environment
– Updating operating system
– Updating database management system
– Upgrades of commercial software
– Patches of external components
– etc.
• Quality improvement
– Improving factors for quality, e.g., maintainability,
performance, usability without change of functional amount

© Copyright MSTB-GTB
Chapter 3 Page 101 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Modification (3/3)

• There is continuous enhancement, e.g., with improved product

versions released every year.
• These deliveries are conveniently synchronized with the
maintenance work so that e.g., twice a year a new version is
released: a maintenance update and a real functional update.
• Anticipating releases is crucial for successful maintenance testing.
• After each delivery the project runs through all the project phases.
• This procedure is therefore also called iterative-incremental software
development. This is the norm nowadays.
 How does the testing have to react to this?
 Do all the test levels have to completely repeat all their tests at each
release of the product (Maintenance testing on each level and for
each type of test)?
© Copyright MSTB-GTB
Chapter 3 Page 102 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Maintenance Testing - Migration and Retirement

• Maintenance testing for migration

– Includes operational tests of the new (software/hardware)
environment as well as of the changed software
– Includes conversion testing
• of the software that is used for the conversion of the data
(e.g. data from another application will be migrated into the
system being maintained)
• the (new) data that result from the conversion
• Maintenance testing for retirement of the system
– Includes testing of data migration or archiving if long data
retention periods are required.

© Copyright MSTB-GTB
Chapter 3 Page 103 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Regression Testing (1/4)

• Through maintenance and also by enhancements, some parts of the

existing software are changed or new software parts are added.
• In both cases, the modified software has to be tested again. This is
called regression testing.
• Regression testing is the repeated testing of an already tested
program, after modification.
• Objective
– to discover any defects introduced or uncovered as a result of
the changes.
 What has to be tested because of a modification? How extensive
does regression testing have to be?
– An analysis of the effects of the changes needs to be
performed to determine the extent of regression testing.

© Copyright MSTB-GTB
Chapter 3 Page 104 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Regression Testing (2/4)

• Extent of regression testing

1. Repeating all tests that detected the fixed defect (re-testing of
defects)?
2. Testing all program statements that have been fixed or
modified (testing modified functionality)?
3. Testing all program parts that have been added (testing new
functionality)?
4. The whole system (complete regression testing)?
• Both, the pure re-testing of defects (1) as well as testing only the
modified program parts (2 and 3) are not enough.
• Seemingly simple local modifications can have unexpected side
effects on other (also distant) parts of the software system.

© Copyright MSTB-GTB
Chapter 3 Page 105 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Regression Testing (3/4)

• Complete regression testing

– In addition to testing fixed defects and added functions, all
existing test cases actually need to be repeated.
– Only then would the test have the same value as the same
test that has been performed in the original version of the
program.
– Such complete regression testing would also have to be
performed when the system environment has been changed,
since this can potentially have an effect on each part of the
system.
• In practise, complete regression testing typically takes too much
time and is too expensive.

© Copyright MSTB-GTB
Chapter 3 Page 106 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Regression Testing (4/4)

• Selection of regression test cases executed

– Repetition only of those tests in the test plan which have high
priority.
– Omit certain functional variations (special cases).
– Restricting the test to only certain configurations (e.g. only
testing English product version, only test on a certain
operation system version etc.).
– Restricting the tests to certain parts of the system or certain
test levels.

© Copyright MSTB-GTB
Chapter 3 Page 107 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 3
Chapter

Testing
Throughout
} Testing in Software Development Models
Test Level  Acceptance Testing
Testing of New Product Versions
the Software
Overview of Test Types
Life Cycle

© Copyright MSTB-GTB
Chapter 3 Page 108 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Big Picture
Test for Development Test
Development
data security levels levels W-Model
Test for
robustness
Maintenance General Iterative-incremental
test V-Model development model
Usability test

Test of compatibility Verification

Reliability test / data conversion and
Testing in software
Validation
development models
Stress test Non-Functional
test Testing Tips for Drivers and
throughout good testing stubs
Load test the lifecycle
Overview of Test
testing types Component first
Volume test test
Test levels
Top
Performance test Test of new Integration down
product versions test
Acceptance Bottom
Functional test Structure-based up
test
test System
test
Change-based Regression Big
test Re-testing Ad-hoc
testing bang

© Copyright MSTB-GTB
Chapter 3 Page 109 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Test Levels and Test Types
• Test objectives vary from step to step.
• Several test types are implemented in different intensity.
• The basic test types are:
– Functional testing (test of functionality)
– Non-functional testing (test of non-functional software
characteristics)
– Structure-based testing (test of software structure/software
architecture)
– Change-oriented testing (test in connection with changes,
also known as changing-application test)
• All types can be implemented on these test levels:
– Component testing - System testing
– Integration testing - Acceptance testing

© Copyright MSTB-GTB
Chapter 3 Page 110 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Functional Testing

• Functionality describes, "what" the system should perform.

• Functionality is delivered from a system, a sub system, or from one
component.
• Test basis are requirements specification, use-cases, or functional
specification.
• Functionality can also be undocumented
(e.g. legitimate expectations of the customer).
• Functional testing proves the external visible behaviour of the
software (Black-box test design approach, refer Chapter 4).
• Application of specification-oriented test design approach, to derive
exit criteria (also known as test completion criteria) and test cases
from the functionality of the software or the system.

© Copyright MSTB-GTB
Chapter 3 Page 111 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Non-Functional Testing

• Non-functional tests prove on the basis of software and system

attributes, "how good" the system works.
• Non-functional tests are e.g.:
– Performance testing - Suitability testing
– Load testing - Maintainability testing
– Stress testing - Reliability testing
– Portability test
• Can be implemented on all test levels.
• Non-functional testing mostly proves the external visible behaviour
of the software (Black-box test design approach, refer Chapter 4).
• The basis are quality models (e.g. ISO 9126, refer Chapter 2).

© Copyright MSTB-GTB
Chapter 3 Page 112 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Structure-based Testing
• Structure-based testing (White-box test design approach, refer
Chapter 5) is based on the internal structure of a component or the
architecture of the software or the system.
• Basics are e.g.
– the control flow within components,
– the calling-hierarchy of procedures or structured menus or
– the structure of abstract models of the software (finite state
machine)
• The objective is to cover all elements of the considered structure via
testing.
• Suitable and sufficient test cases need to be designed.
• Structure oriented testing are applied in component and integration
testing, or as an addition in higher test levels (e.g. to cover menu
structures).

© Copyright MSTB-GTB
Chapter 3 Page 113 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Change-oriented Testing

• Testing in relation to modifications

(Re-testing and regression testing)
• If a failure was detected and corrected, a re-test must be conducted to
ensure that the failure was successfully eliminated (Re-testing)
• When tested programs (or program parts) were modified, it needs to
be shown that no new failures were built in or failures which were
undetected have become visible (Regression testing).
• Regression tests must be executed if the software environment was
changed.
• Re-testing and regression testing must be done several times and
must be capable of repetition.
• Regression testing comprises functional, non-functional and also
structure oriented testing (on all test levels).

© Copyright MSTB-GTB
Chapter 3 Page 114 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
✓ Summary (1/3)

• The general V-Model defines basic test levels:

component testing, integration testing, system testing and
acceptance testing and differentiates between verification and
validation.
• The component test is testing every single software component.
• The integration test checks the interaction of tested components.
• Functional and non-functional system test consider the whole
system from the user point of view.
• During the acceptance test, the client inspects the product
regarding contractual and user acceptance.
• If the system should run in many production environments, alpha-
und beta-testing provides additional possibilities to collect
experiences on pre-release versions of the product.

© Copyright MSTB-GTB
Chapter 3 Page 115 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Summay (2/3) – Comparing the Test Levels
Criteria Component Testing Integration Testing System Testing Acceptance Testing

Testing Finding defects in software Finding defects in Verification if specified Gaining confidence in the
Objectives (components), that can be tested inferfaces and in requirements (functional, system or in a certain
separately interactions between non-functional) are non-functional attribute
integrated components. fulfilled by product.

Test Basis Component specification Software and Specified for system User requirements
Detailed Design system design and requirements  System requirements
Data model  Architecture Use cases  Uses cases
Program code  Workflows Functional  Business processes
 Application cases specificaiton  Risk analysis report
Business processes
 Risk analysis report

Typical Test Isolated software components Individual components to System guidelines, Business processes of
Objects (partitions, unit, module) be integrated, sub user and operation integrated systems
 Components, programs systems and added guidelines  Operation and
Data conversion program, standard components  System configuration maintenance processes
migration program Implementation of data and configuration data  User approach
Database module base  Forms
 Infrastructure  Reports
 Interfaces  Configuration data
 System configuration
data

Testing Development environment, Monitoring of exchange of Test management tools,

Tools debugging-support, data between components GUI automated tools
static analysis tools , by testing monitors
Component testing environment

Testing Stub, driver, simulators Reuse/extension of stubs, Testing and production Testing and production
Harness drivers, simulators from environment should environment should
the component testing correspond as much © Copyright
as MSTB-GTB
correspond as much as
Chapter 3 Page 116 Software Testing Foundations Certified Tester
possible. V 2.2.0 / 2017
possible.
✓ Summary (3/3)

• Due to correction (maintenance) and planned development, a

software product gets changed and extended throughout its life
cycle.
• Each of these modified versions must be re-tested.
• What extent the regression testing has, must be established in an
individual risk assessment.
• The basic test types are functional and non-functional testing, as
well as structural and change-based testing.

© Copyright MSTB-GTB
Chapter 3 Page 117 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
??? You should now be able to answer the following questions

• Explain the particular phases of the general V-Model.

• Define the terms verification and validation.
• Justify why verification is so important, even though validation is
carried out (and vice versa).
• Characterize the typical test objects in component testing.
• Name the test objectives of integration testing.
• Which integration strategies can be differentiated?

© Copyright MSTB-GTB
Chapter 3 Page 118 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
??? Now you should be able to answer the following questions

• What are the reasons to execute tests in a separate test

infrastructure?
• Explain the testing based on requirements.
• Define load testing, performance testing, stress testing.
What are the different attributes?
• What are the differences between re-testing and regression
testing?
• Following the general V-Model, in which project phase is the test
concept developed?
• What are the objectives of re-testing and regression testing?
• What test types can be differentiated?

© Copyright MSTB-GTB
Chapter 3 Page 119 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
 Finally

© Copyright MSTB-GTB
Chapter 3 Page 120 Software Testing Foundations Certified Tester
V 2.2.0 / 2017