Department of ISE BMS Institute of Technology & Mgmt 1

Agenda
1. Integration Testing Strategies
2. Testing Components and assemblies
3. System Testing
4. Acceptance Testing
5. Regression Testing
6. Usability Testing
7. Regression Testing Selection Techniques
8. Test Case prioritization and Selective Execution
9. Levels of Testing and Integration Testing
10. Traditional view of testing levels
11. Alternative life cycle models
12. The SATM System
13. Separating Integration and System Testing
14. A Closer look at the SATM System
15. Decomposition Based
16. Call Graph Based
17. Path Based Integrations

Department of ISE BMS Institute of Technology & Mgmt 2

INTEGRATION TESTING is a level of software testing where individual units are
combined and tested as a group. The purpose of this level of testing is to expose
faults in the interaction between integrated units. Test drivers and test stubs are
used to assist in Integration Testing.
Department of ISE BMS Institute of Technology & Mgmt 3
Integration Testing Strategies

Bottom - up testing (test harness).

Top - down testing (stubs).
Modified top - down testing - test levels
independently.
Big Bang.
Sandwich testing.

Department of ISE BMS Institute of Technology & Mgmt 4

Top-Down Integration Testing
 Main program used as a test driver and stubs are substitutes
for components directly subordinate to it.
 Subordinate stubs are replaced one at a time with real
components (following the depth-first or breadth-first
approach).
 Tests are conducted as each component is integrated.
 On completion of each set of tests and other stub is
replaced with a real component.
 Regression testing may be used to ensure that new errors
not introduced.

Department of ISE BMS Institute of Technology & Mgmt 5

Bottom-Up Integration Testing
Low level components are combined in clusters
that perform a specific software function.
A driver (control program) is written to coordinate
test case input and output.
The cluster is tested.
Drivers are removed and clusters are combined
moving upward in the program structure.

Department of ISE BMS Institute of Technology & Mgmt 6

Big Bang Integration Testing

Department of ISE BMS Institute of Technology & Mgmt 7

Sandwich Testing

Department of ISE BMS Institute of Technology & Mgmt 8

  Bottom - Up Top - Down Big Bang Sandwich

Integration Early Early   Early

Time to get Late Early Late Early

working
program

Drivers Yes No Yes Yes

Stub No Yes Yes Yes

Parallelism Medium Low High Medium

Test Easy Hard Easy Medium

specification

Product Easy Hard Easy Hard

control seq.

Department of ISE BMS Institute of Technology & Mgmt 9

Department of ISE Unit-7:Fault-Based
BMS Institute of Technology & Mgmt Testing, Test Execution 10
Department of ISE Unit-7:Fault-Based
BMS Institute of Technology & Mgmt Testing, Test Execution 11
 Testing Components and assemblies
Working Definition of Component

Reusable unit of deployment and composition

◦ Deployed and integrated multiple times
◦ Integrated by different teams (usually)
 Component producer is distinct from component user
Characterized by an interface or contract
 Describes access points, parameters, and all functional and non-
functional behavior and conditions for using the component
 No other access (e.g., source code) is usually available
Often larger grain than objects or packages
◦ Example: A complete database system may be a
component

Department of ISE BMS Institute of Technology & Mgmt

Components — Related Concepts
Framework
 Skeleton or micro-architecture of an application
 May be packaged and reused as a component, with “hooks” or
“slots” in the interface contract
Design patterns
 Logical design fragments
 Frameworks often implement patterns, but patterns are not
frameworks. Frameworks are concrete, patterns are abstract
Component-based system
 A system composed primarily by assembling components,
often “Commercial off-the-shelf” (COTS) components
 Usually includes application-specific “glue code”

Department of ISE BMS Institute of Technology & Mgmt

Component Interface Contracts
Application programming interface (API) is
distinct from implementation
◦ Example: DOM interface for XML is distinct from
many possible implementations, from different
sources
Interface
includes everything that must be
known to use the component
◦ More than just method signatures, exceptions, etc
◦ May include non-functional characteristics like
performance, capacity, security
◦ May include dependence on other components

Department of ISE BMS Institute of Technology & Mgmt

Challenges in Testing Components
The component builder’s challenge:
◦ Impossible to know all the ways a component may be
used
◦ Difficult to recognize and specify all potentially
important properties and dependencies

The component user’s challenge:

◦ No visibility “inside” the component
◦ Often difficult to judge suitability for a particular use
and context

Department of ISE BMS Institute of Technology & Mgmt

Testing a Component: Producer View

First: Thorough unit and subsystem testing

◦ Includes thorough functional testing based on
application program interface (API)
◦ Rule of thumb: Reusable component requires at least
twice the effort in design, implementation, and testing
as a subsystem constructed for a single use (often
more)
Second: Thorough acceptance testing
◦ Based on scenarios of expected use
◦ Includes stress and capacity testing
 Find and document the limits of applicability

Department of ISE BMS Institute of Technology & Mgmt

Testing a Component: User View
Not primarily to find faults in the component
Major question: Is the component suitable for
this application?
◦ Primary risk is not fitting the application context:
 Unanticipated dependence or interactions with environment
 Performance or capacity limits
 Missing functionality, misunderstood API
◦ Risk high when using component for first time
Reducing risk: Trial integration early
◦ Often worthwhile to build driver to test model
scenarios, long before actual integration

Department of ISE BMS Institute of Technology & Mgmt

 Adapting and Testing a Component

Application
QuickTime™ and a
None decompressor
are needed to see this picture.

Adaptor

Component

Applications often access components through an

adaptor, which can also be used by a test driver
System Testing
SYSTEM TESTING is a level of testing that validates the complete and
fully integrated software product. The purpose of a system test is to
evaluate the end-to-end system specifications. Usually, the software is
only one element of a larger computer-based system.

Recovery testing
◦ checks system’s ability to recover from failures
Security testing
◦ verifies that system protection mechanism prevents
improper penetration or data alteration
Stress testing
◦ program is checked to see how well it deals with
abnormal resource demands
Performance testing
◦ tests the run-time performance of software 19
 Acceptance Testing
Making sure the software works correctly for
intended user in his or her normal work
environment.
Alpha test
◦ version of the complete software is tested by customer
under the supervision of the developer at the developer’s
site
Beta test
◦ version of the complete software is tested by customer at
his or her own site without the developer being present

Department of ISE BMS Institute of Technology & Mgmt 20

 Acceptance Testing Approaches
Benchmark Test: BENCHMARK TESTING measures a
repeatable set of quantifiable results that serves as a point of reference against
which products/services can be compared. The purpose of benchmark
testing results is to compare the present and future software releases with their
respective benchmarks.
Pilot Testing: PILOT TESTING is defined as a type of
Software Testing that verifies a component of the system or the entire system
under a real-time operating condition. The purpose of the Pilot Test is to
evaluate the feasibility, time, cost, risk, and performance of a research project.
Parallel Testing: Parallel testing means testing multiple
applications or subcomponents of one application concurrently to reduce
the test time. Parallel tests consist of two or more parts (projects or project
suites) that check different parts or functional characteristics of an application.

Department of ISE BMS Institute of Technology & Mgmt 21

 Regression Testing
Check for defects propagated to other modules by
changes made to existing program
◦ Representative sample of existing test cases is used to
exercise all software functions.
◦ Additional test cases focusing software functions likely to
be affected by the change.
◦ Tests cases that focus on the changed software
components.

Department of ISE BMS Institute of Technology & Mgmt 22

 Usability Testing
Usability testing is a technique used in user-centered interaction
design to evaluate a product by testing it on users. This can be seen as
an irreplaceable usability practice, since it gives direct input on how
real users use the system. This is in contrast with usability
inspection methods where experts use different methods to evaluate a
user interface without involving users.

Usability testing focuses on measuring a human-made product's

capacity to meet its intended purpose. Examples of products that
commonly benefit from usability testing are food, consumer
products, web sites or web applications, computer interfaces,
documents, and devices.

Usability testing measures the usability, or ease of use, of a specific

object or set of objects, whereas general human–computer
interaction studies attempt to formulate universal principles.

Department of ISE BMS Institute of Technology & Mgmt 23

Regression Testing Selection Techniques

Regression testing is a necessary and expensive maintenance task

performed on modified programs to ensure that the changes have not
adversely effected the unchanged code of the program.
One strategy is to rerun the entire test suit on the changed program.
This is a heavy resource and time consuming process.
A solution to this is: Regression test selection techniques: selects a subset
of test cases, thus reducing the time and resources required.

Selection Techniques:

Department of ISE BMS Institute of Technology & Mgmt 24

Most of the selection techniques are based on the information about the
code of the program and the modified version. Some however are based
on the program specifications.

Following are some of the code based techniques, which are used for this
study

Selection technique algorithms used for study

Safe: selects all the test cases that cover/execute the changed
methods at least once.
Minimization: selects a minimum set of test cases that execute all the
changed methods.
Random25: selects randomly 25% of the total test cases. Random50:
selects randomly 50% of the total test cases.
Random75: selects randomly 75% of the total test cases.

Department of ISE BMS Institute of Technology & Mgmt 25

 Test Case prioritization and Selective Execution

Regression testing activities such as test case selection and test case
prioritization are ordinarily based on the criteria which focused around
code coverage, code modifications and test execution costs. The
approach mainly based on the multiple criteria of code coverage which
performs efficient selection of test case. The method mainly aims to
maximize the coverage size by executing the test cases effectively

Department of ISE BMS Institute of Technology & Mgmt 26

The goal of regression testing is to ensure that changes to the system have
not introduced errors. One approach is to rerun all the test cases in the
existing test suite and check for new faults. But rerunning the entire test suite
is often too costly.
To make the execution of test cases more cost effective, two major
approaches are made use of. They are the Regression Test Selection (RTS)
and Regression Test Prioritization (RTP) techniques.

Many RTS and RTP techniques consider a single criterion for optimization of
test cases. But, the use of a single criterion severely limits the ability of the
resulting regression test suite to locate faults. Harman et al., induce the need
of multiple criteria and provides a list of criteria with different weights.

The two criteria for selection are code coverage and sum coverage of the
program. Code coverage assumes that there exist test cases that effectively
cover the changed area of code of the software. Sum coverage is a new
approach that maximizes the minimum sum of coverage across all software
elements.
The selected test cases are prioritized using a greedy algorithm to maximize
the minimum sum of coverage across all software elements.

Department of ISE BMS Institute of Technology & Mgmt 27

 Levels of Testing and Integration Testing

Traditional View of Testing Levels

The traditional model of software development is the Waterfall model,

which is drawn as a V in. In this view, information produced in one of the
development phases constitutes the basis for test case identification at
that level.

Nothing controversial here: we certainly would hope that system test

cases are somehow correlated with the requirements specification, and
that unit test cases are derived from the detailed design of the unit. Two
observations: there is a clear presumption of functional testing here, and
there is an implied “bottom-up” testing order.

Department of ISE BMS Institute of Technology & Mgmt 28

Alternative Life Cycle Models
Since the early 1980s, practitioners have devised alternatives in response
to shortcomings of the traditional waterfall model of software development
Common to all of these alternatives is the shift away from the functional
decomposition to an emphasis on composition. Decomposition is a perfect
fit both to the top-down progression of the waterfall model and to the
bottom-up testing order.
One of the major weaknesses of waterfall development cited by is the
over-reliance on this whole paradigm. Functional decomposition can only
be well done when the system is completely understood, and it promotes
analysis to the near exclusion of synthesis. The result is a very long
separation between requirements specification and a completed system,
and during this interval, there is no opportunity for feedback from the
customer. Composition, on the other hand, is closer the way people work:
start with something known and understood, then add to it gradually, and
maybe remove undesired portions.
There is a very nice analogy with positive and negative sculpture. In
negative sculpture, work proceeds by removing unwanted material, as in
the mathematician’s view of sculpting Michelangelo’s David: start with a
piece of marble, and simply chip away all non-David. Positive sculpture is
often done with a medium like wax.
Department of ISE BMS Institute of Technology & Mgmt 29
The central shape is approximated, and then wax is either added or
removed until the desired shape is attained. Think about the
consequences of a mistake: with negative sculpture, the whole work
must be thrown away, and restarted. With positive sculpture, the
erroneous part is simply removed and replaced. The centrality of
composition in the alternative models has a major implication for
integration testing.

Waterfall Spin-offs

There are three mainline derivatives of the waterfall model: incremental

development, evolutionary development, and the Spiral model [Boehm
88]. Each of these involves a series of increments or builds, Within a
build, the normal waterfall phases from detailed design through testing
occur, with one important difference: system testing is split into two
steps, regression and progression testing

Department of ISE BMS Institute of Technology & Mgmt 30

 An Object-Oriented Life Cycle Model
When software is developed with an object orientation, none of our life cycle models
fit very well. The main reasons: the object orientation is highly compositional in
nature, and there is dense interaction among the construction phases of object-
oriented analysis, object-oriented design, and object-oriented programming. We
could show this with pronounced feedback loops among waterfall phases, but the
fountain model [Henderson-Sellers 90] is a much more appropriate metaphor. In the
fountain model, the foundation is the requirements analysis of real world systems

Department of ISE BMS Institute of Technology & Mgmt 31

 Formulations of the SATM System
The Simple Automatic Teller Machine (SATM) system. there are function
buttons B1, B2, and B3, a digit keypad with a cancel key, slots for printer
receipts and ATM cards, and doors for deposits and cash withdrawals. The
SATM system is described here in two ways: with a structured analysis
approach, and with an object-oriented approach. These descriptions are not
complete, but they contain detail sufficient to illustrate the testing techniques
under discussion.

SATM with Structured Analysis

The structured analysis approach to requirements specification is the most widely
used method in the world. It enjoys extensive CASE tool support as well as
commercial training, and is described in numerous texts. The technique is based
on three complementary models: function, data, and control. Here we use data
flow diagrams for the functional models, entity/relationship models for data, and
finite state machine models for the control aspect of the SATM system. The
functional and data models were drawn with the Deft CASE tool from Sybase Inc.
That tool identifies external devices (such as the terminal doors) with lower case
letters, and elements of the functional decomposition with numbers (such as 1.5
for the Validate Card function).
Department of ISE BMS Institute of Technology & Mgmt 32
The open and filled arrowheads on flow arrows signify whether the flow
item is simple or compound. The portions of the SATM system shown
here pertain generally to the personal identification number (PIN)
verification portion of the system.

Department of ISE Unit-7:Fault-Based

BMS Institute of Technology & Mgmt Testing, Test Execution 33
 The Deft CASE tool distinguishes between simple and compound
flows, where compound flows may be decomposed into other flows,
which may themselves be compound. The graphic appearance of this
choice is that simple flows have filled arrowheads, while compound
flows have open arrowheads. As an example, the compound flow
“screen” has the following decomposition

The SATM Terminal

Context Diagram of the SATM

System

Department of ISE BMS Institute of Technology & Mgmt 34

The Structured Analysis approach
models shown here are not complete
but they contain sufficient details to
illustrate the testing techniques.

The Structured analysis approach to

requirements specifications is still
widely used.

It Enjoys extensive CASE tool support.

The Techniques used are based on

three complementary models: function,
data and control.

Here we use dataflow diagrams for

functional model, the entity relationship
model for data and finite state machine Level-1 Dataflow Diagram of
models for the control aspects of SATM the SATM System
Department of ISE BMS Institute of Technology & Mgmt 35
The Different Screens are shown along with
E-R diagram of the major data structure in the SATM
-Customer
-Accounts
-Terminals E-R Model of the SATM
-Transactions. System

Department of ISE BMS Institute of Technology & Mgmt 36

The Upper level finite state
machine which divides the
system into states that
correspond to stages of
customer usage.
Other Choices are possible
for instance, we might
choose states to be screens
displayed.

Finite state machines can be

hierarchically decomposed in
much the same way as Upper Level SATM Finite
dataflow diagrams can. State Machine

Department of ISE BMS Institute of Technology & Mgmt 37

The Decomposition of the Await PIN
state. Here the state transitions are
caused either by events at the ATM
terminal or by data conditions.

When a transition occurs a

corresponding action may also occur.

We choose to use screen displays as

such actions, this choice will prove to
be very handy when we develop
system-level test cases. PIN Entry finite State
Machine
The function, data and control
models are the basis for design
activities in the waterfall model

Department of ISE BMS Institute of Technology & Mgmt 38

Decomposition Based Integration in
Software Testing
Decomposing the portion of a system model that is of interest for a
given testing requirement, into a tree of subsystems by exploiting
information on model variable dependency.
The technique generates tests for the whole system model by
merging tests built from those subsystems.

Department of ISE Unit-7:Fault-Based

BMS Institute of Technology & Mgmt Testing, Test Execution 39
 A Decomposition tree for
the SATM System

The Pseudocode shown here is for SATM system and it is decomposed

into tree structure for different functionality - Functional Decomposition

Department of ISE BMS Institute of Technology & Mgmt 40

Functional Decomposition

Department of ISE BMS Institute of Technology & Mgmt 41

 SATM functional decomposition tree

SATM Units and Abbreviated Names

Department of ISE BMS Institute of Technology & Mgmt 42
• The decomposition tree is the basis of integration testing. It is important
to remember that such a decomposition is primarily a packaging
partition of the system.

• As software design moves into more detail, the added information. The
functional decomposition tree into a unit calling graph.

• The Unit calling graph is the directed graph in which nodes are program
units and edges runs from node A to node B.

• Drawing a call graphs do not scale up well.

• Both the drawings and the adjacency matrix provide insights to the
tester.

• Node with a higher degree will be important to integration testing and

paths from the main program(node-1) to the sink nodes can be used to
identify contents of builds for an incremental development.

Department of ISE BMS Institute of Technology & Mgmt 43

 Adjacency Matrix
In graph theory and computer science, an adjacency matrix is a
square matrix used to represent a finite graph. The elements of
the matrix indicate whether pairs of vertices are adjacent or not in the
graph. In the special case of a finite simple graph, the adjacency
matrix is a (0,1)-matrix with zeros on its diagonal.

Department of ISE Unit-7:Fault-Based

BMS Institute of Technology & Mgmt Testing, Test Execution 44
Adjacency Matrix for the SATM Call Graph

Department of ISE BMS Institute of Technology & Mgmt 45

SATM Call graph is shown in the
graph. Some of the hierarchy is
obscured to reduce the confusion
in the drawing.

SATM Call Graph

Department of ISE BMS Institute of Technology & Mgmt 46

 Top-Down Integration
Department of ISE BMS Institute of Technology & Mgmt 47
At the uppermost level, we would have stubs for the four
components in the first level decomposition.

There would be four integration sessions, in each one component

would be actual code and other three would be stubs.

Top-down integration follows a breadth-first traversal of the

functional decomposition tree.

Department of ISE BMS Institute of Technology & Mgmt 48

 Bottom Up Integration

Department of ISE BMS Institute of Technology & Mgmt 49

Bottom-up integration is a “mirror image” to the top-down order,
with the difference that stubs are replaced by driver modules that
emulate units at the next level up in the tree.

In bottom-up integration, we start with the leaves of the

decomposition tree (units like ControlDoor and DispenseCash), and
test them with specially coded drivers.
There is probably less throw-away code in drivers than there is in
stubs. Recall we had one stub for each child node in the
decomposition tree.
Most systems have a fairly high fan-out near at the leaves, so in the
bottom-up integration order, we won’t have as many drivers. This is
partially offset by the fact that the driver modules will be more
complicated

Department of ISE BMS Institute of Technology & Mgmt 50

 Bottom-up Integration

Department of ISE BMS Institute of Technology & Mgmt 51

The idea behind pair-wise integration is to eliminate the stub/driver
development effort. Rather than develop tubs and/or drivers, why
not use the actual code? At first, this sounds like big bang
integration, but we restrict a session to just a pair of units in the call
graph. The end result is that we have one integration test session
for each edge in the call graph

Pairwise Integration
Department of ISE BMS Institute of Technology & Mgmt 52
We can let the mathematics carry us still further by borrowing the notion of a
“neighborhood” from topology. (This isn’t too much of a stretch - graph theory
is a branch of topology.) We (informally) define the neighborhood of a node in
a graph to be the set of nodes that are one edge away from the given node.
In a directed graph, this means all the immediate predecessor nodes and all
the immediate successor nodes (notice that these correspond to the set of
stubs and drivers of the node).

Neighbourhoods Integration
Department of ISE BMS Institute of Technology & Mgmt 53
The eleven neighborhoods for the SATM example (based on the
call graph in Figure 4.2) are given in Table 3.

Department of ISE BMS Institute of Technology & Mgmt 54

 Path Integration

Path Integration testing is defined when a unit test executes, some path
of source statements is traversed.

Suppose that there is a call to another unit along such a path: at that
point, control is passed from the calling unit to the called unit, where
some other path of source statements is traversed.

Department of ISE BMS Institute of Technology & Mgmt 55

MM-Path
An MM-Path in object-oriented software is a sequence of method
executions linked by messages. P-path is Procedure-path , means direct
call of public method. MM-path is Method-message path , means all
method, which were called.

MM-Path Graph

Department of ISE BMS Institute of Technology & Mgmt 56

There are two possibilities: abandon the single entry, single exit precept and
treat such calls as an exit followed by an entry, or “suppress” the call
statement because control eventually returns to the calling unit anyway.

The suppression choice works well for unit testing, but it is antithetical to
integration testing.

MM-Path across three units

Department of ISE BMS Institute of Technology & Mgmt 57

The first guideline for MM-Paths: points of quiescence are “natural”
endpoints for an MM-Path. Our second guideline also serves to
distinguish integration from system testing.

Our second guideline: atomic system functions are an upper limit for MM-
Paths: we don’t want MMPaths to cross ASF boundaries. This means
that ASFs represent the seam between integration and system testing.
They are the largest item to be tested by integration testing, and the
smallest item for system testing. We can test an ASF at both levels.
Again, the digit entry ASF is a good example.

During system testing, the port input event is a physical key press that is
detected by KeySensor and sent to GetPIN as a string variable. (Notice
that KeySensor performs the physical to logical transition.) GetPIN
determines whether a digit key or the cancel key was pressed, and
responds accordingly.

An Atomic System Function ( ASF): is an action that is observable

at the system level in terms of port input and output events.

Department of ISE BMS Institute of Technology & Mgmt 58

 MM-Path graph derived from
previous MM-Path

The ASF terminates with either screen 2 or 4 being displayed.

Rather than require system keystrokes and visible screen displays,
we could use a driver to provide these, and test the digit entry ASF
via integration testing. We can see this using our continuing
example.

Department of ISE BMS Institute of Technology & Mgmt 59

 Conclusion
In a nut shell we have seen a brief Integration Testing Strategies,
Testing Components and assemblies, System Testing, Acceptance Testing,
Regression Testing, Usability Testing, Regression Testing Selection Techniques,
Test Case prioritization and Selective Execution, Levels of Testing and Integration
Testing, Traditional view of testing levels, Alternative life cycle models, The
SATM System, Separating Integration and System Testing, A Closer look at the
SATM System, Decomposition Based, Call Graph Based and Path Based
Integrations.

Department of ISE BMS Institute of Technology & Mgmt 60

VTU Questions on Module-5
1. Explain traditional view of testing levels
2. Explain alternative life cycle models
3. With a neat diagram explain waterfall spin-offs
4. With a neat diagram explain specification-based life cycle models
5. Explain the SATM system with a neat diagram
6. Explain separating integration and system testing
7. Write short notes on structural insights
8 .Write short notes on behavioral insights
9. Explain Decomposition-Based integration with a neat diagram
10. Write short notes on
a) Top-Down Integration
b) Bottom-Up Integration
c) Sandwich integration
d) Call graph-based integration
e) Pair wise integration
f) Neighborhood integration
11. Explain the comparison of integration testing strategies

Department of ISE Unit-7:Fault-Based

BMS Institute of Technology & Mgmt Testing, Test Execution 61