3.

1 INTEGRATION TESTING (Type of testing and Phase of testing)

 A system is made up of multiple components or modules that can comprise hardware and
software.
 Integration is defined as the set of interactions among components.
 Integration testing is testing the interaction between the modules and interaction with
other systems externally.
The need for an Integration Testing is to make sure that your components satisfy the following
requirements: Functional; Performance; Reliability.

Approaches/Methodologies/Strategies of Integration Testing

Software Engineering defines variety of strategies to execute Integration testing, viz.
  Big Bang Approach
  Incremental Approach: which is further divided into the following:
 Top Down Approach
 Bottom Up Approach
 Sandwich Approach - Combination of Top Down and Bottom Up
Below are the different strategies, the way they are executed and their limitations as well
advantages.
Big Bang Approach
Here all components are integrated together at once and then tested.
Advantages
 Convenient for small systems.
Disadvantages
 Fault Localization is difficult.
 Given the sheer number of interfaces that need to be tested in this approach, some
interfaces link to be tested could be missed easily.
 Since the Integration testing can commence only after "all" the modules are designed, the
testing team will have less time for execution in the testing phase.
 Since all modules are tested at once, high-risk critical modules are not isolated and tested
on priority. Peripheral modules which deal with user interfaces are also not isolated and
tested on priority.
Incremental Approach
In this approach, testing is done by joining two or more modules that are logically related. Then
the other related modules are added and tested for the proper functioning. The process continues
until all of the modules are joined and tested successfully.
Incremental Approach, in turn, is carried out by two different Methods:
 Bottom Up
 Top Down
Incremental Approach is carried out by using dummy programs called Stubs and Drivers. Stubs
and Drivers do not implement the entire programming logic of the software module but just
simulate data communication with the calling module.
What is Stub and Driver?
Stub: Is called by the Module under Test.
Driver: Calls the Module to be tested.

Integration testing as a type of testing (Integration testing means testing of interfaces).

 Internal interfaces are those that provide communication across two modules within a
project or product, internal to the product, and not exposed to the customer or external
developers.
 External interfaces are those that are visible outside the product to third party developers
and solution providers. A method of achieving interfaces is by providing Application
Programming Interfaces (API).

Not all interactions between the modules are known and explained through interfaces. Explicit
interfaces are documented interfaces. Implicit interfaces are those which are known internally to
the software engineers but are not documents In situations where architecture or design
documents do not clear by explain all interfaces among components, some additional test cases
are generated and included with others and this approach is termed as gray box testing.

The order in which the interfaces are tested are categorized as follows:

 Bottom-up integration: The piecing together of systems to give rise to grander systems,
thus making the original systems sub-systems of the emergent system.

 Top-down integration: The breaking down of a system to gain insight into its
compositional sub-systems.

 Bi-directional / Sandwich integration

Top-Down Integration

Assume a new product where components become available one after another in the order of
component numbers. The integration starts with testing the interface between C1 and C2. All
interfaces shown in figure covering all the arrows have to be tested together. The tested order is
given in table.

When one or two components are added to the product in each increment, the integration testing
method pertains to only to those new interfaces that are added only. If an addition has an impact
on the functionality of component 5, then integration testing for the new release needs to include
only the steps, 4, 7 and 9. To optimize the number of steps, steps 6 and 7, steps 8 and 9 can be
combined and executed in single step.

If a component at a higher level requires a modification every time a module gets added to the
bottom, then for each component addition integration testing needs to be repeated starting from
step 1. The orders in which the interfaces are tested may follow a depth first approach or breadth
first approach.

Advantages of Top-down Testing

Drivers do not have to be written when top down testing is used.

It provides early working module of the program and so design defects can be found and
corrected early.

Disadvantages of Top-down Testing

Stubs have to be written with utmost care as they will simulate setting of output parameters.
It is difficult to have other people or third parties to perform this testing, mostly developers
will have to spend time on this.
Bottom-up Integration

In this approach, the components for a new product development become available in reverse
order, start from the bottom.
Double arrow denotes both the logical flow of components (top to bottom) and
integration approach (bottom to up). That means the logical flow of the product can be
different from the integration path. The tested order is given in table.

Advantages of Bottom-up Testing

Behavior of the interaction points is crystal clear, as components are added in the controlled
manner and tested repetitively.
Appropriate for applications where bottom up design methodology is used.

Disadvantages of Bottom-up Testing

Writing and maintaining test drivers is more difficult than writing stubs.
This approach is not suitable for the software development using top-down approach.

Bi-directional Integration

It is a combination of the above two approaches. The individual components (1,2,3,4,5) are
tested separately. Then the bi-directional integration is performed with the use of stubs and
drivers. After the functionality of these integrated components is tested, the stubs and drivers are
discarded. Once components 6,7,8 are available, this method then focuses only on those
components, as these are the components that are new and need focus. This approach is also
called sandwich integration. The tested order is given in table
  Combines top-down strategy with bottom-up strategy
 The system is viewed as having three layers:
A target layer in the middle
A layer above the target
A layer below the target
Testing converges at the target layer
 How do you select the target layer if there are more than 3 layers?
Heuristic: Try to minimize the number of stubs and drivers

Steps in Integration Testing (Summary)

1. Based on the integration strategy, select a component to be tested. Unit test all the classes
in the component.
2. Put selected component together; do any preliminary fix-up necessary to make the
integration test operational (drivers, stubs).
3. Do functional testing: Define test cases that exercise all uses cases with the selected
component.
4. Do structural testing: Define test cases that exercise the selected component.
5. Execute performance tests.
 6. Keep records of the test cases and testing activities.
7. Repeat steps 1 to 7 until the full system is tested.

The primary goal of integration testing is to identify errors in the (current) component
configuration.

3.1.1 DESIGNING INTEGRATION TESTS

Integration tests for procedural software can be designed using a black or white box approach.
The tester needs to insure the parameters are of the correct type and in the correct order. The
tester must also insure that once the parameters are passed to a routine they are used correctly.
Testers must insure that test cases are designed so that all modules in the structure chart are
called at least once, and all called modules are called by every caller.

Coverage requirements for the internal logic of each of the integrated units should be achieved
during unit tests. When units are integrated and subsystems are to be tested as a whole, new tests
will have to be designed to cover their functionality and adherence to performance and other
requirements.

Sources for development of black box or functional tests at the integration level are:

The requirements documents and

The user manual.

Testers need to work with requirements analysts to insure that the requirements are:

estable,
Accurate and
Complete.

Black box tests should be developed to insure proper functionally and ability to handle
subsystem stress. For example, in a transaction-based subsystem the testers want to determine
the limits in number of transactions that can be handled.

Integration testing of clusters of classes also involves building test harnesses which in this case
are special classes of objects built especially for testing. Whereas,

in class testing we evaluated intra-class method interactions,

at the cluster level we test interclass method interaction as well.

Unlike procedural-oriented systems, integration for object-oriented systems usually does not
occur one unit at a time. A group of cooperating classes is selected for test as a cluster.

If developers have used the Coad and Yourdon‘s approach, then a subject layer could be used to
represent a cluster.
Jorgenson et al. have reported on a notation for a cluster that helps to formalize object oriented
integration.

A method-message path is described as a sequence of method executions linked by messages. An

atomic system function is an input port event (start event) followed by a set of method messages
paths and terminated by an output port event (system response). Murphy et al. define clusters as
classes that are closely coupled and work together to provide a unified behavior.
Some examples of clusters are:
Groups of classes that produce a report, or monitor and control a device.

Integration testing as a phase of testing

Integration testing as a phase involves different activities and different types of testing have to be
done in that phase. This is a testing phase that should ensure completeness and coverage of
testing for functionality. To achieve this, the focus should not only be on planned test case
execution but also on unplanned testing, which is termed as ―ad hoc testing‖. This approach
helps in locating some problems which are difficult to find by test teams but also difficult to
imagine in the first place.

3.1.2 INTEGRATION TEST PLANNING

Integration test must be planned. Planning can begin when high-level design is complete so that
the system architecture is defined. Other documents relevant to integration test planning are:

the requirements document,

the user manual and
usage scenarios.

These documents contain

structure charts,
state charts,
data dictionaries,
cross-reference tables,
module interface descriptions,
data flow descriptions,
messages and event descriptions

The strategy for integration should be defined. For procedural-oriented system the order of
integration of the units should be defined. Consider the fact that the testing objectives are to
assemble components into subsystems and to demonstrate that the subsystem functions properly
with the integration test cases. For object-oriented systems a working definition of a cluster or
similar construct must be described, and relevant test cases must be specified.

A detailed description of a Cluster Test Plan includes the following items:

 clusters this cluster is dependent on
a natural language description of the functionality of the cluster to be tested
list of classes in the cluster
a set of cluster test cases

One of the goals of integration test is to build working subsystems, and then combine these into
the system as a whole. When planning for integration test the planner selects subsystems to build
based upon the requirements and user needs. Developers may want to show clients that certain
key subsystems have been assembled and are minimally functional.

3.1.3 SCENARIO TESTING

When the functionality of different components are combined and tested together for a sequence
of related operations, they are called scenarios. Scenario testing is a planned activity to explore
different usage patterns and combine them into test case called scenario test cases.

A set of realistic user activities that are used for evaluation the product. Methods to evolve
scenarios are:

System scenarios
Role based scenarios

System scenarios

The set of activities used for scenario testing covers several components in the system. The
various approaches are:
1. Story line: Develop a story line that combines various activities of the product that may be
executed by an end user. Eg. User enters his office, logs into system, checks mail, responds to
mail, compiles programs, performs unit testing etc.
2. Life cycle / State transition: Consider an object, derive the different transitions /
modifications that happen to the object, and derive scenarios to cover them. Eg. Account
open, deposit money, perform withdrawal, calculate interest etc. Different transformations
applied to the object “money” becomes different scenarios.
3. Deployment / Implementation stories from customer: Develop a scenario from a known
customer deployment / implementation details and create a set of activities by various users in
that implementation.
4. Battle ground : Create some scenarios to justify that “the product works” and some scenarios
to “try and break the system” to justify “the product does not work”.

Any activity in the scenario is always a continuation of the previous activity, and depends on or
is impacted by the results of previous activities. Considering only one aspect would make
scenarios ineffective. A right mix of scenarios using the various approaches explained is very
critical for the effectiveness of scenario testing.
Coverage is always a big question with respect to functionality in scenario testing. By using a
simple technique, some comfort feeling can be generated on the coverage of activities by
scenario testing.

End-user activity Frequency Priority Applicable environment No. of items

covered
Login to application High High W2000, 2003, XP 10
Create an object High Mediu W2000, XP 7
m
Modify parameters Medium Mediu W2000, XP 5
m
List object parameter Low Mediu W2000, 2003, XP 3
m
Compose email Medium Mediu W2000, XP 6
m
Attach files Low Low W2000, XP 2
Send composed mail High High W2000, XP 10

It is clear that important activities have been very well covered by set of scenarios in the system
scenario test.

Role based / Use case scenarios

Use case scenario is a stepwise procedure on how a user intends to use a system, with different
user roles and associated parameters. It can include stories, pictures and deployment details. A
use case can involve several roles or class of users who typically perform different activities
based on the role. Use can scenarios term the users with different roles as actors. What the
product should do for a particular activity is termed as system behavior. Users with a specific
role to interact between the actors and the system are called agents.

Eg. Cash withdrawal from a bank

Customer fill up a cheque
Gives it to an official
Official verifies the balance
Gives required cash

Customer – actor, clerk – agent, system response – computer gives balance in account.

This way of describing different roles in test cases helps in testing the product without getting
into the details of the product.

Actor : need concerned only about getting cash. Not concerned about what official is
doing and what command he uses to interact with computer
Agent : not concerned about the logic of how computer works. Concerned about
whether he will get money or not.
Testers using the use case model, with one person testing the actions and other person testing the
system response, complement each other‘s testing as well as testing the business and the
implementation aspect of the product at the same time.

In a completely automated system involving the customer and the system, use cases can be
written without considering the agent portion.

Actor System response

Users likes to withdraw cash and inserts card in ATM Request for password or PIN
User fills password or PIN Validate the password or PIN
Give menu for process
User selects account type Ask user for amount to withdraw
User fills the amount of case required Check availability of funds
Update account balance
Prepare receipt
Dispense cash
Retrieve cash from ATM Print receipt

This way of documenting a scenario and testing makes it simple and also makes it realistic for
customer usage.

Example 1: Test Scenario for e-Commerce Application

For an e-Commerce Application, a few test scenarios would be

Test Scenario 1: Check the Login Functionality

In order to help you understand the difference Test Scenario and Test Cases, specific test cases
for this Test Scenario would be

1. Check system behavior when valid email id and password is entered.

2. Check system behavior when invalid email id and valid password is entered.
3. Check system behavior when valid email id and invalid password is entered.
4. Check system behavior when invalid email id and invalid password is entered.
5. Check system behavior when email id and password are left blank and Sign in entered.
6. Check Forgot your password is working as expected
7. Check system behavior when valid/invalid phone number and password is entered.
8. Check system behavior when "Keep me signed" is checked

As evident, Test Cases are more specific.

Test Scenario 2: Check the Search Functionality

Test Scenario 3: Check the Product Description Page

Test Scenario 4: Check the Payments Functionality

Test Scenario 5: Check the Order History

Apart from these 5 scenarios here is the list of all other scenarios

 Check Home Page behavior for returning customers

 Check Category/Product Pages
 Check Customer Service/Contact Pages
 Check Daily Deals pages

3.1.4 DEFECT BASH ELIMINATION

Defect bash is an ad hoc testing where people performing different roles in an organization
test the product together at the same time. What is to be tested is left to an individual‘s
decision and creativity. They can try some operations which are beyond the product
specifications.

Advantages

Enabling people ―cross boundaries and test beyond assigned areas‖.

Different people performing different roles together in the organization – “Testing is
not for tester’s alone”.
Letting everyone to use the product before delivery.
 Bringing fresh pairs of eyes to uncover new defects – “Fresh eyes have less bias”.
Brining people of different levels understanding to test the product together randomly –
“users of software are not same”.
Let testing don‘t wait for lack of / time taken for documentation – “Does testing wait
till all documentation is done”.
Even though it is said that defect bash is an ad hoc testing, not all activities of defect bash are
unplanned. All the activities are planned activities, except for what to be tested. The involved
steps are:

Choosing the frequency and duration of defect bash

Frequent defect bash incur low return on investment.

Too few defect bash may not meet the objective of finding all defects.
Duration optimization big saving.
Small duration amount of testing that is done may not meet the objective.

Selecting the right product build

Regression tested build ideal as all new features and defect fixes would have
been already tested.
Intermediate build (code functionality is evolving) / Untested build make the
purpose and outcome of a defect bash ineffective.

Communicating the objective of defect bash

The objective should be,

 To find a large number of uncovered defects
 To find out system requirements (cpu, memory, disk etc)
 To find the non-reproducible or random defects
 Defects that a test engineer would find easily should not be the objective.

Setting up and monitoring the lab

Finding the right configuration, resources (hardware, software, set of people) should be the plan
before starting defect bash. The majority of defect bash fail due to inadequate hardware, wrong
software configurations and perceptions related to performance and scalability of the software.

 The defects that are in the product, as reported by the users functional defects
 The defects that are revealed while monitoring the resources (memory leak, long
turnaround time, missed requests, high impact etc.) non-functional defects.
 Defect bash test is a unique testing method which can bring out both these defects.

Taking actions and fixing issues

Many defects could be duplicate defects. It is difficult to solve all the problems if they are taken
one by one and fixed in code. The defects need to be classified into issues at a higher level, so
that a similar outcome can be avoided in future defect bashed. There could be one defect
associated with an issue and there could be several defects that can be called as an issue.

Optimizing the effort involved in defect bash

An approach to reduce the defect bash effort is to conduct “micro level” defect bashes before
conducting one on a large scale.

To prevent component level defect emerging during integration testing, a micro level defect bash
can also be done to unearth feature level defects before integration testing.

Here, the defect bash can be classified into:

 Feature / component defect bash
 Integration defect bash
 Product defect bash

Example.
3 product defect bashed conducted in 2 hours with 100 people.
Total effort involved is 3 * 2 * 100 = 600 person hours.
If feature / component test team and integration test team, that has 100 people
each, conduct 2 rounds of micro level bashes, which can find out 1/3 of defects,
then effort saving is 20%.

Total effort involved in 2 rounds of product bashes = 400 man hours

Effort involved in 2 rounds of feature bash = 2 * 2 * 10 – 40
Effort involved in 2 rounds of integration bash = 2 * 2 * 10 – 40

Effort saved = 600 – (A+B+C) = 600 – 480 = 120 person hours, or 20%.

3.2 SYSTEM TESTING

The testing conducted on the complete integrated products and solutions to evaluate system
compliance with specified requirements on functional and non-functional aspects is called
system testing.

A system is complete set of integrated components that together deliver product functionality
and features. System testing helps in uncovering the defects that may not be directly attributable
to a module or an interface. System testing brings out issues that are fundamental to design,
architecture and code of whole product.
Functional testing Vs non-functional testing

Testing aspects Functional testing Non-functional testing

Involves Product features and Quality factors
functionality
Tests Product behavior Behavior and experience
Result conclusion Simple steps written to check Huge data collected and
expected results analyzed
Results varies due to Product implementation Product implementation,
resources and configurations
Testing focus Defect detection Qualification of product
Knowledge required Product and domain Product, domain, design,
architecture, statistical skills
Failures normally due to Code Architecture, design and code
Testing phase Unit, component, integration, system
system

ACCEPTANCE TESTING

Acceptance testing is a phase after system testing that is normally done by the customers or
representative of the customer. Acceptance test is performed by the client, not by the developer.
Acceptance test cases are normally small in number and are not written with the intention of
finding defects. However, the purpose of this test is to enable customers and users to determine if
the system built really meets their needs and expectations. Acceptance testing is done by the
customer or by the representative of the customer to check whether the product is ready for use
in the real-life environment.

Acceptance tests are written to execute near real-life scenarios.

Acceptance Criteria:

1. Product acceptance
2. Procedure acceptance
3. Service level agreements

Selection test cases for Acceptance Testing

 End-to-end functionality verification 
 Domain tests 
 User scenario tests 
 Basic sanity tests 
 New functionality 
 A few non-functionality 
 Tests pertaining to legal obligations and service level agreements
 Acceptance test data

METHOD
Usually, Black Box Testing method is used in Acceptance Testing. Testing does not normally
follow a strict procedure and is not scripted but is rather ad-hoc.

TASKS
Acceptance Test Plan
Prepare Review Rework Baseline
Acceptance Test Cases/Checklist
Prepare Review Rework Baseline
Acceptance Test
Perform

When is it performed?
Acceptance Testing is performed after System Testing and before making the system available
for actual use.
Who performs it?

Internal Acceptance Testing (Also known as Alpha Testing) is performed by members of the
organization that developed the software but who are not directly involved in the project
(Development or Testing). Usually, it is the members of Product Management, Sales and/or
Customer Support.

External Acceptance Testing is performed by people who are not employees of the
organization that developed the software.

 Customer Acceptance Testing is performed by the customers of the organization that

developed the software. They are the ones who asked the organization to develop the
software. [This is in the case of the software not being owned by the organization that
developed it.]
 User Acceptance Testing (Also known as Beta Testing) is performed by the end users
of the software. They can be the customers themselves or the customer’s customers.

Types of Acceptance testing

Benchmarking: a predetermined set of test cases corresponding to typical usage
conditions is executed against the system.
Pilot Testing: users employ the software as a small-scale experiment or in a controlled
Environment.
Alpha-Testing: pre-release closed / in-house user testing.
Beta-Testing: pre-release public user testing.
Parallel Testing: old and new software are used together and the old software is
gradually phased out.
 TWO MARK QUESTIONS

1. List the levels of Testing or Phases of testing.

 Unit Test
 Integration Test
 System Test
 Acceptance Test

2. Define integration Test.

At the integration level, several components are tested as a group and the tester
investigates component interactions.

3. Define System Testing.

The testing conducted on the complete integrated products and solutions to evaluate
system compliance with specified requirements on functional and non-functional aspects is
called system testing.

4. Define Alpha and Beta Test.

Alpha test: This test takes place at the developer’s site.
Beta Test: Beta tests ends the software to a cross-section of users who install it and
use it under real world working conditions.

5. What are the approaches that are used to develop the software?
There are two major approaches to software development
 Bottom-Up
 Top_Down
These approaches are supported by two major types of programming languages. They are
 procedure_oriented
 Object_oriented

6. Goals of Integration test.

  To detects defects that occur on the interface of the units.
 To assemble the individual units into working subsystems and finally a completed
system that ready for system test.

7. What are the Integration strategies?

 Top_ Down: In this strategy integration of the module begins with testing the upper
level modules.
 Bottom_ Up: In this strategy integration of the module begins with testing the
lowest level modules.

8. List the different types of system testing.

 Functional testing
 Performance testing
 Stress testing
 Configuration testing
 Security testing
 Recovery testing
The other types of system Testing are:
 Reliability & Usability testing.

9. Define functional Testing.

Functional tests at the system level are used to ensure that the behavior of the system adheres
to the requirement specifications.

10. What are the two major requirements in the Performance testing.
 Functional Requirement: User describe what functions the software should perform. We
test for compliance of the requirement at the system level with the functional based system
test.
 Quality Requirement: They are nonfunctional in nature but describe quality levels expected
for the software.
11. What are the steps for top down integration?
 Main control module is used as a test driver and stubs are substituted for all components
directly subordinate to the main module.
 Depending on integration approach (Depth or breadth first) subordinate stubs are replaced
one at a time with actual components.
 Tests are conducted as each component is integrated.
 The completion of each set of tests another stub is replaced with real component
 Regression testing may be conducted to ensure that new errors have not been introduced.

12. What is the importance of acceptance testing?

During acceptance test the development organization must show that the software meets all
of the client’s requirements. Very often final payments for system development depend on
the quality of the software as observed during the acceptance test.

13. What is Stub and Driver?

Stub: Is called by the Module under Test.
Driver: Calls the Module to be tested.

14. What are the documents relevant to integration test planning?

the requirements document,
the user manual and
usage scenarios.
These documents contain
structure charts,
state charts,
data dictionaries,
cross-reference tables,
module interface descriptions,
data flow descriptions,
 messages and event descriptions.

15. What is Cluster?

A cluster consists of classes that are related and they may work together to support a required
functionality for the complete system.

16. Differentiate between testing and debugging.

Testing as a dual purpose process reveal defects and to evaluate quality attributes. Debugging
or fault localization is the process of locating the fault or defect repairing the code, and
retesting the code.

17. Define an Actor and an Agent in a Bank Scenario.

Actor: needs to be concerned only about getting cash. Not concerned about what the
official is doing and what command he uses to interact with computer.
Agent: not concerned about the logic of how computer works. Concerned about
whether he will get money or not.

18. Define Acceptance Testing.

Acceptance testing is a phase after system testing that is normally done by the customers or
representative of the customer. Acceptance test is performed by the client, not by the
developer.

19. When is acceptance testing performed?

Acceptance Testing is performed after System Testing and before making the system
available for actual use.

20. What are the types of acceptance testing?

Benchmarking: a predetermined set of test cases corresponding to typical usage
conditions is executed against the system.
Pilot Testing: users employ the software as a small-scale experiment or in a controlled
Environment.
 Alpha-Testing: pre-release closed / in-house user testing.
Beta-Testing: pre-release public user testing.
Parallel Testing: old and new software are used together and the old software is
gradually phased out.

21.What is Big Bang Approach?

Here all the components are integrated together at once and then tested.

22. What do you mean by Incremental Approach?

In this approach, testing is done by joining two or more modules that are logically related.
Then the other related modules are added and tested for the proper functioning. The process
continues until all of the modules are joined and tested successfully.

23. What is Scenario Testing?

Scenario Testing is a variant of Software Testing where Scenarios are Used for Testing.
Scenarios help in an Easier Way of Testing of the more complicated Systems.

24. What is Hybrid / Sandwich Integration?

The hybrid / sandwich strategy is a combination of Top Down and Bottom up approaches.
Here, top modules are tested with lower modules, at the same time lower modules are
integrated with top modules and tested.

25. How do you perform Integration Testing?

The Integration test procedure irrespective of the Software testing strategies are:

1. Prepare the Integration Tests Plan.

2. Design the Test Scenarios, Cases and Scripts.
3. Execute the test cases followed by reporting the defects.
4. Track and re-test the defects.
5. Steps 3 and 4 are repeated until the completion of Integration is successful.