Chapter 2: What is Software Testing

 At the end of this chapter ,the students should be

able to achieve the following basic objectives
 To identify testing in different context in software
testing and quality assurance
 To discuss about basics of software testing
 to define about software testing process
 Software Testing Basics
 In any profession it is a must that you have a firm
understanding of the basic concepts of the profession.
 We will therefore briefly review the most important
issues in testing to make sure that the fundamentals are
present in our minds at all times.
 Definitions in Testing
 what is testing all about?
 Most of us have an idea of what testing is something about
finding errors.
 But further than that the confusion is fairly big.
 Let’s try to seek help in the standards.
IEEE 610 (Software Engineering Terminology): “The
process of operating a system or component under
specified conditions, observing or recording the results,
and making an evaluation of some aspect of the system or
component.”
IEEE 829 (Test Documentation):“The process of
analyzing a software item to detect the difference between
existing and required conditions (that is, bugs) and to
evaluate the features of the software items.”
 Definitions in Testing Cont.
BS 7925-1 (Software Testing–Vocabulary): “Process
of exercising software to verify that it satisfies
requirements and to detect errors.”
ISTQB(International Software Testing Qualifications
Board) Glossary of Terms used in Software Testing V
1.0: “The process consisting of all life cycle activities,
both static and dynamic, concerned with planning,
preparation and evaluation of software products and
related work products to determine that they satisfy
specified requirements, to demonstrate that they are fit
for purpose and to detect defects.”
 Definitions in Testing Cont.
 There is one term they all agree on: process.
 Testing is a process.
 what does that process entail? IEEE 610 and BS 7925-1,
respectively, talk about “operating” and “exercising”; that
is, the idea that testing requires the software to run on a
computer.
 This is what is also called “dynamic testing.” IEEE 829
broadens the idea to “analyzing,” thus including “static
testing.” And
 ISTQB takes the full step and includes both “dynamic
and static.” Testing is both dynamic and static.
 Definitions in Testing Cont.
 we have to conclude: Testing can be done on any work
product or product (where the difference is that work
products are not delivered to the customer).
 Testing gathers information about the quality of the
object under test.
 The quality is the amount of fulfillment of expectations..
 The product that should fulfill these expectations.
 The question is: Does it?
We test to be able to answer that question.
 Testing Processes
 A process is a series of activities performed to fulfill a
purpose and produce a tangible output based on a given
input.
 The tangible output is the way of communicating how
the purpose of the process has been fulfilled.
 The process view on software development is gaining
more and more interest.
 Process models are defined to assist organizations in
process improvement that is, in making their work more
structured and efficient..
 Like all processes the test process can be viewed at
different levels of detail.
 Processes in General
 Processes can be described and hence monitored and
improved.
 A process description must always include:
A definition of the input
A list of activities—the procedure
A description of the output
 For a more comprehensive and more useful process
description the following information could also be
included:
 Entry criteria—What must be in place before we can start?
 Purpose—A description of what must be achieved ?
 Role—Who is going to perform the activities?
 Processes in General Cont.
 Methods, techniques, tools—How exactly are we going to
perform the activities?
Measurements—What metrics are we going to collect for
the process?
Templates—What should the output look like?
Verification points—Are we on the right track?
Exit criteria—What do we need to fulfill before we can say
that we have finished?
 Monitoring Processes
 For testing processes it is of course important for the test
leader to know how the testing is performed and
progressing.
 processes should be the target processes for improvement
activities and to be able to predict and later determine the
effect of process improvement activities
 Processes Depend on Each Other
 The input to a process must be the output from one or more
proceeding processes
 The output from a process must be the input to one or more
other processes.
 even the final product, which is the input to the maintenance
process.
 A process model could be in a textual form or it could be
graphical, as shown in the following figure.
 Here, for example, the output from the top left process serves
as input to the top-middle process and to the lower-left
process.
 Processes Depend on Each Other Cont.

 The Overall Generic Test Process
 The generic test process defined in the ISTQB foundation
 syllabus can be described like this:
 The purpose of the test process is to provide information to
assure the quality of the product, decisions, and the
processes for a testing assignment.
 The inputs on which this process is based are:
 Test strategy
 Project plan
 Master test plan
 Information about how the testing is progressing
The Overall Generic Test Process Cont.
 The activities are:
 Test planning and control
Test development
Test analysis and design
 Test implementation and execution
Evaluating exit criteria and reporting
 Test closure activities
The Overall Generic Test Process Cont.
 The output consists of:
 Level test plan
 Test specification in the form of test conditions, test design,
test cases, and test procedures and/or test scripts
 Test environment design and specification and actual test
environment including test data
 Test logs
 Progress reports
 Test summary report
 Test experience report
The Overall Generic Test Process Cont.
 The generic test process is applicable for
the dynamic and static testing.
the development and maintenance of a product.
 For dynamic testing
 Component testing
Integration testing
System testing
Acceptance testing
 For static test
 reviews,
static analysis (automated static test)
The Overall Generic Test Process Cont.
 There are dependencies between software process based on
outputs.
 Some of them are described in the following sections.
 The test activities need not be performed in strict sequential
order.
Test planning and control are constant activities in the
sense that they are not just done once in the beginning of
the test assignment.
Monitoring of the process should be done on an ongoing
basis, and controlling and re-planning activities performed
when the need arises.
Sometimes test analysis and design is performed in parallel
with test implementation and execution.
 The Overall Generic Test Process Cont.
 The generic test process is iterative—not a simple
straightforward process.
 we’ll have to perform the activities more than once in an
iterative way before the exit criteria have been fulfilled.
 The Overall Generic Test Process Cont.
 The resulting iterations may be:
1. The defect is in the test object.
 A calculation does not give the expected result, and it appears
that the algorithm for the calculation has been coded wrongly.
 When the defect has been corrected we must retest the
software using the test procedure that encountered the
failure in the first place.
 We’ll probably also perform some regression testing.
2. The defect is in the test procedure
 A calculation does not give the expected result, but here it
appears that the test case was wrong.
 The Overall Generic Test Process Cont.
 The defect must be corrected and the new test case(s)
must be executed.
 This iteration usually goes back to the analysis and design
activity.
 The second activity from which an iteration may occur is
the evaluation of the exit criteria.
 This is where we find out if the exit criteria are not met.
 The resulting iteration in this situation may be:
 The Overall Generic Test Process Cont.
3. More test cases must be specified to increase coverage,
and these must then be executed.
In the checking it turns out that the decision coverage for a
component is only 87%. One more test case is designed and
when this is executed the coverage reaches 96%.
4. The exit criteria are relaxed or strengthened in the test
plan.
The coverage is found to be too small because of an-error
handling routine
that is very hard to reach. The required coverage for the
component is relaxed to 85%.
 Other Testing Processes
 The test process defined in the ISTQB syllabus is just
one example of a possible testing process.
 Each organization should create its own test process
suitable for the specific circumstances in the
organization.
 A test process may be created from scratch or it may be
created as a tailoring of a standard process.
 The various process improvement models that exist
provide frameworks for processes.
 One of the test specific models, Test Process Improvement
Model (TPI), defines a list of 20 key areas.
 Other Testing Processes Cont.
 The 20 key areas are grouped into four so-called
cornerstones as follows:
Life cycle:-Test strategy, life cycle model, moment of
involvement
Techniques:-Estimating and planning, test specification
techniques, static test techniques, metrics
Infrastructure:-Test tools, test environment, office
environment
Organization:-Commitment and motivation, test
functions and training, scope of methodology,
communication, reporting, defect management, testware
management, test process management, evaluating, low-
level testing
 Other Testing Processes Cont.
 Another process model is the Critical Testing Processes
(CTP).
 This model also defines a number of process areas.
 In this model the process areas are grouped into four
classes:
Plan:-Establish context, analyze risks, estimate, plan
Prepare:-Grow and train team, create testware, test
environment, and test processes
Perform:-Receive test object(s), execute and log tests
Perfect:-Report bugs, report test results, manage changes
 Test Planning and Control
 The purpose of the test planning process is
 to verify the mission of the testing,
to define the objectives of the testing, and
to make the necessary decisions to transform the test
strategy into an operational plan for the performance
of the actual testing task at hand.
 The planning must first be done at the overall level
resulting in a master test plan.
 The detailed planning for each test level is based on this
master test plan.
 Identical planning principles apply for the overall
planning and the detailed planning.
 Test Planning and Control Cont.
 The purpose of the control part is
to ensure that the planned activities are on track by
monitoring what is going on and
take corrective actions as appropriate.
 The inputs on which this process is based are:
Test strategy
 Master test plan
Information about how the testing is progressing
 Test Planning and Control Cont.
 The activities are:
 Verify the mission and define the objectives of the testing
Decide and document how the general test strategy and the
project test plan apply to the specific test level: what, how,
where, who
 Make decisions and initiate corrective actions as appropriate
as the testing progresses.
 The output consists of:
 Level test plan
 Test Planning and Control Cont.
Test planning and control phase consists of the
following contents .
 Input to Test Planning and Control
 Documentation of Test Planning and Control
 Activities in Test Planning
 Defining Test Object and Test Basis
Defining the Approach
Defining the Completion Criteria
 Defining Work Products and Their Relationships
Scoping the Test Effort
 Activities in Test Control
 Metrics for Test Planning and Control
 Input to Test Planning and Control
 The decisions to make in the test planning and control process
are guided by the expected contents of the test plan.
 Don’t get it wrong:
 The decisions are not made for the purpose of writing the plan,
but for the purpose of getting agreement and commitment of all
the stakeholders in the test to be performed.
 The planning and control of the test are continuous activities.
 The initial planning will take place first.
 Information from monitoring what is going on as the testing
progresses may cause controlling actions to be taken.
 These actions will usually involve new planning and necessary
corrections to be made in the plan when it no longer reflects the
reality
 Documentation of Test Planning and Control
 A template could be based on the IEEE 829 standard. This
standard suggests the following contents of a test plan.
 Test plan identifier
1. Introduction (scope, risks, and objectives)
2. Test item(s) (test object(s))
3. Features to be tested
4. Features not to be tested
5. Approach (targets, techniques, templates)
6. Item pass/fail criteria (exit criteria including coverage criteria)
7. Suspension criteria and resumption requirements
8. Test deliverables (work products)
 Documentation of Test Planning and Control
Cont.
9. Testing tasks (analysis, design, implementation, execution,
evaluation, reporting, and closure; all broken down into more
detailed activities in an appropriate work break down structure)
10. Environmental needs
11. Responsibilities
12. Staffing and training needs
13. Schedule
14. Risks and contingencies
 The level test plan produced and maintained in this process is
input to all the other detailed test processes.
 They all have the level test plan as their reference point for
information and decisions.
 Activities in Test Planning
 The planning of the acceptance testing can start as soon
as a draft of the user requirements is available.
 Early planning has a number of advantages. It provides
for example :-
 time to do a proper planning job,
 adequate time to include the stakeholders,
 early visibility of potential problems, and
 means of influencing the development plan (e.g., to develop in a
sequence that expedites testing).
 The test planning for a test level must verify the mission
and define the objectives.
 that is the goal or purpose, for the testing assignment.
 Based on this the more detailed planning can take place.
 Activities in Test Planning Cont.
 Defining Test Object and Test Basis
 The test planning must identify the test basis and define
what it is we are going to test in relation to this.
 This includes determination of the coverage to achieve
for the appropriate coverage item(s).
 The expected coverage must be documented in the level
test plan as (part of) the completion criteria.
 The coverage items depend on the test basis.
 Examples of the most common test basis and
corresponding coverage items are listed in the following
table.
Activities in Test Planning Cont.
 Activities in Test Planning Cont.

 Standards, both internal and external to the organization,

may also be used as the test basis.
 Defining the Approach
 The approach must at least cover:
 The test methods(static and dynamic ) and test techniques to
use.
 The structure of the test specification to be produced and used
 The tools to be used
 The interface with configuration management
 Measurements to collect
 Important constraints, such as availability or “fixed” deadline
 Activities in Test Planning Cont.
 Defining the Completion Criteria
 The completion criteria are derived from the strategy and
should be based on a risk analysis
 the higher the risk, the stricter the completion criteria;
the lower the risk the less demanding and specific the
completion criteria.
 It is important to decide up front which completion
criteria should be fulfilled before the test may be stopped.
 The completion criteria guide the specification of the test
and the selection of test case design techniques.
 Activities in Test Planning Cont.
 The most appropriate completion criteria vary from test
level to test level.
 Completion criteria for the test may be specified as
follows:
 Specified coverage has been achieved
 Specified number of failures found per test effort has been
achieved
 No known serious faults
 The benefits of the system are bigger than known problems
 Coverage is a very often used measurement and
completion criteria in testing.
 Test coverage is the degree, expressed as a percentage, to
which the coverage items have been exercised by a test.
 Activities in Test Planning Cont.

 Activities in Test Planning Cont.
 Defining Work Products and Their Relationships
 The number of deliverables, their characteristics, and
estimates of their sizes must be defined.
 this is used as input for the detailed estimation and scheduling
of all the test activities.
 Typical deliveries or work products from a test level are:
 Level test plan(s):-is/are the plan being specified in this
process.
 Test specification(s):-is a collective term for the result of the
test design and implementation activities.
 Test environment(s)
 Test logs and journals
 Test reports
 Activities in Test Planning Cont.
 It is important that the structure of the test specification is
outlined in the level test plan.
 Test specifications may be structured in many ways.
 Each organization must figure out which structure is the most
suitable for them.
 The structure shown and explained here is based on the
structure suggested in IEEE 829. A full test specification may
consists of:
A test design consisting of a number of test groups (or
designs) with test conditions and high-level test cases
derived from the basis documentation.
 The relationships between the elements in the basis
documentation and the high-level test cases are many-to-many
relationships.
 Activities in Test Planning Cont.
A number of low-level test cases extracted from the
high-level test cases and being made explicit with precise
input and output specifications .
A number of test procedures each encompassing a
number of test cases to be executed in sequence.
 The relationships between high-level test cases and test
procedures may also be complicated and include many-to-
many relationships.
Activities in Test Planning Cont.
 Activities in Test Planning Cont.
 Scoping the Test Effort
 exhaustive testing is a utopian goal.
 We have three mutually dependent parameters that we as testers
need to balance.
 The parameters are:
 Time: The available calendar time
 Resources: The available people and tools
 Quality: The quality of the testing
 These parameters form what we call the quality triangle, that
is the triangle for the quality of work we can deliver.
 The basic principle of the quality triangle is: It is not possible
to change one of the parameters and leave the other two
unchanged—and still be in balance!
 Test coverage is a measure for the quality of the test.
 Activities in Test Planning Cont.
Work Breakdown Structure
 The overall tasks are planning, monitoring, control,
analysis, design, implementation, execution, evaluation,
reporting, and closure.
 all broken down into more detailed activities in an appropriate
work breakdown structure.
 The tasks, together with resources and responsibilities,
are input items to the test schedule.
 A list and a description of every single task must
therefore be produced.
 Activities in Test Planning Cont.
 Be conscientious: remember to remember EVERYTHING!
 Be as detailed as necessary to get a precise estimate.
Defining Test Roles
 The roles to handle the testing tasks may be:
Test leader (manager or responsible)
Test analyst/designer
 Test executer
Reviewer/inspector
Domain expert
Test environment responsible
(Test)tool responsible
 Activities in Test Planning Cont.
 Test teams are formed by all these roles.
 We need different teams depending on which test phase
we are working in, but the principles are the same:
All relevant roles must be present and filled in the team
A role can be filled by one person or more people,
depending on the size of the testing assignment at hand
One person can fill one role or more roles, again
depending on the size.
 The roles each require a number of specific skills.
 If these skills are not available in the people, you must
describe any training needs here.
 The training should then be part of the activities to put in
the schedule.
 Activities in Test Planning Cont.
Producing the Schedule
 the staffing and the estimates are brought together and
transformed into a schedule.
 Risk analysis may be used to prioritize the testing for the
scheduling:
the higher the risk, the more time for testing and the earlier
the scheduled start of the testing task.
 The result of this is a schedule that shows precisely
 who should do
 what at which point in time and for how long.
 Estimations for all the tasks are input to the scheduling.
 Once the tasks are estimated they can be fitted into the project
time line.
 The schedule is also based on the actual people performing
the tasks, the people’s efficiency, and their availability.
 Activities in Test Control
 The control part of test management is about staying in
control and making necessary corrections to the plan
when it no longer reflects the reality.
 Measurements are collected in the test monitoring
activities for all the detailed activities in the test
processes,
 these measurements are analyzed to understand and follow the
actual progress of the planned test activities and the resulting
coverage.
 Decisions must be made if things are deviating
significantly from the plan, and corrective actions may be
necessary
 Metrics for Test Planning and Control
 Metrics to be defined for the monitoring and control of
the test planning and control activities themselves may
include:
 Number of tasks commenced over time
Task completion percentage over time
 Number of tasks completed over time
 Time spent on each task over time
 This will of course have to be compared to the estimates
and schedule of the test planning and control activities.