SOFTWARE TESTING 17IS63

Module-4
Syllabus
Process Framework :Basic principles: Sensitivity, redundancy, restriction, partition, visibility,
Feedback, the quality process, Planning and monitoring, Quality goals, Dependability properties,
Analysis Testing, Improving the process, Organizational factors.
Planning and Monitoring the Process: Quality and process, Test and analysis strategies and
plans, Risk planning, monitoring the process, Improving the process, the quality team
Documenting Analysis and Test: Organizing documents, Test strategy document, Analysis and
test plan, Test design specifications documents, Test and analysis reports.

Process Framework

Basic Principles
Principles provide a rationale for defining, selecting and applying techniques and
methods. Six principles that characterize various approaches and techniques for analysis
and testing:
1. Sensitivity
2. Redundancy
3. Restriction
4. Partition
5. Visibility
6. Feedback

1. Sensitivity:

The sensitivity principle states that it is better to fail every time than sometimes.

Consider the cost of detecting and repairing a software fault.

 If it is detected immediately, then the cost of correction is very small
 If a fault is detected in inspection or unit testing, the cost is still relatively small.
 If the first failure is detected in system or acceptance testing, the cost is very high
indeed, and the most costly faults are those detected by customers in the field.
RANGANATHA 1
 SOFTWARE TESTING 17IS63

A fault that triggers a failure on every execution is unlikely to survive past unit testing. A
characteristic of faults that escape detection, they trigger failures only rarely or in
combination with circumstances which is difficult to control.

Ex: The small C program has three faulty calls to string copy procedures. The call to
strcpy, strncpy, and stringCopy all pass a source string "Muddled," which is too long to
fit in the array middle.

strcpy: The fault may or may not cause an observable failure depending on the
arrangement of memory
strncpy: The standard recommendation is to use strncpy in place of strcpy. While strncpy
avoids overwriting other memory, it truncates the input without warning and sometimes
without properly null-terminating the output.
stringCopy: If the function stringCopy is used, which ensure that, if the target string is
too long, the program always fails in an observable manner.

#include <assert.h>
char before[ ] = "=Before=";
char middle[ ] = "Middle";
char after[ ] = "=After=";
void show()
{
printf("%s\n%s\n%s\n", before, middle, after);
}
void stringCopy(char *target, const char *source, int howBig);
int main(int argc, char *argv)
{
show();
strcpy(middle, "Muddled"); /* Fault, but may not fail */
show();
strncpy(middle, "Muddled", sizeof(middle)); /* Fault, may not fail */

RANGANATHA 2
 SOFTWARE TESTING 17IS63

show();
stringCopy(middle, "Muddled",sizeof(middle)); /* Guaranteed to fail */
show();
}
void stringCopy(char *target, const char *source, int howBig)
{
assert(strlen(source) < howBig);
strcpy(target, source);
}

The sensitivity principle says, try to make these faults easier to detect by making them
cause failure more often. It can be applied in three main ways: At the design level, the
analysis and testing level and at the environment level

 Ex: Replacing strcpy and strncpy with stringCopy in the above program which is
an example of application of the sensitivity principle in design. Run-time array
bounds checking is an example of the sensitivity principle applied at the language
level.

The sensitivity principle applied to design and code. The sensitivity principle can also be
applied to test and analysis techniques.

2. Redundancy
 Redundancy is the opposite of independence. If one part of a software artifact such
as program, design document, etc. constrains the content of another, then they are
not entirely independent, and it is possible to check them for consistency
 Redundancy can introduced in software test and analysis, to detect faults that
could lead to differences between intended behavior and actual behavior, so the
most valuable form of redundancy is in the form of an explicit, redundant
statement of intent

RANGANATHA 3
 SOFTWARE TESTING 17IS63

 Redundancy can be introduced with an automatic, algorithmic check for

consistency. It has the advantage of being much cheaper and more thorough than
dynamic testing or manual inspection.
 Ex: Static type checking is a classic application of this principle: The type
declaration is a statement of intent that is at least partly redundant with the use of a
variable in the source code. The type declaration constrains other parts of the code,
so a consistency check (type check) can be applied.
 Checkable redundancy is not limited to program source code, nor is it something
that can be introduced only by programming language designers but also one can
also intentionally introduce redundancy in other software artifacts, even those that
are not entirely formal.
 When redundancies is already present then make sure the information is
represented in a way that facilitates cheap, thorough consistency checks.
3. Restriction
 When there are no acceptably cheap and effective ways to check a property,
sometimes one can change the problem by checking a different, more restrictive
property or by limiting the check to a smaller, more restrictive class of programs.

 Ex: Consider the problem of ensuring that each variable is initialized before it is
used, on every execution. This property is simple but it is not possible for a
compiler or analysis tool to precisely determine whether it holds.

 In below program, can the variable k ever be uninitialized the first time i is added
to it? If someCondition(0) always returns true, then k will be initialized to zero on
the first time through the loop, before k is incremented, so there is no potential for
a run-time error

Static void questionable()

{
int k;
for (int i=0; i < 10; ++i)
{
if (someCondition(i))
{
k=0;

RANGANATHA 4
 SOFTWARE TESTING 17IS63

}
else
{
k+=i;
}
}
System.out.println (k);
}
}

 Java provides a solution to this problem is to enforce a stricter, simpler condition:

A program is not permitted to have any syntactic control paths on which an
uninitialized reference could occur, regardless of whether those paths could
actually be executed. The above program has a path, so the Java compiler rejects
it.
 From the above example it is clear that the restriction principle is useful mainly
during design and specification. In other words, restriction is mainly a principle to
be applied in design for test. Often it can be applied not only to solve a single
problem but also at architectural level to simplify a whole set of analysis
problems.

4. Partition
 Partition is known as "divide and conquer," is a general engineering principle.
Dividing a complex problem into sub-problems to be attacked and solved
independently is the most common problem-solving strategy.

 Partitioning can be applied both at process and technique levels. At the process
level, divide complex activities into sets of simple activities that can be attacked
independently.
Ex: Testing is divided into unit, integration, subsystem, and system testing. In this
way, we can focus on different sources of faults at different steps and at each step.

 Many static analysis techniques first construct a model of a system and then
analyze the model. In this way they divide the overall analysis into two subtasks:
1. Simplify the system to make the proof of the desired properties feasible
2. Then prove the property with respect to the simplified model.

RANGANATHA 5
 SOFTWARE TESTING 17IS63

 For a program it is impossible to execute every conceivable input, so systematic

testing strategies must identify a finite number of classes of test cases to execute.
Whether the classes are derived from specifications (functional testing) or from
program structure (structural testing), the process testing proceeds by dividing the
sources of information into significant elements and creating test cases that cover
each such element or certain combinations of elements.

5. Visibility
 Visibility means the ability to measure progress or status against goals.
 The visibility principle can encounter mainly in the form of process visibility and
schedule visibility: ability to judge the state of development against a project
schedule. Quality process ability to measuring achieved quality against quality
goals.
 The principle of visibility involves setting goals that can be assessed as well as
devising methods to assess their realization.
 Visibility is closely related to observability: The ability to extract useful
information from a software artifact. The architectural design and build plan of a
system determines what will be observable at each stage of development, which in
turn determines the visibility of progress against goals at that stage.
 A variety of simple techniques can be used to improve observability. Ex: Internet
protocols like HTTP and SMTP are based on the exchange of simple textual
commands. The choice of simple, human-readable text rather than a more compact
binary encoding has a small cost in performance and a large payoff in
observability. Use of human-readable and human-editable files is advisable
wherever the performance cost is acceptable.

6. Feedback

 Feedback is another principle that applies to analysis and testing. Feedback applies
both to the process itself (process improvement) and to individual techniques

 Systematic inspection and walkthrough derive part of their success from feedback.
Ex: Participants in inspection are guided by checklists, and checklists are revised
and refined based on experience. New checklist items may be derived from root

RANGANATHA 6
 SOFTWARE TESTING 17IS63

cause analysis, analyzing previously observed failures to identify the initial errors
that lead to them.

Quality Process

Quality process: set of activities and responsibilities

– focused primarily on ensuring adequate dependability
– concerned with project schedule or with product usability
• The quality process provides a framework for
– selecting and arranging activities
– Considering interactions and trade-offs with other important goals.
Interactions and tradeoffs
Example high dependability vs. time to market
• Mass market products: – better to achieve a reasonably high degree of dependability on a tight
schedule than to achieve ultra-high dependability on a much longer schedule
• Critical medical devices: – better to achieve ultra-high dependability on a much longer
schedule than a reasonably high degree of dependability on a tight schedule
Properties of the Quality Process
• Completeness: Appropriate activities are planned to detect each important class of faults.
• Timeliness: Faults are detected at a point of high leverage (as early as possible)
• Cost-effectiveness: Activities are chosen depending on cost and effectiveness
– Cost must be considered over the whole development cycle and product life
– The dominant factor is usually the cost of repeating an activity through many change cycles.

Planning and Monitoring

• The quality process
– Balances several activities across the whole development process
– Selects and arranges them to be as cost-effective as possible

RANGANATHA 7
 SOFTWARE TESTING 17IS63

– Improves early visibility

• Quality goals can be achieved only through careful planning
• Planning is integral to the quality process

Process Visibility
• A process is visible to the extent that one can answer the question
– How does our progress compare to our plan?
– Example: Are we on schedule? How far ahead or behind?
• The quality process has not achieved adequate visibility if one cannot gain strong confidence in
the quality of the software system before it reaches final testing
– quality activities are usually placed as early as possible
• design test cases at the earliest opportunity (not ``just in time'')
• uses analysis techniques on software artifacts produced before actual code.
– motivates the use of “proxy” measures
• Ex: the number of faults in design or code is not a true measure of reliability, but we may count
faults discovered in design inspections as an early indicator of potential quality problems

A&T Strategy
• Identifies company- or project-wide standards that must be satisfied
– Procedures required, e.g., for obtaining quality certificates
– Techniques and tools that must be used
– Documents that must be produced

A&T Plan
• A comprehensive description of the quality process that includes:
– Objectives and scope of a&t activities
– Documents and other items that must be available
– Items to be tested
– Features to be tested and not to be tested
– Analysis and test activities
– Staff involved in a&t

RANGANATHA 8
 SOFTWARE TESTING 17IS63

– Constraints
– Pass and fail criteria
– Schedule
– Deliverables
– Hardware and software requirements
– Risks and contingencies
Quality Goals
• Process qualities (visibility,....)
• Product qualities – internal qualities (maintainability,..)
– External qualities
• Usefulness qualities:
– Usability, performance, security, portability, interoperability
• Dependability
– Correctness, reliability, safety, robustness

Dependability Properties

1. Correctness: – A program is correct if it is consistent with its specification

• Seldom practical for non-trivial systems
2. Reliability: – likelihood of correct function for some ``unit'' of behavior
• relative to a specification and usage profile

RANGANATHA 9
 SOFTWARE TESTING 17IS63

• statistical approximation to correctness (100% reliable = correct)

3. Safety: – preventing hazards
4. Robustness – acceptable (degraded) behavior under extreme conditions

Mean time between failures (MTBF) is yet another measure of reliability, also using time as
the unit of execution. The hypothetical network switch that typically fails once in a 24- hour
period and takes about an hour to recover has a mean time between failures of 23 hours. Note
that availability does not distinguish between two failures of 30 minutes each and one failure
lasting an hour, while MTBF does.

Analysis
• Analysis includes
– Manual inspection techniques
– automated analyses
• Can be applied at any development stage
• Particularly well suited at the early stages of specifications an design

Inspection
• Can be applied to essentially any document
– requirements statements
– architectural and detailed design documents
– test plans and test cases
– program source code
• May also have secondary benefits
– Spreading good practices
– instilling shared standards of quality.
• Takes a considerable amount of time
• re-inspecting a changed component can be expensive
• used primarily
– where other techniques are inapplicable

RANGANATHA 10
 SOFTWARE TESTING 17IS63

– where other techniques do not provide sufficient coverage

Automatic Static Analysis

• More limited in applicability
– can be applied to some formal representations of requirements models
– not to natural language documents
• Are selected when available
– substituting machine cycles for human effort makes them particularly cost-effective

Testing
• Executed late in development
• Start as early as possible
• Early test generation has several advantages
– Tests generated independently from code, when the specifications are fresh in the mind
of analysts
– The generation of test cases may highlight inconsistencies and incompleteness of the
corresponding specifications
– Tests may be used as compendium of the specifications by the programmers

Improving the Process

• Long lasting errors are common
• It is important to structure the process for
– Identifying the most critical persistent faults
– Tracking them to frequent errors
– Adjusting the development and quality processes to eliminate errors
• Feedback mechanisms are the main ingredient of the quality process for identifying and
removing errors

RANGANATHA 11
 SOFTWARE TESTING 17IS63

Organizational factors
• Different teams for development and quality?
– Separate development and quality teams is common in large organizations
– Indistinguishable roles is postulated by some methodologies (extreme programming)
• Different roles for development and quality?
– Test designer is a specific role in many organizations
– Mobility of people and roles by rotating engineers over development and testing tasks
among different projects is a possible option

Planning and Monitoring the Process And Documenting Analysis and Test

Any complex process requires planning and monitoring. The quality process requires
coordination of many different activities over a period that spans a full development cycle and
beyond. Planning is necessary to order, provision, and coordinate all the activities that support a
quality goal, and monitoring of actual status against a plan is required to steer and adjust the
process.

Overview
Planning involves scheduling activities, allocating resources, and devising observable,
unambiguous milestones against which progress and performance can be monitored. Monitoring
means answering the question, "How are we doing?"
Quality planning is one aspect of project planning, and quality processes must be closely
coordinated with other development processes. Coordination among quality and development
tasks may constrain ordering (e.g., unit tests are executed after creation of program units). It may
shape tasks to facilitate coordination; for example, delivery may be broken into smaller
increments to allow early testing. Some aspects of the project plan, such as feedback and design
for testability, may belong equally to the quality plan and other aspects of the project plan.
Quality planning begins at the inception of a project and is developed with the overall project
plan, instantiating and building on a quality strategy that spans several projects. Like the overall
project plan, the quality plan is developed incrementally, beginning with the feasibility study and
continuing through development and delivery.
Formulation of the plan involves risk analysis and contingency planning. Execution of the plan
involves monitoring, corrective action, and planning for subsequent releases and projects.

RANGANATHA 12
 SOFTWARE TESTING 17IS63

Allocating responsibility among team members is a crucial and difficult part of planning. When
one person plays multiple roles, explicitly identifying each responsibility is still essential for
ensuring that none are neglected.

Quality and Process

A software plan involves many intertwined concerns, from schedule to cost to usability and
dependability. Despite the intertwining, it is useful to distinguish individual concerns and
objectives to lessen the likelihood that they will be neglected, to allocate responsibilities, and to
make the overall planning process more manageable.
For example, a mature software project plan will include architectural design reviews, and the
quality plan will allocate effort for reviewing testability aspects of the structure and build order.
Clearly, design for testability is an aspect of software design and cannot be carried out by a
separate testing team in isolation. It involves both test designers and other software designers in
explicitly evaluating testability as one consideration in selecting among design alternatives. The
objective of incorporating design for testability in the quality process is primarily to ensure that it
is not overlooked and secondarily to plan activities that address it as effectively as possible.
An appropriate quality process follows a form similar to the overall software process in which it
is embedded. In a strict (and unrealistic) waterfall software process, one would follow the "V
model" in a sequential manner, beginning unit testing only as implementation commenced
following completion of the detailed design phase, and finishing unit testing before integration
testing commenced. In the XP "test first" method, unit testing is conflated with subsystem and
system testing. A cycle of test design and test execution is wrapped around each small-grain
incremental development step. The role that inspection and peer reviews would play in other
processes is filled in XP largely by pair programming. A typical spiral process model lies
somewhere between, with distinct planning, design, and implementation steps in several
increments coupled with a similar unfolding of analysis and test activities.
A general principle, across all software processes, is that the cost of detecting and repairing a
fault increases as a function of time between committing an error and detecting the resultant
faults. Thus, whatever the intermediate work products in software plan, an efficient quality plan
will include a matched set of intermediate validation and verification activities that detect most
faults within a short period of their introduction. Any step in a software process that is not paired
with a validation or verification step is an opportunity for defects to fester, and any milestone in
a project plan that does not include a quality check is an opportunity for a misleading assessment
of progress.
The particular verification or validation step at each stage depends on the nature of the
intermediate work product and on the anticipated defects. For example, anticipated defects in a
requirements statement might include incompleteness, ambiguity, inconsistency, and over
ambition relative to project goals and resources. A review step might address some of these, and
automated analyses might help with completeness and consistency checking.

RANGANATHA 13
 SOFTWARE TESTING 17IS63

The evolving collection of work products can be viewed as a set of descriptions of different parts
and aspects of the software system, at different levels of detail. Portions of the implementation
have the useful property of being executable in a conventional sense, and are the traditional
subject of testing, but every level of specification and design can be both the subject of
verification activities and a source of information for verifying other artifacts. A typical
intermediate artifact - say, a subsystem interface definition or a database schema - will be subject
to the following steps:
Internal consistency check Check the artifact for compliance with structuring rules that define
"well-formed" artifacts of that type. An important point of leverage is defining the syntactic and
semantic rules thoroughly and precisely enough that many common errors result in detectable
violations. This is analogous to syntax and strong-typing rules in programming languages, which
are not enough to guarantee program correctness but effectively guard against many simple
errors.
External consistency check Check the artifact for consistency with related artifacts. Often this
means checking for conformance to a "prior" or "higher-level" specification, but consistency
checking does not depend on sequential, top-down development - all that is required is that the
related information from two or more artifacts be defined precisely enough to support detection
of discrepancies.
Consistency usually proceeds from broad, syntactic checks to more detailed and expensive
semantic checks, and a variety of automated and manual verification techniques may be applied.
Generation of correctness conjectures Correctness conjectures, which can be test outcomes or
other objective criteria, lay the groundwork for external consistency checks of other work
products, particularly those that are yet to be developed or revised. Generating correctness
conjectures for other work products will frequently motivate refinement of the current product.
For example, an interface definition may be elaborated and made more precise so that
implementations can be effectively tested.
Test and Analysis Strategies
Lessons of past experience are an important asset of organizations that rely heavily on technical
skills. A body of explicit knowledge, shared and refined by the group, is more valuable than
islands of individual competence. Organizational knowledge in a shared and systematic form is
more amenable to improvement and less vulnerable to organizational change, including the loss
of key individuals. Capturing the lessons of experience in a consistent and repeatable form is
essential for avoiding errors, maintaining consistency of the process, and increasing development
efficiency.
Cleanroom
The Cleanroom process model, introduced by IBM in the late 1980s, pairs development with
V&V activities and stresses analysis over testing in the early phases. Testing is left for system
certification. The Cleanroom process involves two cooperating teams, the development and the

RANGANATHA 14
 SOFTWARE TESTING 17IS63

quality teams, and five major activities: specification, planning, design and verification, quality
certification, and feedback

In the specification activity, the development team defines the required behavior of the system,
while the quality team defines usage scenarios that are later used for deriving system test suites.
The planning activity identifies incremental development and certification phases.
After planning, all activities are iterated to produce incremental releases of the system. Each
system increment is fully deployed and certified before the following step. Design and code
undergo formal inspection ("Correctness verification") before release. One of the key premises
underpinning the Cleanroom process model is that rigorous design and formal inspection
produce "nearly fault-free software."
Usage profiles generated during specification are applied in the statistical testing activity to
gauge quality of each release. Another key assumption of the Cleanroom process model is that
usage profiles are sufficiently accurate that statistical testing will provide an accurate measure of
quality as perceived by users.[a] Reliability is measured in terms of mean time between failures
(MTBF) and is constantly controlled after each release. Failures are reported to the development
team for correction, and if reliability falls below an acceptable range, failure data is used for
process improvement before the next incremental release.

Software organizations can develop useful, organization-specific strategies because of

similarities among projects carried out by a particular organization in a set of related application

RANGANATHA 15
 SOFTWARE TESTING 17IS63

domains. Test and analysis strategies capture commonalities across projects and provide
guidelines for maintaining consistency among quality plans.
A strategy is distinguished from a plan in that it is not specific to a single project. Rather, it
provides guidance and a general framework for developing quality plans for several projects,
satisfying organizational quality standards, promoting homogeneity across projects, and making
both the creation and execution of individual project quality plans more efficient.
The quality strategy is an intellectual asset of an individual organization prescribing a set of
solutions to problems specific to that organization. Among the factors that particularize the
strategy are:
Structure and size Large organizations typically have sharper distinctions between development
and quality groups, even if testing personnel are assigned to development teams. In smaller
organizations, it is more common for a single person to serve multiple roles. Where
responsibility is distributed among more individuals, the quality strategy will require more
elaborate attention to coordination and communication, and in general there will be much greater
reliance on documents to carry the collective memory.
In a smaller organization, or an organization that has devolved responsibility to small, semi-
autonomous teams, there is typically less emphasis on formal communication and documents but
a greater emphasis on managing and balancing the multiple roles played by each team member.
Overall process We have already noted the intertwining of quality process with other aspects of
an overall software process, and this is of course reflected in the quality strategy. For example, if
an organization follows the Cleanroom methodology, then inspections will be required but unit
testing forbidden. An organization that adopts the XP methodology is likely to follow the "test
first" and pair programming elements of that approach, and in fact would find a more document-
heavy approach a difficult fit.
Notations, standard process steps, and even tools can be reflected in the quality strategy to the
extent they are consistent from project to project. For example, if an organization consistently
uses a particular combination of UML diagram notations to document subsystem interfaces, then
the quality strategy might include derivation of test designs from those notations, as well as
review and analysis steps tailored to detect the most common and important design flaws at that
point. If a particular version and configuration control system is woven into process
management, the quality strategy will likewise exploit it to support and enforce quality process
steps.
Application domain The domain may impose both particular quality objectives (e.g., privacy
and security in medical records processing), and in some cases particular steps and
documentation required to obtain certification from an external authority. For example, the
RTCA/DO-178B standard for avionics software requires testing to the modified
condition/decision coverage (MC/DC) criterion.

SRET

RANGANATHA 16
 SOFTWARE TESTING 17IS63

The software reliability engineered testing (SRET) approach, developed at AT&T in the early
1990s, assumes a spiral development process and augments each coil of the spiral with rigorous
testing activities. SRET identifies two main types of testing: development testing, used to find
and remove faults in software at least partially developed in-house, and certification testing, used
to either accept or reject outsourced software.
The SRET approach includes seven main steps. Two initial, quick decision-making steps
determine which systems require separate testing and which type of testing is needed for each
system to be tested. The five core steps are executed in parallel with each coil of a spiral
development process.

The five core steps of SRET are:

Define "Necessary" Reliability Determine operational models, that is, distinct patterns of
system usage that require separate testing, classify failures according to their severity, and
engineer the reliability strategy with fault prevention, fault removal, and fault tolerance
activities.
Develop Operational Profiles Develop both overall profiles that span operational models and
operational profiles within single operational models.
Prepare for Testing Specify test cases and procedures.
Execute Tests
Interpret Failure Data Interpretation of failure data depends on the type of testing. In
development testing, the goal is to track progress and compare present failure intensities with
objectives. In certification testing, the goal is to determine if a software component or system
should be accepted or rejected.

Extreme Programming (XP)

RANGANATHA 17
 SOFTWARE TESTING 17IS63

The extreme programming methodology (XP) emphasizes simplicity over generality, global
vision and communication over structured organization, frequent changes over big releases,
continuous testing and analysis over separation of roles and responsibilities, and continuous
feedback over traditional planning.
Customer involvement in an XP project includes requirements analysis (development,
refinement, and prioritization of user stories) and acceptance testing of very frequent iterative
releases. Planning is based on prioritization of user stories, which are implemented in short
iterations. Test cases corresponding to scenarios in user stories serve as partial specifications.

Test cases suitable for batch execution are part of the system code base and are implemented
prior to the implementation of features they check ("test-first"). Developers work in pairs,
incrementally developing and testing a module. Pair programming effectively conflates a review
activity with coding. Each release is checked by running all the tests devised up to that point of
development, thus essentially merging unit testing with integration and system testing. A failed
acceptance test is viewed as an indication that additional unit tests are needed.

Although there are no standard templates for analysis and test strategies, we can identify a few
elements that should be part of almost any good strategy. A strategy should specify common
quality requirements that apply to all or most products, promoting conventions for
unambiguously stating and measuring them, and reducing the likelihood that they will be
overlooked in the quality plan for a particular project. A strategy should indicate a set of
documents that is normally produced during the quality process, and their contents and
relationships. It should indicate the activities that are prescribed by the overall process
organization. Often a set of standard tools and practices will be prescribed, such as the interplay
of a version and configuration control tool with review and testing procedures. In addition, a
strategy includes guidelines for project staffing and assignment of roles and responsibilities.

RANGANATHA 18
 SOFTWARE TESTING 17IS63

Test and Analysis Plans

An analysis and test plan details the steps to be taken in a particular project. A plan should
answer the following questions:
 What quality activities will be carried out?
 What are the dependencies among the quality activities and between quality and
development activities?
 What resources are needed and how will they be allocated?
 How will both the process and the evolving product be monitored to maintain an
adequate assessment of quality and early warning of quality and schedule problems?
Each of these issues is addressed to some extent in the quality strategy, but must be elaborated
and particularized. This is typically the responsibility of a quality manager, who should
participate in the initial feasibility study to identify quality goals and estimate the contribution of
test and analysis tasks on project cost and schedule.
To produce a quality plan that adequately addresses the questions above, the quality manager
must identify the items and features to be verified, the resources and activities that are required,
the approaches that should be followed, and criteria for evaluating the results.
Items and features to be verified circumscribe the target of the quality plan. While there is an
obvious correspondence between items to be developed or modified and those to undergo testing,
they may differ somewhat in detail. For example, overall evaluation of the user interface may be
the purview of a separate human factors group. The items to be verified, moreover, include many
intermediate artifacts such as requirements specifications and design documents, in addition to
portions of the delivered system. Approaches to be taken in verification and validation may vary
among items. For example, the plan may prescribe inspection and testing for all items and
additional static analyses for multi-threaded subsystems.
Quality goals must be expressed in terms of properties satisfied by the product and must be
further elaborated with metrics that can be monitored during the course of the project. For
example, if known failure scenarios are classified as critical, severe, moderate, and minor, then
we might decide in advance that a product version may enter end-user acceptance testing only
when it has undergone system testing with no outstanding critical or severe failures.
Defining quality objectives and process organization in detail requires information that is not all
available in the early stages of development. Test items depend on design decisions; detailed
approaches to evaluation can be defined only after examining requirements and design
specifications; tasks and schedule can be completed only after the design; new risks and
contingencies may be introduced by decisions taken during development. On the other hand, an
early plan is necessary for estimating and controlling cost and schedule. The quality manager
must start with an initial plan based on incomplete and tentative information, and incrementally
refine the plan as more and better information becomes available during the project.
After capturing goals as well as possible, the next step in construction of a quality plan is to
produce an overall rough list of tasks. The quality strategy and past experience provide a basis

RANGANATHA 19
 SOFTWARE TESTING 17IS63

for customizing the list to the current project and for scaling tasks appropriately. For example,
experience (preferably in the form of collected and analyzed data from past projects, rather than
personal memory) might suggest a ratio of 3:5 for person-months of effort devoted to integration
test relative to coding effort. Historical data may also provide scaling factors for the application
domain, interfaces with externally developed software, and experience of the quality staff. To the
extent possible, the quality manager must break large tasks into component subtasks to obtain
better estimates, but it is inevitable that some task breakdown must await further elaboration of
the overall project design and schedule.
The manager can start noting dependencies among the quality activities and between them and
other activities in the overall project, and exploring arrangements of tasks over time. The main
objective at this point is to schedule quality activities so that assessment data are provided
continuously throughout the project, without unnecessary delay of other development activities.
For example, the quality manager may note that the design and implementation of different
subsystems are scheduled in different phases, and may plan subsystem testing accordingly.
Where there is a choice between scheduling a quality activity earlier or later, the earliest point
possible is always preferable. However, the demand on resources (staff time, primarily) must be
leveled over time, and often one must carefully schedule the availability of particular critical
resources, such as an individual test designer with expertise in a particular technology.
Maintaining a consistent level of effort limits the number of activities that can be carried on
concurrently, and resource constraints together with the objective of minimizing project delays
tends to force particular orderings on tasks.
If one has a choice between completing two tasks in four months, or completing the first task in
two months and then the second in another two months, the schedule that brings one task to
completion earlier is generally advantageous from the perspective of process visibility, as well as
reduced coordination overhead. However, many activities demand a fraction of a person's
attention over a longer period and cannot be compressed. For example, participation in design
and code inspection requires a substantial investment of effort, but typically falls short of a full-
time assignment.
Since delayed inspections can be a bottleneck in progress of a project, they should have a high
priority when they can be carried out, and are best interleaved with tasks that can be more
flexibly scheduled.
While the project plan shows the expected schedule of tasks, the arrangement and ordering of
tasks are also driven by risk. The quality plan, like the overall project plan, should include an
explicit risk plan that lists major risks and contingencies.
A key tactic for controlling the impact of risk in the project schedule is to minimize the
likelihood that unexpected delay in one task propagates through the whole schedule and delays
project completion. One first identifies the critical paths through the project schedule. Critical
paths are chains of activities that must be completed in sequence and that have maximum overall
duration. Tasks on the critical path have a high priority for early scheduling, and likewise the
tasks on which they depend (which may not themselves be on the critical path) should be

RANGANATHA 20
 SOFTWARE TESTING 17IS63

scheduled early enough to provide some schedule slack and prevent delay in the inception of the
critical tasks.
A critical dependence occurs when a task on a critical path is scheduled immediately after some
other task on the critical path, particularly if the length of the critical path is close to the length of
the project. Critical dependence may occur with tasks outside the quality plan part of the overall
project plan.
The primary tactic available for reducing the schedule risk of a critical dependence is to
decompose a task on the critical path, factoring out subtasks that can be performed earlier. For
example, an acceptance test phase late in a project is likely to have a critical dependence on
development and system integration. One cannot entirely remove this dependence, but its
potential to delay project completion is reduced by factoring test design from test execution.
Figure 8.1 shows alternative schedules for a simple project that starts at the beginning of January
and must be completed by the end of May. In the top schedule, indicated as CRITICAL
SCHEDULE, the tasks Analysis and design, Code and Integration, Design and execute
subsystem tests, and Design and execute system tests form a critical path that spans the duration
of the entire project. A delay in any of the activities will result in late delivery. In this schedule,
only the Produce user documentation task does not belong to the critical path, and thus only
delays of this task can be tolerated

RANGANATHA 21
 SOFTWARE TESTING 17IS63

Figure 8.1: Three possible simple schedules with different risks and resource allocation. The bars indicate
the duration of the tasks. Diamonds indicate milestones, and arrows between bars indicate precedence
between tasks.
In the middle schedule, marked as UNLIMITED RESOURCES, the test design and execution
activities are separated into distinct tasks. Test design tasks are scheduled early, right after
analysis and design, and only test execution is scheduled after Code and integration. In this way

RANGANATHA 22
 SOFTWARE TESTING 17IS63

the tasks Design subsystem tests and Design system tests are removed from the critical path,
which now spans 16 weeks with a tolerance of 5 weeks with respect to the expected termination
of the project. This schedule assumes enough resources for running Code and integration,
Production of user documentation, Design of subsystem tests, and Design of system tests.
The LIMITED RESOURCES schedule at the bottom of Figure 8.1 rearranges tasks to meet
resource constraints. In this case we assume that test design and execution, and production of
user documentation share the same resources and thus cannot be executed in parallel. We can see
that, despite the limited parallelism, decomposing testing activities and scheduling test design
earlier results in a critical path of 17 weeks, 4 weeks earlier than the expected termination of the
project. Notice that in the example, the critical path is formed by the tasks Analysis and design,
Design subsystem tests, Design system tests, Produce user documentation, Execute subsystem
tests, and Execute system tests. In fact, the limited availability of resources results in
dependencies among Design subsystem tests, Design system tests and Produce user
documentation that last longer than the parallel task Code and integration.
The completed plan must include frequent milestones for assessing progress. A rule of thumb is
that, for projects of a year or more, milestones for assessing progress should occur at least every
three months. For shorter projects, a reasonable maximum interval for assessment is one quarter
of project duration.
Figure 8.2 shows a possible schedule for the initial analysis and test plan for the business logic of
the Chipmunk Web presence in the form of a GANTT diagram. In the initial plan, the manager
has allocated time and effort to inspections of all major artifacts, as well as test design as early as
practical and ongoing test execution during development. Division of the project into major parts
is reflected in the plan, but further elaboration of tasks associated with units and smaller
subsystems must await corresponding elaboration of the architectural design. Thus, for example,
inspection of the shopping facilities code and the unit test suites is shown as a single aggregate
task. Even this initial plan does reflect the usual Chipmunk development strategy of regular
"synch and stabilize" periods punctuating development, and the initial quality plan reflects the
Chipmunk strategy of assigning responsibility for producing unit test suites to developers, with
review by a member of the quality team

RANGANATHA 23
 SOFTWARE TESTING 17IS63

Figure 8.2: Initial schedule for quality activities in development of the business logic subsystem
of the Chipmunk Web presence, presented as a GANTT diagram.
The GANTT diagram shows four main groups of analysis and test activities: design inspection,
code inspection, test design, and test execution. The distribution of activities over time is
constrained by resources and dependence among activities. For example, system test execution
starts after completion of system test design and cannot finish before system integration (the sync
and stablize elements of development framework) is complete. Inspection activities are
constrained by specification and design activities. Test design activities are constrained by
limited resources. Late scheduling of the design of integration tests for the administrative
business logic subsystem is necessary to avoid overlap with design of tests for the shopping
functionality subsystem.
The GANTT diagram does not highlight intermediate milestones, but we can easily identify two
in April and July, thus dividing the development into three main phases. The first phase (January
to April) corresponds to requirements analysis and architectural design activities and terminates
with the architectural design baseline. In this phase, the quality team focuses on design
inspection and on the design of acceptance and system tests. The second phase (May to July)
corresponds to subsystem design and to the implementation of the first complete version of the
system. It terminates with the first stabilization of the administrative business logic subsystem. In
this phase, the quality team completes the design inspection and the design of test cases. In the
final stage, the development team produces the final version, while the quality team focuses on
code inspection and test execution.

RANGANATHA 24
 SOFTWARE TESTING 17IS63

Absence of test design activities in the last phase results from careful identification of activities
that allowed early planning of critical tasks.

Risk Planning
Risk is an inevitable part of every project, and so risk planning must be a part of every plan.
Risks cannot be eliminated, but they can be assessed, controlled, and monitored.
The risk plan component of the quality plan is concerned primarily with personnel risks,
technology risks, and schedule risk. Personnel risk is any contingency that may make a qualified
staff member unavailable when needed. For example, the reassignment of a key test designer
cannot always be avoided, but the possible consequences can be analyzed in advance and
minimized by careful organization of the work. Technology risks in the quality plan include risks
of technology used specifically by the quality team and risks of quality problems involving other
technology used in the product or project. For example, changes in the target platform or in the
testing environment, due to new releases of the operating system or to the adoption of a new
testing tool suite, may not be schedulable in advance, but may be taken into account in the
organization of the testing environment. Schedule risk arises primarily from optimistic
assumptions in the quality plan. For example, underestimating scaffolding design and
maintenance is a common mistake that cannot always be avoided, but consequences can be
mitigated (e.g., by allowing for a reasonable slack time that can absorb possible delays). Many
risks and the tactics for controlling them are generic to project management (e.g., cross-training
to reduce the impact of losing a key staff member). Here we focus on risks that are specific to
quality planning or for which risk control measures play a special role in the quality plan.
The duration of integration, system, and acceptance test execution depends to a large extent on
the quality of software under test. Software that is sloppily constructed or that undergoes
inadequate analysis and test before commitment to the code base will slow testing progress. Even
if responsibility for diagnosing test failures lies with developers and not with the testing group, a
test execution session that results in many failures and generates many failure reports is
inherently more time consuming than executing a suite of tests with few or no failures. This
schedule vulnerability is yet another reason to emphasize earlier activities, in particular those that
provide early indications of quality problems. Inspection of design and code (with quality team
participation) can help control this risk, and also serves to communicate quality standards and
best practices among the team.
If unit testing is the responsibility of developers, test suites are part of the unit deliverable and
should undergo inspection for correctness, thoroughness, and automation. While functional and
structural coverage criteria are no panacea for measuring test thoroughness, it is reasonable to
require that deviations from basic coverage criteria be justified on a case-by-case basis. A
substantial deviation from the structural coverage observed in similar products may be due to
many causes, including inadequate testing, incomplete specifications, unusual design, or

RANGANATHA 25
 SOFTWARE TESTING 17IS63

implementation decisions. The modules that present unusually low structural coverage should be
inspected to identify the cause.
The cost of analysis and test is multiplied when some requirements demand a very high level of
assurance. For example, if a system that has previously been used in biological research is
modified or redeveloped for clinical use, one should anticipate that all development costs, and
particularly costs of analysis and test, will be an order of magnitude higher. In addition to the
risk of underestimating the cost and schedule impact of stringent quality requirements, the risk of
failing to achieve the required dependability increases. One important tactic for controlling this
risk is isolating critical properties as far as possible in small, simple components. Of course these
aspects of system specification and architectural design are not entirely within control of the
quality team; it is crucial that at least the quality manager, and possibly other members of the
quality team, participate in specification and design activities to assess and communicate the
impact of design alternatives on cost and schedule.
Architectural design is also the primary point of leverage to control cost and risks of testing
systems with complex external interfaces. For example, the hardware platform on which an
embedded system must be tested may be a scarce resource, in demand for debugging as well as
testing. Preparing and executing a test case on that platform may be time-consuming, magnifying
the risk that system and operational testing may go over schedule and delay software delivery.
This risk may be reduced by careful consideration of design-for-testability in architectural
design. A testable design isolates and minimizes platform dependencies, reducing the portion of
testing that requires access to the platform. It will typically provide additional interfaces to
enhance controllability and observability in testing. A considerable investment in test
scaffolding, from self-diagnosis to platform simulators, may also be warranted.
Risks related both to critical requirements and limitations on testability can be partially
addressed in system specifications and programming standards. For example, it is notoriously
difficult to detect race conditions by testing multi-threaded software. However, one may impose
a design and programming discipline that prevents race conditions, such as a simple monitor
discipline with resource ordering. Detecting violations of that discipline, statically and
dynamically, is much simpler than detecting actual data races.
This tactic may be reflected in several places in the project plan, from settling on the
programming discipline in architectural design to checking for proper use of the discipline in
code and design inspections, to implementation or purchase of tools to automate compliance
checking.

Monitoring the Process

RANGANATHA 26
 SOFTWARE TESTING 17IS63

The quality manager monitors progress of quality activities, including results as well as schedule,
to identify deviations from the quality plan as early as possible and take corrective action.
Effective monitoring, naturally, depends on a plan that is realistic, well organized, and
sufficiently detailed with clear, unambiguous milestones and criteria. We say a process is visible
to the extent that it can be effectively monitored.
Successful completion of a planned activity must be distinguished from mere termination, as
otherwise it is too tempting to meet an impending deadline by omitting some planned work.
Skipping planned verification activities or addressing them superficially can seem to accelerate a
late project, but the boost is only apparent; the real effect is to postpone detection of more faults
to later stages in development, where their detection and removal will be far more threatening to
project success.
For example, suppose a developer is expected to deliver unit test cases as part of a work unit. If
project deadlines are slipping, the developer is tempted to scrimp on designing unit tests and
writing supporting code, perhaps dashing off a few superficial test cases so that the unit can be
committed to the code base. The rushed development and inadequate unit testing are nearly
guaranteed to leave bugs that surface later, perhaps in integration or system testing, where they
will have a far greater impact on project schedule. Worst of all, they might be first detected in
operational use, reducing the real and perceived quality of the delivered product. In monitoring
progress, therefore, it is essential to include appropriate metrics of the thoroughness or
completeness of the activity.
Monitoring produces a surfeit of detail about individual activities. Managers need to make
decisions based on an overall understanding of project status, so raw monitoring information
must be aggregated in ways that provide an overall picture.
Risk Management in the Quality Plan: Risks Generic to Process Management
The quality plan must identify potential risks and define appropriate control tactics. Some risks
and control tactics are generic to process management, while others are specific to the quality
process. Here we provide a brief overview of some risks generic to process management. Risks
specific to the quality process are summarized in the sidebar on page 391
Personnel Risks Example Control Tactics
A staff member is lost (becomes ill, Cross train to avoid overdependence on individuals;
changes employer, etc.) or is encourage and schedule continuous education; provide
underqualified for task (the project open communication with opportunities for staff self-
plan assumed a level of skill or assessment and identification of skills gaps early in the
familiarity that the assigned member project; provide competitive compensation and promotion
did not have). policies and a rewarding work environment to retain staff;
include training time in the project schedule.
Technology Risks Example Control Tactics
Many faults are introduced Anticipate and schedule extra time for testing unfamiliar

RANGANATHA 27
 SOFTWARE TESTING 17IS63

interfacing to an unfamiliar interfaces; invest training time for COTS components and
commercial off-the-shelf (COTS) for training with new tools; monitor, document, and
component. publicize common errors and correct idioms; introduce
new tools in lower-risk pilot projects or prototyping
exercises.
Test and analysis automation tools Introduce new tools in lower-risk pilot projects or
do not meet expectations. prototyping exercises; anticipate and schedule time for
training with new tools.
COTS components do not meet Include COTS component qualification testing early in
quality expectations. project plan; introduce new COTS components in lower-
risk pilot projects or prototyping exercises.
Schedule Risks Example Control Tactics
Inadequate unit testing leads to Track and reward quality unit testing as evidenced by low-
unanticipated expense and delays in fault densities in integration.
integration testing.
Difficulty of scheduling meetings Set aside times in a weekly schedule in which inspections
makes inspection a bottleneck in take precedence over other meetings and other work; try
development. distributed and asynchronous inspection techniques, with
a lower frequency of face-to-face inspection meetings.
Open table as spreadsheet

Risk Management in the Quality Plan: Risks Specific to Quality Management

Here we provide a brief overview of some risks specific to the quality process. Risks generic to
process management are summarized in the sidebar at page 390.
Development Risks Example Control Tactics
Poor quality software delivered to Provide early warning and feedback; schedule
testing group or inadequate unit test and inspection of design, code and test suites; connect
analysis before committing to the code development and inspection to the reward system;
base. increase training through inspection; require coverage
or other criteria at unit test level.
Executions Risks Example Control Tactics
Execution costs higher than planned; Minimize parts that require full system to be executed;
scarce resources available for testing inspect architecture to assess and improve testability;
(testing requires expensive or complex increase intermediate feedback; invest in scaffolding.
machines or systems not easily
available.)

RANGANATHA 28
 SOFTWARE TESTING 17IS63

Requirements Risks Example Control Tactics

High assurance critical requirements. Compare planned testing effort with former projects
with similar criticality level to avoid underestimating
testing effort; balance test and analysis; isolate critical
parts, concerns and properties.
Open table as spreadsheet

One key aggregate measure is the number of faults that have been revealed and removed, which
can be compared to data obtained from similar past projects. Fault detection and removal can be
tracked against time and will typically follow a characteristic distribution similar to that shown in
Figure 20.3. The number of faults detected per time unit tends to grow across several system
builds, then to decrease at a much lower rate (usually half the growth rate) until it stabilizes

Figure 8.3: A typical distribution of faults for system builds through time.
An unexpected pattern in fault detection may be a symptom of problems. If detected faults stop
growing earlier than expected, one might hope it indicates exceptionally high quality, but it
would be wise to consider the alternative hypothesis that fault detection efforts are ineffective. A
growth rate that remains high through more than half the planned system builds is a warning that
quality goals may be met late or not at all, and may indicate weaknesses in fault removal or lack
of discipline in development (e.g., a rush to add features before delivery, with a consequent
deemphasis on quality control).
A second indicator of problems in the quality process is faults that remain open longer than
expected. Quality problems are confirmed when the number of open faults does not stabilize at a
level acceptable to stakeholders.
The accuracy with which we can predict fault data and diagnose deviations from expectation
depends on the stability of the software development and quality processes, and on availability of
data from similar projects. Differences between organizations and across application domains are
wide, so by far the most valuable data is from similar projects in one's own organization.
The faultiness data in Figure 8.3 are aggregated by severity levels. This helps in better
understanding the process. Growth in the number of moderate faults late in the development

RANGANATHA 29
 SOFTWARE TESTING 17IS63

process may be a symptom of good use of limited resources concentrated in removing critical
and severe faults, not spent solving moderate problems.
Accurate classification schemata can improve monitoring and may be used in very large projects,
where the amount of detailed information cannot be summarized in overall data. The orthogonal
defect classification (ODC) approach has two main steps: (1) fault classification and (2) fault
analysis.
ODC fault classification is done in two phases: when faults are detected and when they are fixed.
At detection time, we record the activity executed when the fault is revealed, the trigger that
exposed the fault, and the perceived or actual impact of the fault on the customer. A possible
taxonomy for activities and triggers is illustrated in the sidebar at page 395. Notice that triggers
depend on the activity. The sidebar at page 396 illustrates a possible taxonomy of customer
impacts.
At fix time, we record target, type, source, and age of the software. The target indicates the
entity that has been fixed to remove the fault, and can be requirements, design, code,
build/package,or documentation/development. The type indicates the type of the fault.
Taxonomies depend on the target. The sidebar at page 396 illustrates a taxonomy of types of
faults removed from design or code. Fault types may be augmented with an indication of the
nature of the fault, which can be: missing, that is, the fault is to due to an omission, as in a
missing statement; incorrect, as in the use of a wrong parameter; or extraneous, that is, due to
something not relevant or pertinent to the document or code, as in a section of the design
document that is not pertinent to the current product and should be removed. The source of the
fault indicates the origin of the faulty modules: in-house, library, ported from other platforms,or
outsourced code.
The age indicates the age of the faulty element - whether the fault was found in new, old (base),
rewritten,or re-fixed code.
The detailed information on faults allows for many analyses that can provide information on the
development and the quality process. As in the case of analysis of simple faultiness data, the
interpretation depends on the process and the product, and should be based on past experience.
The taxonomy of faults, as well as the analysis of faultiness data, should be refined while
applying the method.
When we first apply the ODC method, we can perform some preliminary analysis using only part
of the collected information:
Distribution of fault types versus activities Different quality activities target different classes
of faults. For example, algorithmic (that is, local) faults are targeted primarily by unit testing,
and we expect a high proportion of faults detected by unit testing to be in this class. If the
proportion of algorithmic faults found during unit testing is unusually small, or a larger than
normal proportion of algorithmic faults are found during integration testing, then one may
reasonably suspect that unit tests have not been well designed. If the mix of faults found during
integration testing contains an unusually high proportion of algorithmic faults, it is also possible
that integration testing has not focused strongly enough on interface faults.

RANGANATHA 30
 SOFTWARE TESTING 17IS63

Distribution of triggers over time during field test Faults corresponding to simple usage
should arise early during field test, while faults corresponding to complex usage should arise
late. In both cases, the rate of disclosure of new faults should asymptotically decrease.
Unexpected distributions of triggers over time may indicate poor system or acceptance test. If
triggers that correspond to simple usage reveal many faults late in acceptance testing, we may
have chosen a sample that is not representative of the user population. If faults continue growing
during acceptance test, system testing may have failed, and we may decide to resume it before
continuing with acceptance testing.
Age distribution over target code Most faults should be located in new and rewritten code,
while few faults should be found in base or re-fixed code, since base and re-fixed code has
already been tested and corrected. Moreover, the proportion of faults in new and rewritten code
with respect to base and re-fixed code should gradually increase. Different patterns may indicate
holes in the fault tracking and removal process or may be a symptom of inadequate test and
analysis that failed in revealing faults early (in previous tests of base or re-fixed code). For
example, an increase of faults located in base code after porting to a new platform may indicate
inadequate tests for portability.
Distribution of fault classes over time The proportion of missing code faults should gradually
decrease, while the percentage of extraneous faults may slowly increase, because missing
functionality should be revealed with use and repaired, while extraneous code or documentation
may be produced by updates. An increasing number of missing faults may be a symptom of
instability of the product, while a sudden sharp increase in extraneous faults may indicate
maintenance problems.

Improving the Process

Many classes of faults that occur frequently are rooted in process and development flaws. For
example, a shallow architectural design that does not take into account resource allocation can
lead to resource allocation faults. Lack of experience with the development environment, which
leads to misunderstandings between analysts and programmers on rare and exceptional cases,
can result in faults in exception handling. A performance assessment system that rewards faster
coding without regard to quality is likely to promote low quality code.
The occurrence of many such faults can be reduced by modifying the process and environment.
For example, resource allocation faults resulting from shallow architectural design can be
reduced by introducing specific inspection tasks. Faults attributable to inexperience with the
development environment can be reduced with focused training sessions. Persistently poor
programming practices may require modification of the reward system.
Often, focused changes in the process can lead to product improvement and significant cost
reduction. Unfortunately, identifying the weak aspects of a process can be extremely difficult,

RANGANATHA 31
 SOFTWARE TESTING 17IS63

and often the results of process analysis surprise even expert managers. The analysis of the fault
history can help software engineers build a feedback mechanism to track relevant faults to their
root causes, thus providing vital information for improving the process. In some cases,
information can be fed back directly into the current product development, but more often it
helps software engineers improve the development of future products. For example, if analysis of
faults reveals frequent occurrence of severe memory management faults in C programs, we
might revise inspection checklists and introduce dynamic analysis tools, but it may be too late to
change early design decisions or select a different programming language in the project
underway. More fundamental changes may be made in future projects.
Root cause analysis (RCA) is a technique for identifying and eliminating process faults. RCA
was first developed in the nuclear power industry and later extended to software analysis. It
consists of four main steps to select significant classes of faults and track them back to their
original causes: What, When, Why, and How.
What are the faults? The goal of this first step is to identify a class of important faults. Faults
are categorized by severity and kind. The severity of faults characterizes the impact of the fault
on the product. Although different methodologies use slightly different scales and terms, all of
them identify a few standard levels, described in Table 8.1
Table 8.1: Standard severity levels for root cause analysis (RCA).
Open table as spreadsheet
Level Description Example
Critical The product is unusable. The fault causes the program to crash.
Severe Some product features cannot be The fault inhibits importing files saved with a
used, and there is no workaround. previous version of the program, and there is no
way to convert files saved in the old format to the
new one.
Moderate Some product features require The fault inhibits exporting in Postscript format.
workarounds to use, and reduce Postscript can be produced using the printing
efficiency, reliability, or facility, but the process is not obvious or
convenience and usability. documented (loss of usability) and requires extra
steps (loss of efficiency).
Cosmetic Minor inconvenience. The fault limits the choice of colors for
customizing the graphical interface, violating the
specification but causing only minor
inconvenience.
The RCA approach to categorizing faults, in contrast to ODC, does not use a predefined set of
categories. The objective of RCA is not to compare different classes of faults over time, or to
analyze and eliminate all possible faults, but rather to identify the few most important classes of
faults and remove their causes. Successful application of RCA progressively eliminates the

RANGANATHA 32
 SOFTWARE TESTING 17IS63

causes of the currently most important faults, which lose importance over time, so applying a
static predefined classification would be useless. Moreover, the precision with which we identify
faults depends on the specific project and process and varies over time.
ODC Classification of Triggers Listed by Activity
Design Review and Code Inspection
Design Conformance A discrepancy between the reviewed artifact and a prior-stage artifact that
serves as its specification.
Logic/Flow An algorithmic or logic flaw.
Backward Compatibility A difference between the current and earlier versions of an artifact
that could be perceived by the customer as a failure.
Internal Document An internal inconsistency in the artifact (e.g., inconsistency between code
and comments).
Lateral Compatibility An incompatibility between the artifact and some other system or
module with which it should interoperate.
Concurrency A fault in interaction of concurrent processes or threads.
Language Dependency A violation of language-specific rules, standards, or best practices.
Side Effects A potential undesired interaction between the reviewed artifact and some other part
of the system.
Rare Situation An inappropriate response to a situation that is not anticipated in the artifact.
(Error handling as specified in a prior artifact design conformance, not rare situation.)
Structural (White-Box) Test
Simple Path The fault is detected by a test case derived to cover a single program element.
Complex Path The fault is detected by a test case derived to cover a combination of program
elements.
Functional (Black-Box) Test
Coverage The fault is detected by a test case derived for testing a single procedure (e.g., C
function or Java method), without considering combination of values for possible parameters.
Variation The fault is detected by a test case derived to exercise a particular combination of
parameters for a single procedure.
Sequencing The fault is detected by a test case derived for testing a sequence of procedure calls.
Interaction The fault is detected by a test case derived for testing procedure interactions.
System Test
Workload/Stress The fault is detected during workload or stress testing.
Recovery/Exception The fault is detected while testing exceptions and recovery procedures.
Startup/Restart The fault is detected while testing initialization conditions during start up or
after possibly faulty shutdowns.
Hardware Configuration The fault is detected while testing specific hardware configurations.
Software Configuration The fault is detected while testing specific software configurations.
Blocked Test Failure occurred in setting up the test scenario.

RANGANATHA 33
 SOFTWARE TESTING 17IS63

ODC Classification of Customer Impact

Installability Ability of the customer to place the software into actual use. (Usability of the
installed software is not included.)
Integrity/Security Protection of programs and data from either accidental or malicious
destruction or alteration, and from unauthorized disclosure.
Performance The perceived and actual impact of the software on the time required for the
customer and customer end users to complete their tasks.
Maintenance The ability to correct, adapt, or enhance the software system quickly and at
minimal cost.
Serviceability Timely detection and diagnosis of failures, with minimal customer impact.
Migration Ease of upgrading to a new system release with minimal disruption to existing
customer data and operations.
Documentation Degree to which provided documents (in all forms, including electronic)
completely and correctly describe the structure and intended uses of the software.
Usability The degree to which the software and accompanying documents can be understood
and effectively employed by the end user.
Standards The degree to which the software complies with applicable standards.
Reliability The ability of the software to perform its intended function without unplanned
interruption or failure.
Accessibility The degree to which persons with disabilities can obtain the full benefit of the
software system.
Capability The degree to which the software performs its intended functions consistently with
documented system requirements.
Requirements The degree to which the system, in complying with document requirements,
actually meets customer expectations
ODC Classification of Defect Types for Targets Design and Code
Assignment/Initialization A variable was not assigned the correct initial value or was not
assigned any initial value.
Checking Procedure parameters or variables were not properly validated before use.
Algorithm/Method A correctness or efficiency problem that can be fixed by reimplementing a
single procedure or local data structure, without a design change.
Function/Class/Object A change to the documented design is required to conform to product
requirements or interface specifications.
Timing/Synchronization The implementation omits necessary synchronization of shared
resources, or violates the prescribed synchronization protocol.
Interface/Object-Oriented Messages Module interfaces are incompatible; this can include
syntactically compatible interfaces that differ in semantic interpretation of communicated data.
Relationship Potentially problematic interactions among procedures, possibly involving
different assumptions but not involving interface incompatibility.

RANGANATHA 34
 SOFTWARE TESTING 17IS63

A good RCA classification should follow the uneven distribution of faults across categories. If,
for example, the current process and the programming style and environment result in many
interface faults, we may adopt a finer classification for interface faults and a coarse-grain
classification of other kinds of faults. We may alter the classification scheme in future projects as
a result of having identified and removed the causes of many interface faults.
Classification of faults should be sufficiently precise to allow identifying one or two most
significant classes of faults considering severity, frequency, and cost of repair. It is important to
keep in mind that severity and repair cost are not directly related. We may have cosmetic faults
that are very expensive to repair, and critical faults that can be easily repaired. When selecting
the target class of faults, we need to consider all the factors. We might, for example, decide to
focus on a class of moderately severe faults that occur very frequently and are very expensive to
remove, investing fewer resources in preventing a more severe class of faults that occur rarely
and are easily repaired.
When did faults occur, and when were they found? It is typical of mature software processes
to collect fault data sufficient to determine when each fault was detected (e.g., in integration test
or in a design inspection). In addition, for the class of faults identified in the first step, we
attempt to determine when those faults were introduced (e.g., was a particular fault introduced in
coding, or did it result from an error in architectural design?).
Why did faults occur? In this core RCA step, we attempt to trace representative faults back to
causes, with the objective of identifying a "root" cause associated with many faults in the class.
Analysis proceeds iteratively by attempting to explain the error that led to the fault, then the
cause of that error, the cause of that cause, and so on. The rule of thumb "ask why six times"
does not provide a precise stopping rule for the analysis, but suggests that several steps may be
needed to find a cause in common among a large fraction of the fault class under consideration.

The 80/20 or Pareto Rule

Fault classification in root cause analysis is justified by the so-called 80/20 or Pareto rule. The
Pareto rule is named for the Italian economist Vilfredo Pareto, who in the early nineteenth
century proposed a mathematical power law formula to describe the unequal distribution of
wealth in his country, observing that 20% of the people owned 80% of the wealth.
Pareto observed that in many populations, a few (20%) are vital and many (80%) are trivial. In
fault analysis, the Pareto rule postulates that 20% of the code is responsible for 80% of the faults.
Although proportions may vary, the rule captures two important facts:
1. Faults tend to accumulate in a few modules, so identifying potentially faulty modules can
improve the cost effectiveness of fault detection.
2. Some classes of faults predominate, so removing the causes of a predominant class of
faults can have a major impact on the quality of the process and of the resulting product.
The predominance of a few classes of faults justifies focusing on one class at a time.

RANGANATHA 35
 SOFTWARE TESTING 17IS63

Tracing the causes of faults requires experience, judgment, and knowledge of the development
process. We illustrate with a simple example. Imagine that the first RCA step identified memory
leaks as the most significant class of faults, combining a moderate frequency of occurrence with
severe impact and high cost to diagnose and repair. The group carrying out RCA will try to
identify the cause of memory leaks and may conclude that many of them result from forgetting to
release memory in exception handlers. The RCA group may trace this problem in exception
handling to lack of information: Programmers can't easily determine what needs to be cleaned
up in exception handlers. The RCA group will ask why once more and may go back to a design
error: The resource management scheme assumes normal flow of control and thus does not
provide enough information to guide implementation of exception handlers. Finally, the RCA
group may identify the root problem in an early design problem: Exceptional conditions were an
afterthought dealt with late in design.
Each step requires information about the class of faults and about the development process that
can be acquired through inspection of the documentation and interviews with developers and
testers, but the key to success is curious probing through several levels of cause and effect.
How could faults be prevented? The final step of RCA is improving the process by removing
root causes or making early detection likely. The measures taken may have a minor impact on
the development process (e.g., adding consideration of exceptional conditions to a design
inspection checklist), or may involve a substantial modification of the process (e.g., making
explicit consideration of exceptional conditions a part of all requirements analysis and design
steps). As in tracing causes, prescribing preventative or detection measures requires judgment,
keeping in mind that the goal is not perfection but cost-effective improvement.
ODC and RCA are two examples of feedback and improvement, which are an important
dimension of most good software processes. Explicit process improvement steps are, for
example, featured in both SRET (sidebar on page 380) and Cleanroom (sidebar on page 378).

The Quality Team

The quality plan must assign roles and responsibilities to people. As with other aspects of
planning, assignment of responsibility occurs at a strategic level and a tactical level. The tactical
level, represented directly in the project plan, assigns responsibility to individuals in accordance
with the general strategy. It involves balancing level of effort across time and carefully managing
personal interactions. The strategic level of organization is represented not only in the quality
strategy document, but in the structure of the organization itself.
The strategy for assigning responsibility may be partly driven by external requirements. For
example, independent quality teams may be required by certification agencies or by a client
organization. Additional objectives include ensuring sufficient accountability that quality tasks
are not easily overlooked; encouraging objective judgment of quality and preventing it from
being subverted by schedule pressure; fostering shared commitment to quality among all team
members; and developing and communicating shared knowledge and values regarding quality.

RANGANATHA 36
 SOFTWARE TESTING 17IS63

Measures taken to attain some objectives (e.g., autonomy to ensure objective assessment) are in
tension with others (e.g., cooperation to meet overall project objectives). It is therefore not
surprising to find that different organizations structure roles and responsibilities in a wide variety
of different ways. The same individuals can play the roles of developer and tester, or most testing
responsibility can be assigned to members of a distinct group, and some may even be assigned to
a distinct organization on a contractual basis. Oversight and accountability for approving the
work product of a task are sometimes distinguished from responsibility for actually performing a
task, so the team organization is somewhat intertwined with the task breakdown.
Each of the possible organizations of quality roles makes some objectives easier to achieve and
some more challenging. Conflict of one kind or another is inevitable, and therefore in organizing
the team it is important to recognize the conflicts and take measures to control adverse
consequences. If an individual plays two roles in potential conflict (e.g., a developer responsible
for delivering a unit on schedule is also responsible for integration testing that could reveal faults
that delay delivery), there must be countermeasures to control the risks inherent in that conflict.
If roles are assigned to different individuals, then the corresponding risk is conflict between the
individuals (e.g., if a developer and a tester do not adequately share motivation to deliver a
quality product on schedule).
An independent and autonomous testing team lies at one end of the spectrum of possible team
organizations. One can make that team organizationally independent so that, for example, a
project manager with schedule pressures can neither bypass quality activities or standards, nor
reallocate people from testing to development, nor postpone quality activities until too late in the
project. Separating quality roles from development roles minimizes the risk of conflict between
roles played by an individual, and thus makes most sense for roles in which independence is
paramount, such as final system and acceptance testing.
An independent team devoted to quality activities also has an advantage in building specific
expertise, such as test design. The primary risk arising from separation is in conflict between
goals of the independent quality team and the developers.
When quality tasks are distributed among groups or organizations, the plan should include
specific checks to ensure successful completion of quality activities. For example, when module
testing is performed by developers and integration and system testing is performed by an
independent quality team, the quality team should check the completeness of module tests
performed by developers, for example, by requiring satisfaction of coverage criteria or inspecting
module test suites. If testing is performed by an independent organization under contract, the
contract should carefully describe the testing process and its results and documentation, and the
client organization should verify satisfactory completion of the contracted tasks.
Existence of a testing team must not be perceived as relieving developers from responsibility for
quality, nor is it healthy for the testing team to be completely oblivious to other pressures,
including schedule pressure. The testing team and development team, if separate, must at least
share the goal of shipping a high-quality product on schedule.

RANGANATHA 37
 SOFTWARE TESTING 17IS63

Independent quality teams require a mature development process to minimize communication

and coordination overhead. Test designers must be able to work on sufficiently precise
specifications and must be able to execute tests in a controllable test environment. Versions and
configurations must be well defined, and failures and faults must be suitably tracked and
monitored across versions.
It may be logistically impossible to maintain an independent quality group, especially in small
projects and organizations, where flexibility in assignments is essential for resource
management. Aside from the logistical issues, division of responsibility creates additional work
in communication and coordination. Finally, quality activities often demand deep knowledge of
the project, particularly at detailed levels (e.g., unit and early integration test). An outsider will
have less insight into how and what to test, and may be unable to effectively carry out the crucial
earlier activities, such as establishing acceptance criteria and reviewing architectural design for
testability. For all these reasons, even organizations that rely on an independent verification and
validation (IV&V) group for final product qualification allocate other responsibilities to
developers and to quality professionals working more closely with the development team.
At the polar opposite from a completely independent quality team is full integration of quality
activities with development, as in some "agile" processes including XP.
Communication and coordination overhead is minimized this way, and developers take full
responsibility for the quality of their work product. Moreover, technology and application
expertise for quality tasks will match the expertise available for development tasks, although the
developer may have less specific expertise in skills such as test design.
The more development and quality roles are combined and intermixed, the more important it is
to build into the plan checks and balances to be certain that quality activities and objective
assessment are not easily tossed aside as deadlines loom. For example, XP practices like "test
first" together with pair programming (sidebar on page 381) guard against some of the inherent
risks of mixing roles. Separate roles do not necessarily imply segregation of quality activities to
distinct individuals.
It is possible to assign both development and quality responsibility to developers, but assign two
individuals distinct responsibilities for each development work product. Peer review is an
example of mixing roles while maintaining independence on an item-by-item basis. It is also
possible for developers and testers to participate together in some activities.
Many variations and hybrid models of organization can be designed. Some organizations have
obtained a good balance of benefits by rotating responsibilities. For example, a developer may
move into a role primarily responsible for quality in one project and move back into a regular
development role in the next. In organizations large enough to have a distinct quality or testing
group, an appropriate balance between independence and integration typically varies across
levels of project organization. At some levels, an appropriate balance can be struck by giving
responsibility for an activity (e.g., unit testing) to developers who know the code best, but with a
separate oversight responsibility shared by members of the quality team. For example, unit tests
may be designed and implemented by developers, but reviewed by a member of the quality team

RANGANATHA 38
 SOFTWARE TESTING 17IS63

for effective automation (particularly, suitability for automated regression test execution as the
product evolves) as well as thoroughness. The balance tips further toward independence at higher
levels of granularity, such as in system and acceptance testing, where at least some tests should
be designed independently by members of the quality team.
Outsourcing test and analysis activities is sometimes motivated by the perception that testing is
less technically demanding than development and can be carried out by lower-paid and lower-
skilled individuals. This confuses test execution, which should in fact be straightforward, with
analysis and test design, which are as demanding as design and programming tasks in
development. Of course, less skilled individuals can design and carry out tests, just as less skilled
individuals can design and write programs, but in both cases the results are unlikely to be
satisfactory.
Outsourcing can be a reasonable approach when its objectives are not merely minimizing cost,
but maximizing independence. For example, an independent judgment of quality may be
particularly valuable for final system and acceptance testing, and may be essential for measuring
a product against an independent quality standard (e.g., qualifying a product for medical or
avionic use). Just as an organization with mixed roles requires special attention to avoid the
conflicts between roles played by an individual, radical separation of responsibility requires
special attention to control conflicts between the quality assessment team and the development
team.
The plan must clearly define milestones and delivery for outsourced activities, as well as checks
on the quality of delivery in both directions: Test organizations usually perform quick checks to
verify the consistency of the software to be tested with respect to some minimal "testability"
requirements; clients usually check the completeness and consistency of test results. For
example, test organizations may ask for the results of inspections on the delivered artifact before
they start testing, and may include some quick tests to verify the installability and testability of
the artifact. Clients may check that tests satisfy specified functional and structural coverage
criteria, and may inspect the test documentation to check its quality. Although the contract
should detail the relation between the development and the testing groups, ultimately,
outsourcing relies on mutual trust between organizations

Documenting Analysis and Test

Mature software processes include documentation standards for all the activities of the software
process, including test and analysis activities. Documentation can be inspected to verify progress
against schedule and quality goals and to identify problems, supporting process visibility,
monitoring, and replicability
Overview
Documentation is an important element of the software development process, including the
quality process. Complete and well-structured documents increase the reusability of test suites

RANGANATHA 39
 SOFTWARE TESTING 17IS63

within and across projects. Documents are essential for maintaining a body of knowledge that
can be reused across projects. Consistent documents provide a basis for monitoring and assessing
the process, both internally and for external authorities where certification is desired. Finally,
documentation includes summarizing and presenting data that forms the basis for process
improvement. Test and analysis documentation includes summary documents designed primarily
for human comprehension and details accessible to the human reviewer but designed primarily
for automated analysis.
Documents are divided into three main categories: planning, specification, and reporting.
Planning documents describe the organization of the quality process and include strategies and
plans for the division or the company, and plans for individual projects. Specification documents
describe test suites and test cases. A complete set of analysis and test specification documents
include test design specifications, test case specification, checklists, and analysis procedure
specifications. Reporting documents include details and summary of analysis and test results.
Organizing Documents
In a small project with a sufficiently small set of documents, the arrangement of other project
artifacts (e.g., requirements and design documents) together with standard content (e.g., mapping
of subsystem test suites to the build schedule) provides sufficient organization to navigate
through the collection of test and analysis documentation. In larger projects, it is common
practice to produce and regularly update a global guide for navigating among individual
documents.
Mature processes require all documents to contain metadata that facilitate their management.
Documents must include some basic information about its context in order to make the document
self-contained, approval indicating the persons responsible for the document and document
history.
Naming conventions help in quickly identifying documents. A typical standard for document
names would include keywords indicating the general scope of the document, its nature, the
specific document, and its version, as in Figure 8.4

RANGANATHA 40
 SOFTWARE TESTING 17IS63

Chipmunk Document Template Document Title

Approvals
issued by name signature date
approved by name signature date
distribution status (internal use only, restricted, …)
distribution list (people to whom the document must be sent)
Open table as spreadsheet
History
version description

Open table as spreadsheet

Table of Contents
 List of sections.
Summary
 Summarize the contents of the document. The summary should clearly explain the
relevance of the document to its possible uses.

RANGANATHA 41
 SOFTWARE TESTING 17IS63

Goals of the document

 Describe the purpose of this document: Who should read it, and why?
Required documents and references
 Provide a reference to other documents and artifacts needed for understanding and
exploiting this document. Provide a rationale for the provided references.
Glossary
 Provide a glossary of terms required to understand this document.
Section 1
 …
Section N
 …

Test Strategy Document

Analysis and test strategies describe quality guidelines for sets of projects, usually for an entire
company or organization. Strategies, and therefore strategy documents, vary widely among
organizations, but we can identify a few key elements that should be included in almost any
well-designed strategy document. These are illustrated in the document excerpt on page 459.
Strategy documents indicate common quality requirements across products. Requirements may
depend on business conditions. For example, a company that produces safety-critical software
may need to satisfy minimum dependability requirements defined by a certification authority,
while a department that designs software embedded in hardware products may need to ensure
portability across product lines. Some requirements on dependability and usability may be
necessary to maintain brand image and market position. For example, a company might decide to
require conformance to W3C-WAI accessibility standards uniformly across the product line.
The strategy document sets out requirements on other quality documents, typically including an
analysis and test plan, test design specifications, test case specifications, test logs, and test
summary reports. Basic document requirements, such as naming and versioning, follow
standards for other project documentation, but quality documents may have additional,
specialized requirements. For example, testing logs for avionics software may be required to
contain references to the version of the simulator used for executing the test before installing the
software on board the aircraft.

Analysis and Test Plan

While the format of an analysis and test strategy vary from company to company, the structure of
an analysis and test plan is more standardized. A typical structure of a test and analysis plan
includes information about items to be verified, features to be tested, the testing approach, pass
and fail criteria, test deliverables, tasks, responsibilities and resources, and environment
constraints. Basic elements are described in the sidebar on page 461.

RANGANATHA 42
 SOFTWARE TESTING 17IS63

The overall quality plan usually comprises several individual plans of limited scope. Each test
and analysis plan should indicate the items to be verified through analysis or testing. They may
include specifications or documents to be inspected, code to be analyzed or tested, and interface
specifications to undergo consistency analysis. They may refer to the whole system or part of it -
like a subsystem or a set of units. Where the project plan includes planned development
increments, the analysis and test plan indicates the applicable versions of items to be verified.
For each item, the plan should indicate any special hardware or external software required for
testing. For example, the plan might indicate that one suite of subsystem tests for a security
package can be executed with a software simulation of a smart card reader, while another suite
requires access to the physical device. Finally, for each item, the plan should reference related
documentation, such as requirements and design specifications, and user, installation, and
operations guides.
An Excerpt of the Chipmunk Analysis and Test Strategy
Document CP05-14.03: Analysis and Test Strategy
…
Applicable Standards and Procedures
Artifact Applicable Standards and Guidelines
Web application Accessibility: W3C-WAI …
Reusable component (internally developed) Inspection procedure: [WB12-03.12]
External component Qualification procedure: [WB12-22.04]
Open table as spreadsheet
…
Documentation Standards
Project documents must be archived according to the standard Chipmunk archive procedure
[WB02-01.02]. Standard required documents include
Document Content & Organization Standard
Quality plan [WB06-01.03]
Test design specifications [WB07-01.01] (per test suite)
Test case specifications [WB08-01.07] (per test suite)
Test logs [WB10-02.13]
Test summary reports [WB11-01.11]
Inspection reports [WB12-09.01]
Open table as spreadsheet
Analysis and Test Activities
Tools

RANGANATHA 43
 SOFTWARE TESTING 17IS63

The following tools are approved and should be used in all development projects. Exceptions
require configuration committee approval and must be documented in the project plan.
Fault logging Chipmunk BgT [WB10-23.01]
…
Open table as spreadsheet
Staff and Roles
A development work unit consists of unit source code, including unit test cases, stubs, and
harnesses, and unit test documentation. A unit may be committed to the project baseline when
the source code, test cases, and test results have passed peer review.
References
[WB02-01.02] Archive Procedure [WB06-01.03] Quality Plan Guidelines
[WB07-01.01] Test Design Specifications Guidelines [WB08-01.07] Test Case Specifications
Guidelines
[WB11-01.11] Summary Reports Template [WB10-02.13] Test Log Template
[WB11-09.01] Inspection Report Template [WB12-03.12] Standard Inspection
Procedures
[WB12-22.04] Quality Procedures for Software [WB12-23.01] BgT Installation Manual
Developed by Third Parties and User Guide
Open table as spreadsheet
…

A test and analysis plan may not address all aspects of software quality and testing activities. It
should indicate the features to be verified and those that are excluded from consideration (usually
because responsibility for them is placed elsewhere). For example, if the item to be verified
includes a graphical user interface, the test and analysis plan might state that it deals only with
functional properties and not with usability, which is to be verified separately by a usability and
human interface design team.
Explicit indication of features not to be tested, as well as those included in an analysis and test
plan, is important for assessing completeness of the overall set of analysis and test activities.
Assumption that a feature not considered in the current plan is covered at another point is a
major cause of missing verification in large projects.
The quality plan must clearly indicate criteria for deciding the success or failure of each planned
activity, as well as the conditions for suspending and resuming analysis and test.
Plans define items and documents that must be produced during verification. Test deliverables
are particularly important for regression testing, certification, and process improvement.
The core of an analysis and test plan is a detailed schedule of tasks. The schedule is usually
illustrated with GANTT and PERT diagrams showing the relation among tasks as well as their

RANGANATHA 44
 SOFTWARE TESTING 17IS63

relation to other project milestones. The schedule includes the allocation of limited resources
(particularly staff) and indicates responsibility for reresources and responsibilities sults.
A quality plan document should also include an explicit risk plan with contingencies. As far as
possible, contingencies should include unambiguous triggers (e.g., a date on which a contingency
is activated if a particular task has not be completed) as well as recovery procedures.
Finally, the test and analysis plan should indicate scaffolding, oracles, and any other software or
hardware support required for test and analysis activities

Test Design Specification Documents

Design documentation for test suites and test cases serve essentially the same purpose as other
software design documentation, guiding further development and preparing for maintenance.
Test suite design must include all the information needed for initial selection of test cases and
maintenance of the test suite over time, including rationale and anticipated evolution.
Specification of individual test cases includes purpose, usage, and anticipated changes.
A Standard Organization of an Analysis and Test Plan
Analysis and test items:
 The items to be tested or analyzed. The description of each item indicates version and
installation procedures that may be required.

Features to be tested:
 The features considered in the plan.
Features not to be tested:
 Features not considered in the current plan.
Approach:
 The overall analysis and test approach, sufficiently detailed to permit identification of the
major test and analysis tasks and estimation of time and resources.
Pass/Fail criteria:
 Rules that determine the status of an artifact subjected to analysis and test.
Suspension and resumption criteria:
 Conditions to trigger suspension of test and analysis activities (e.g., an excessive failure
rate) and conditions for restarting or resuming an activity.
Risks and contingencies:
 Risks foreseen when designing the plan and a contingency plan for each of the identified
risks.
Deliverables:
 A list all A&T artifacts and documents that must be produced.
Task and schedule:

RANGANATHA 45
 SOFTWARE TESTING 17IS63

 A complete description of analysis and test tasks, relations among them, and relations
between A&T and development tasks, with resource allocation and constraints. A task
schedule usually includes GANTT and PERT diagrams.
Staff and responsibilities:
 Staff required for performing analysis and test activities, the required skills, and the
allocation of responsibilities among groups and individuals. Allocation of resources to
tasks is described in the schedule.
Environmental needs:
 Hardware and software required to perform analysis or testing activities.

Test design specification documents describe complete test suites (i.e., sets of test cases that
focus on particular aspects, elements, or phases of a software project). They may be divided into
unit, integration, system, and acceptance test suites, if we organize them by the granularity of the
tests, or functional, structural, and performance test suites, if the primary organization is based
on test objectives. A large project may include many test design specifications for test suites of
different kinds and granularity, and for different versions or configurations of the system and its
components. Each specification should be uniquely identified and related to corresponding
project documents, as illustrated in the sidebar on page 463.
Test design specifications identify the features they are intended to verify and the approach used
to select test cases. Features to be tested should be cross-referenced to relevant parts of a
software specification or design document.
A test design specification also includes description of the testing procedure and pass/fail
criteria. The procedure indicates steps required to set up the testing environment and perform the
tests, and includes references to scaffolding and oracles. Pass/fail criteria distinguish success
from failure of a test suite as a whole. In the simplest case a test suite execution may be
determined to have failed if any individual test case execution fails, but in system and acceptance
testing it is common to set a tolerance level that may depend on the number and severity of
failures.
A test design specification logically includes a list of test cases. Test case specifications may be
physically included in the test design specification document, or the logical inclusion may be
implemented by some form of automated navigation. For example, a navigational index can be
constructed from references in test case specifications.
Individual test case specifications elaborate the test design for each individual test case, defining
test inputs, required environmental conditions and procedures for test execution, as well as
expected outputs or behavior. The environmental conditions may include hardware and software
as well as any other requirements. For example, while most tests should be executed
automatically without human interaction, intervention of personnel with certain special skills
(e.g., a device operator) may be an environmental requirement for some.
A test case specification indicates the item to be tested, such as a particular module or product
feature. It includes a reference to the corresponding test design document and describes any

RANGANATHA 46
 SOFTWARE TESTING 17IS63

dependence on execution of other test cases. Like any standard document, a test case
specification is labeled with a unique identifier.

Test and Analysis Reports

Reports of test and analysis results serve both developers and test designers. They identify open
faults for developers and aid in scheduling fixes and revisions. They help test designers assess
and refine their approach, for example, noting when some class of faults is escaping early test
and analysis and showing up only in subsystem and system testing.

Functional Test Design Specification of check configuration

Test Suite Identifier
WB07-15.01
Features to Be Tested
Functional test for check configuration, module specification WB02-15.32.[a]
Approach
Combinatorial functional test of feature parameters, enumerated by category- partition method
over parameter table on page 3 of this document.[b]
Procedure
Designed for conditional inclusion in nightly test run. Build target T02 15 32 11 includes JUnit
harness and oracles, with test reports directed to standard test log. Test environment includes
table MDB 15 32 03 for loading initial test database state.
Test cases[c]
WB07-15.01.C01 malformed model number
WB07-15.01.C02 model number not in DB
… …
WB07- valid model number with all legal required slots and some legal optional
15.01.C09[d] slots
… …
WB07-15.01.C19 empty model DB
WB07-15.01.C23 model DB with a single element
WB07-15.01.C24 empty component DB
WB07-15.01.C29 component DB with a single element
[d]
See sample test case specification, page 464.
Open table as spreadsheet
Pass/Fail Criterion

RANGANATHA 47
 SOFTWARE TESTING 17IS63

Successful completion requires correct execution of all test cases with no violations in test log.
Test Case Specification for check configuration
Test Case Identifier
WB07-15.01.C09[a]
Test items
Module check configuration of the Chipmunk Web presence system, business logic subsystem.
Input specification
Test Case Specification:
Model No. valid
No. of required slots for selected model (#SMRS) many
No. of optional slots for selected model (#SMOS) many
Correspondence of selection with model slots complete
No. of required components with selection ≠ empty = No. of required slots
No. of optional components with select ≠ empty < No. of optional slots
Required component selection all valid
Optional component selection all valid
No. of models in DB many
No. of components in DB many
Open table as spreadsheet
Test case:
Model number Chipmunk C20
#SMRS 5
Screen 13"
Processor Chipmunk II plus
Hard disk 30 GB
RAM 512 MB
OS RodentOS 3.2 Personal Edition
#SMOS 4
External storage device DVD player
Open table as spreadsheet

Output Specification
 return value valid

RANGANATHA 48
 SOFTWARE TESTING 17IS63

Environment Needs
Execute with ChipmunkDBM v3.4 database initialized from table MDB 15 32 03.
Special Procedural Requirements
 none
Intercase Dependencies
 none

A prioritized list of open faults is the core of an effective fault handling and repair procedure.
Failure reports must be consolidated and categorized so that repair effort can be managed
systematically, rather than jumping erratically from problem to problem and wasting time on
duplicate reports. They must be prioritized so that effort is not squandered on faults of relatively
minor importance while critical faults are neglected or even forgotten.
Other reports should be crafted to suit the particular needs of an organization and project,
including process improvement. Summary reports serve primarily to track progress and status.
They may be as simple as confirmation that the nightly build-and-test cycle ran successfully with
no new failures, or they may provide somewhat more information to guide attention to potential
trouble spots. Detailed test logs are designed for selective reading, and include summary tables
that typically include the test suites executed, the number of failures, and a breakdown of failures
into those repeated from prior test execution, new failures, and test cases that previously failed
but now execute correctly.
In some domains, such as medicine or avionics, the content and form of test logs may be
prescribed by a certifying authority. For example, some certifications require test execution logs
signed by both the person who performed the test and a quality inspector, who ascertains
conformance of the test execution with test specifications.

Planning and Monitoring the Process And Documenting Analysis and Test

Any complex process requires planning and monitoring. The quality process requires
coordination of many different activities over a period that spans a full development cycle and
beyond. Planning is necessary to order, provision, and coordinate all the activities that support a
quality goal, and monitoring of actual status against a plan is required to steer and adjust the
process.

Overview
Planning involves scheduling activities, allocating resources, and devising observable,
unambiguous milestones against which progress and performance can be monitored. Monitoring
means answering the question, "How are we doing?"

RANGANATHA 49
 SOFTWARE TESTING 17IS63

Quality planning is one aspect of project planning, and quality processes must be closely
coordinated with other development processes. Coordination among quality and development
tasks may constrain ordering (e.g., unit tests are executed after creation of program units). It may
shape tasks to facilitate coordination; for example, delivery may be broken into smaller
increments to allow early testing. Some aspects of the project plan, such as feedback and design
for testability, may belong equally to the quality plan and other aspects of the project plan.
Quality planning begins at the inception of a project and is developed with the overall project
plan, instantiating and building on a quality strategy that spans several projects. Like the overall
project plan, the quality plan is developed incrementally, beginning with the feasibility study and
continuing through development and delivery.
Formulation of the plan involves risk analysis and contingency planning. Execution of the plan
involves monitoring, corrective action, and planning for subsequent releases and projects.
Allocating responsibility among team members is a crucial and difficult part of planning. When
one person plays multiple roles, explicitly identifying each responsibility is still essential for
ensuring that none are neglected.

Quality and Process

A software plan involves many intertwined concerns, from schedule to cost to usability and
dependability. Despite the intertwining, it is useful to distinguish individual concerns and
objectives to lessen the likelihood that they will be neglected, to allocate responsibilities, and to
make the overall planning process more manageable.
For example, a mature software project plan will include architectural design reviews, and the
quality plan will allocate effort for reviewing testability aspects of the structure and build order.
Clearly, design for testability is an aspect of software design and cannot be carried out by a
separate testing team in isolation. It involves both test designers and other software designers in
explicitly evaluating testability as one consideration in selecting among design alternatives. The
objective of incorporating design for testability in the quality process is primarily to ensure that it
is not overlooked and secondarily to plan activities that address it as effectively as possible.
An appropriate quality process follows a form similar to the overall software process in which it
is embedded. In a strict (and unrealistic) waterfall software process, one would follow the "V
model" in a sequential manner, beginning unit testing only as implementation commenced
following completion of the detailed design phase, and finishing unit testing before integration
testing commenced. In the XP "test first" method, unit testing is conflated with subsystem and
system testing. A cycle of test design and test execution is wrapped around each small-grain
incremental development step. The role that inspection and peer reviews would play in other
processes is filled in XP largely by pair programming. A typical spiral process model lies
somewhere between, with distinct planning, design, and implementation steps in several
increments coupled with a similar unfolding of analysis and test activities.

RANGANATHA 50
 SOFTWARE TESTING 17IS63

A general principle, across all software processes, is that the cost of detecting and repairing a
fault increases as a function of time between committing an error and detecting the resultant
faults. Thus, whatever the intermediate work products in software plan, an efficient quality plan
will include a matched set of intermediate validation and verification activities that detect most
faults within a short period of their introduction. Any step in a software process that is not paired
with a validation or verification step is an opportunity for defects to fester, and any milestone in
a project plan that does not include a quality check is an opportunity for a misleading assessment
of progress.
The particular verification or validation step at each stage depends on the nature of the
intermediate work product and on the anticipated defects. For example, anticipated defects in a
requirements statement might include incompleteness, ambiguity, inconsistency, and over
ambition relative to project goals and resources. A review step might address some of these, and
automated analyses might help with completeness and consistency checking.
The evolving collection of work products can be viewed as a set of descriptions of different parts
and aspects of the software system, at different levels of detail. Portions of the implementation
have the useful property of being executable in a conventional sense, and are the traditional
subject of testing, but every level of specification and design can be both the subject of
verification activities and a source of information for verifying other artifacts. A typical
intermediate artifact - say, a subsystem interface definition or a database schema - will be subject
to the following steps:
Internal consistency check Check the artifact for compliance with structuring rules that define
"well-formed" artifacts of that type. An important point of leverage is defining the syntactic and
semantic rules thoroughly and precisely enough that many common errors result in detectable
violations. This is analogous to syntax and strong-typing rules in programming languages, which
are not enough to guarantee program correctness but effectively guard against many simple
errors.
External consistency check Check the artifact for consistency with related artifacts. Often this
means checking for conformance to a "prior" or "higher-level" specification, but consistency
checking does not depend on sequential, top-down development - all that is required is that the
related information from two or more artifacts be defined precisely enough to support detection
of discrepancies.
Consistency usually proceeds from broad, syntactic checks to more detailed and expensive
semantic checks, and a variety of automated and manual verification techniques may be applied.
Generation of correctness conjectures Correctness conjectures, which can be test outcomes or
other objective criteria, lay the groundwork for external consistency checks of other work
products, particularly those that are yet to be developed or revised. Generating correctness
conjectures for other work products will frequently motivate refinement of the current product.
For example, an interface definition may be elaborated and made more precise so that
implementations can be effectively tested.

RANGANATHA 51
 SOFTWARE TESTING 17IS63

Test and Analysis Strategies

Lessons of past experience are an important asset of organizations that rely heavily on technical
skills. A body of explicit knowledge, shared and refined by the group, is more valuable than
islands of individual competence. Organizational knowledge in a shared and systematic form is
more amenable to improvement and less vulnerable to organizational change, including the loss
of key individuals. Capturing the lessons of experience in a consistent and repeatable form is
essential for avoiding errors, maintaining consistency of the process, and increasing development
efficiency.
Cleanroom
The Cleanroom process model, introduced by IBM in the late 1980s, pairs development with
V&V activities and stresses analysis over testing in the early phases. Testing is left for system
certification. The Cleanroom process involves two cooperating teams, the development and the
quality teams, and five major activities: specification, planning, design and verification, quality
certification, and feedback

In the specification activity, the development team defines the required behavior of the system,
while the quality team defines usage scenarios that are later used for deriving system test suites.
The planning activity identifies incremental development and certification phases.
After planning, all activities are iterated to produce incremental releases of the system. Each
system increment is fully deployed and certified before the following step. Design and code
undergo formal inspection ("Correctness verification") before release. One of the key premises

RANGANATHA 52
 SOFTWARE TESTING 17IS63

underpinning the Cleanroom process model is that rigorous design and formal inspection
produce "nearly fault-free software."
Usage profiles generated during specification are applied in the statistical testing activity to
gauge quality of each release. Another key assumption of the Cleanroom process model is that
usage profiles are sufficiently accurate that statistical testing will provide an accurate measure of
quality as perceived by users.[a] Reliability is measured in terms of mean time between failures
(MTBF) and is constantly controlled after each release. Failures are reported to the development
team for correction, and if reliability falls below an acceptable range, failure data is used for
process improvement before the next incremental release.

Software organizations can develop useful, organization-specific strategies because of

similarities among projects carried out by a particular organization in a set of related application
domains. Test and analysis strategies capture commonalities across projects and provide
guidelines for maintaining consistency among quality plans.
A strategy is distinguished from a plan in that it is not specific to a single project. Rather, it
provides guidance and a general framework for developing quality plans for several projects,
satisfying organizational quality standards, promoting homogeneity across projects, and making
both the creation and execution of individual project quality plans more efficient.
The quality strategy is an intellectual asset of an individual organization prescribing a set of
solutions to problems specific to that organization. Among the factors that particularize the
strategy are:
Structure and size Large organizations typically have sharper distinctions between development
and quality groups, even if testing personnel are assigned to development teams. In smaller
organizations, it is more common for a single person to serve multiple roles. Where
responsibility is distributed among more individuals, the quality strategy will require more
elaborate attention to coordination and communication, and in general there will be much greater
reliance on documents to carry the collective memory.
In a smaller organization, or an organization that has devolved responsibility to small, semi-
autonomous teams, there is typically less emphasis on formal communication and documents but
a greater emphasis on managing and balancing the multiple roles played by each team member.
Overall process We have already noted the intertwining of quality process with other aspects of
an overall software process, and this is of course reflected in the quality strategy. For example, if
an organization follows the Cleanroom methodology, then inspections will be required but unit
testing forbidden. An organization that adopts the XP methodology is likely to follow the "test
first" and pair programming elements of that approach, and in fact would find a more document-
heavy approach a difficult fit.
Notations, standard process steps, and even tools can be reflected in the quality strategy to the
extent they are consistent from project to project. For example, if an organization consistently
uses a particular combination of UML diagram notations to document subsystem interfaces, then
the quality strategy might include derivation of test designs from those notations, as well as

RANGANATHA 53
 SOFTWARE TESTING 17IS63

review and analysis steps tailored to detect the most common and important design flaws at that
point. If a particular version and configuration control system is woven into process
management, the quality strategy will likewise exploit it to support and enforce quality process
steps.
Application domain The domain may impose both particular quality objectives (e.g., privacy
and security in medical records processing), and in some cases particular steps and
documentation required to obtain certification from an external authority. For example, the
RTCA/DO-178B standard for avionics software requires testing to the modified
condition/decision coverage (MC/DC) criterion.

SRET
The software reliability engineered testing (SRET) approach, developed at AT&T in the early
1990s, assumes a spiral development process and augments each coil of the spiral with rigorous
testing activities. SRET identifies two main types of testing: development testing, used to find
and remove faults in software at least partially developed in-house, and certification testing, used
to either accept or reject outsourced software.
The SRET approach includes seven main steps. Two initial, quick decision-making steps
determine which systems require separate testing and which type of testing is needed for each
system to be tested. The five core steps are executed in parallel with each coil of a spiral
development process.

The five core steps of SRET are:

Define "Necessary" Reliability Determine operational models, that is, distinct patterns of
system usage that require separate testing, classify failures according to their severity, and

RANGANATHA 54
 SOFTWARE TESTING 17IS63

engineer the reliability strategy with fault prevention, fault removal, and fault tolerance
activities.
Develop Operational Profiles Develop both overall profiles that span operational models and
operational profiles within single operational models.
Prepare for Testing Specify test cases and procedures.
Execute Tests
Interpret Failure Data Interpretation of failure data depends on the type of testing. In
development testing, the goal is to track progress and compare present failure intensities with
objectives. In certification testing, the goal is to determine if a software component or system
should be accepted or rejected.

Extreme Programming (XP)

The extreme programming methodology (XP) emphasizes simplicity over generality, global
vision and communication over structured organization, frequent changes over big releases,
continuous testing and analysis over separation of roles and responsibilities, and continuous
feedback over traditional planning.
Customer involvement in an XP project includes requirements analysis (development,
refinement, and prioritization of user stories) and acceptance testing of very frequent iterative
releases. Planning is based on prioritization of user stories, which are implemented in short
iterations. Test cases corresponding to scenarios in user stories serve as partial specifications.

Test cases suitable for batch execution are part of the system code base and are implemented
prior to the implementation of features they check ("test-first"). Developers work in pairs,
incrementally developing and testing a module. Pair programming effectively conflates a review
activity with coding. Each release is checked by running all the tests devised up to that point of
development, thus essentially merging unit testing with integration and system testing. A failed
acceptance test is viewed as an indication that additional unit tests are needed.

RANGANATHA 55
 SOFTWARE TESTING 17IS63

Although there are no standard templates for analysis and test strategies, we can identify a few
elements that should be part of almost any good strategy. A strategy should specify common
quality requirements that apply to all or most products, promoting conventions for
unambiguously stating and measuring them, and reducing the likelihood that they will be
overlooked in the quality plan for a particular project. A strategy should indicate a set of
documents that is normally produced during the quality process, and their contents and
relationships. It should indicate the activities that are prescribed by the overall process
organization. Often a set of standard tools and practices will be prescribed, such as the interplay
of a version and configuration control tool with review and testing procedures. In addition, a
strategy includes guidelines for project staffing and assignment of roles and responsibilities.

Test and Analysis Plans

An analysis and test plan details the steps to be taken in a particular project. A plan should
answer the following questions:
 What quality activities will be carried out?
 What are the dependencies among the quality activities and between quality and
development activities?
 What resources are needed and how will they be allocated?
 How will both the process and the evolving product be monitored to maintain an
adequate assessment of quality and early warning of quality and schedule problems?
Each of these issues is addressed to some extent in the quality strategy, but must be elaborated
and particularized. This is typically the responsibility of a quality manager, who should
participate in the initial feasibility study to identify quality goals and estimate the contribution of
test and analysis tasks on project cost and schedule.
To produce a quality plan that adequately addresses the questions above, the quality manager
must identify the items and features to be verified, the resources and activities that are required,
the approaches that should be followed, and criteria for evaluating the results.
Items and features to be verified circumscribe the target of the quality plan. While there is an
obvious correspondence between items to be developed or modified and those to undergo testing,
they may differ somewhat in detail. For example, overall evaluation of the user interface may be
the purview of a separate human factors group. The items to be verified, moreover, include many
intermediate artifacts such as requirements specifications and design documents, in addition to
portions of the delivered system. Approaches to be taken in verification and validation may vary
among items. For example, the plan may prescribe inspection and testing for all items and
additional static analyses for multi-threaded subsystems.
Quality goals must be expressed in terms of properties satisfied by the product and must be
further elaborated with metrics that can be monitored during the course of the project. For
example, if known failure scenarios are classified as critical, severe, moderate, and minor, then

RANGANATHA 56
 SOFTWARE TESTING 17IS63

we might decide in advance that a product version may enter end-user acceptance testing only
when it has undergone system testing with no outstanding critical or severe failures.
Defining quality objectives and process organization in detail requires information that is not all
available in the early stages of development. Test items depend on design decisions; detailed
approaches to evaluation can be defined only after examining requirements and design
specifications; tasks and schedule can be completed only after the design; new risks and
contingencies may be introduced by decisions taken during development. On the other hand, an
early plan is necessary for estimating and controlling cost and schedule. The quality manager
must start with an initial plan based on incomplete and tentative information, and incrementally
refine the plan as more and better information becomes available during the project.
After capturing goals as well as possible, the next step in construction of a quality plan is to
produce an overall rough list of tasks. The quality strategy and past experience provide a basis
for customizing the list to the current project and for scaling tasks appropriately. For example,
experience (preferably in the form of collected and analyzed data from past projects, rather than
personal memory) might suggest a ratio of 3:5 for person-months of effort devoted to integration
test relative to coding effort. Historical data may also provide scaling factors for the application
domain, interfaces with externally developed software, and experience of the quality staff. To the
extent possible, the quality manager must break large tasks into component subtasks to obtain
better estimates, but it is inevitable that some task breakdown must await further elaboration of
the overall project design and schedule.
The manager can start noting dependencies among the quality activities and between them and
other activities in the overall project, and exploring arrangements of tasks over time. The main
objective at this point is to schedule quality activities so that assessment data are provided
continuously throughout the project, without unnecessary delay of other development activities.
For example, the quality manager may note that the design and implementation of different
subsystems are scheduled in different phases, and may plan subsystem testing accordingly.
Where there is a choice between scheduling a quality activity earlier or later, the earliest point
possible is always preferable. However, the demand on resources (staff time, primarily) must be
leveled over time, and often one must carefully schedule the availability of particular critical
resources, such as an individual test designer with expertise in a particular technology.
Maintaining a consistent level of effort limits the number of activities that can be carried on
concurrently, and resource constraints together with the objective of minimizing project delays
tends to force particular orderings on tasks.
If one has a choice between completing two tasks in four months, or completing the first task in
two months and then the second in another two months, the schedule that brings one task to
completion earlier is generally advantageous from the perspective of process visibility, as well as
reduced coordination overhead. However, many activities demand a fraction of a person's
attention over a longer period and cannot be compressed. For example, participation in design
and code inspection requires a substantial investment of effort, but typically falls short of a full-
time assignment.

RANGANATHA 57
 SOFTWARE TESTING 17IS63

Since delayed inspections can be a bottleneck in progress of a project, they should have a high
priority when they can be carried out, and are best interleaved with tasks that can be more
flexibly scheduled.
While the project plan shows the expected schedule of tasks, the arrangement and ordering of
tasks are also driven by risk. The quality plan, like the overall project plan, should include an
explicit risk plan that lists major risks and contingencies.
A key tactic for controlling the impact of risk in the project schedule is to minimize the
likelihood that unexpected delay in one task propagates through the whole schedule and delays
project completion. One first identifies the critical paths through the project schedule. Critical
paths are chains of activities that must be completed in sequence and that have maximum overall
duration. Tasks on the critical path have a high priority for early scheduling, and likewise the
tasks on which they depend (which may not themselves be on the critical path) should be
scheduled early enough to provide some schedule slack and prevent delay in the inception of the
critical tasks.
A critical dependence occurs when a task on a critical path is scheduled immediately after some
other task on the critical path, particularly if the length of the critical path is close to the length of
the project. Critical dependence may occur with tasks outside the quality plan part of the overall
project plan.
The primary tactic available for reducing the schedule risk of a critical dependence is to
decompose a task on the critical path, factoring out subtasks that can be performed earlier. For
example, an acceptance test phase late in a project is likely to have a critical dependence on
development and system integration. One cannot entirely remove this dependence, but its
potential to delay project completion is reduced by factoring test design from test execution.
Figure 8.1 shows alternative schedules for a simple project that starts at the beginning of January
and must be completed by the end of May. In the top schedule, indicated as CRITICAL
SCHEDULE, the tasks Analysis and design, Code and Integration, Design and execute
subsystem tests, and Design and execute system tests form a critical path that spans the duration
of the entire project. A delay in any of the activities will result in late delivery. In this schedule,
only the Produce user documentation task does not belong to the critical path, and thus only
delays of this task can be tolerated

RANGANATHA 58
 SOFTWARE TESTING 17IS63

Figure 8.1: Three possible simple schedules with different risks and resource allocation. The bars indicate
the duration of the tasks. Diamonds indicate milestones, and arrows between bars indicate precedence
between tasks.
In the middle schedule, marked as UNLIMITED RESOURCES, the test design and execution
activities are separated into distinct tasks. Test design tasks are scheduled early, right after
analysis and design, and only test execution is scheduled after Code and integration. In this way

RANGANATHA 59
 SOFTWARE TESTING 17IS63

the tasks Design subsystem tests and Design system tests are removed from the critical path,
which now spans 16 weeks with a tolerance of 5 weeks with respect to the expected termination
of the project. This schedule assumes enough resources for running Code and integration,
Production of user documentation, Design of subsystem tests, and Design of system tests.
The LIMITED RESOURCES schedule at the bottom of Figure 8.1 rearranges tasks to meet
resource constraints. In this case we assume that test design and execution, and production of
user documentation share the same resources and thus cannot be executed in parallel. We can see
that, despite the limited parallelism, decomposing testing activities and scheduling test design
earlier results in a critical path of 17 weeks, 4 weeks earlier than the expected termination of the
project. Notice that in the example, the critical path is formed by the tasks Analysis and design,
Design subsystem tests, Design system tests, Produce user documentation, Execute subsystem
tests, and Execute system tests. In fact, the limited availability of resources results in
dependencies among Design subsystem tests, Design system tests and Produce user
documentation that last longer than the parallel task Code and integration.
The completed plan must include frequent milestones for assessing progress. A rule of thumb is
that, for projects of a year or more, milestones for assessing progress should occur at least every
three months. For shorter projects, a reasonable maximum interval for assessment is one quarter
of project duration.
Figure 8.2 shows a possible schedule for the initial analysis and test plan for the business logic of
the Chipmunk Web presence in the form of a GANTT diagram. In the initial plan, the manager
has allocated time and effort to inspections of all major artifacts, as well as test design as early as
practical and ongoing test execution during development. Division of the project into major parts
is reflected in the plan, but further elaboration of tasks associated with units and smaller
subsystems must await corresponding elaboration of the architectural design. Thus, for example,
inspection of the shopping facilities code and the unit test suites is shown as a single aggregate
task. Even this initial plan does reflect the usual Chipmunk development strategy of regular
"synch and stabilize" periods punctuating development, and the initial quality plan reflects the
Chipmunk strategy of assigning responsibility for producing unit test suites to developers, with
review by a member of the quality team

RANGANATHA 60
 SOFTWARE TESTING 17IS63

Figure 8.2: Initial schedule for quality activities in development of the business logic subsystem
of the Chipmunk Web presence, presented as a GANTT diagram.
The GANTT diagram shows four main groups of analysis and test activities: design inspection,
code inspection, test design, and test execution. The distribution of activities over time is
constrained by resources and dependence among activities. For example, system test execution
starts after completion of system test design and cannot finish before system integration (the sync
and stablize elements of development framework) is complete. Inspection activities are
constrained by specification and design activities. Test design activities are constrained by
limited resources. Late scheduling of the design of integration tests for the administrative
business logic subsystem is necessary to avoid overlap with design of tests for the shopping
functionality subsystem.
The GANTT diagram does not highlight intermediate milestones, but we can easily identify two
in April and July, thus dividing the development into three main phases. The first phase (January
to April) corresponds to requirements analysis and architectural design activities and terminates
with the architectural design baseline. In this phase, the quality team focuses on design
inspection and on the design of acceptance and system tests. The second phase (May to July)
corresponds to subsystem design and to the implementation of the first complete version of the
system. It terminates with the first stabilization of the administrative business logic subsystem. In
this phase, the quality team completes the design inspection and the design of test cases. In the
final stage, the development team produces the final version, while the quality team focuses on
code inspection and test execution.

RANGANATHA 61
 SOFTWARE TESTING 17IS63

Absence of test design activities in the last phase results from careful identification of activities
that allowed early planning of critical tasks.

Risk Planning
Risk is an inevitable part of every project, and so risk planning must be a part of every plan.
Risks cannot be eliminated, but they can be assessed, controlled, and monitored.
The risk plan component of the quality plan is concerned primarily with personnel risks,
technology risks, and schedule risk. Personnel risk is any contingency that may make a qualified
staff member unavailable when needed. For example, the reassignment of a key test designer
cannot always be avoided, but the possible consequences can be analyzed in advance and
minimized by careful organization of the work. Technology risks in the quality plan include risks
of technology used specifically by the quality team and risks of quality problems involving other
technology used in the product or project. For example, changes in the target platform or in the
testing environment, due to new releases of the operating system or to the adoption of a new
testing tool suite, may not be schedulable in advance, but may be taken into account in the
organization of the testing environment. Schedule risk arises primarily from optimistic
assumptions in the quality plan. For example, underestimating scaffolding design and
maintenance is a common mistake that cannot always be avoided, but consequences can be
mitigated (e.g., by allowing for a reasonable slack time that can absorb possible delays). Many
risks and the tactics for controlling them are generic to project management (e.g., cross-training
to reduce the impact of losing a key staff member). Here we focus on risks that are specific to
quality planning or for which risk control measures play a special role in the quality plan.
The duration of integration, system, and acceptance test execution depends to a large extent on
the quality of software under test. Software that is sloppily constructed or that undergoes
inadequate analysis and test before commitment to the code base will slow testing progress. Even
if responsibility for diagnosing test failures lies with developers and not with the testing group, a
test execution session that results in many failures and generates many failure reports is
inherently more time consuming than executing a suite of tests with few or no failures. This
schedule vulnerability is yet another reason to emphasize earlier activities, in particular those that
provide early indications of quality problems. Inspection of design and code (with quality team
participation) can help control this risk, and also serves to communicate quality standards and
best practices among the team.
If unit testing is the responsibility of developers, test suites are part of the unit deliverable and
should undergo inspection for correctness, thoroughness, and automation. While functional and
structural coverage criteria are no panacea for measuring test thoroughness, it is reasonable to
require that deviations from basic coverage criteria be justified on a case-by-case basis. A
substantial deviation from the structural coverage observed in similar products may be due to
many causes, including inadequate testing, incomplete specifications, unusual design, or

RANGANATHA 62
 SOFTWARE TESTING 17IS63

implementation decisions. The modules that present unusually low structural coverage should be
inspected to identify the cause.
The cost of analysis and test is multiplied when some requirements demand a very high level of
assurance. For example, if a system that has previously been used in biological research is
modified or redeveloped for clinical use, one should anticipate that all development costs, and
particularly costs of analysis and test, will be an order of magnitude higher. In addition to the
risk of underestimating the cost and schedule impact of stringent quality requirements, the risk of
failing to achieve the required dependability increases. One important tactic for controlling this
risk is isolating critical properties as far as possible in small, simple components. Of course these
aspects of system specification and architectural design are not entirely within control of the
quality team; it is crucial that at least the quality manager, and possibly other members of the
quality team, participate in specification and design activities to assess and communicate the
impact of design alternatives on cost and schedule.
Architectural design is also the primary point of leverage to control cost and risks of testing
systems with complex external interfaces. For example, the hardware platform on which an
embedded system must be tested may be a scarce resource, in demand for debugging as well as
testing. Preparing and executing a test case on that platform may be time-consuming, magnifying
the risk that system and operational testing may go over schedule and delay software delivery.
This risk may be reduced by careful consideration of design-for-testability in architectural
design. A testable design isolates and minimizes platform dependencies, reducing the portion of
testing that requires access to the platform. It will typically provide additional interfaces to
enhance controllability and observability in testing. A considerable investment in test
scaffolding, from self-diagnosis to platform simulators, may also be warranted.
Risks related both to critical requirements and limitations on testability can be partially
addressed in system specifications and programming standards. For example, it is notoriously
difficult to detect race conditions by testing multi-threaded software. However, one may impose
a design and programming discipline that prevents race conditions, such as a simple monitor
discipline with resource ordering. Detecting violations of that discipline, statically and
dynamically, is much simpler than detecting actual data races.
This tactic may be reflected in several places in the project plan, from settling on the
programming discipline in architectural design to checking for proper use of the discipline in
code and design inspections, to implementation or purchase of tools to automate compliance
checking.

RANGANATHA 63
 SOFTWARE TESTING 17IS63

Monitoring the Process

The quality manager monitors progress of quality activities, including results as well as schedule,
to identify deviations from the quality plan as early as possible and take corrective action.
Effective monitoring, naturally, depends on a plan that is realistic, well organized, and
sufficiently detailed with clear, unambiguous milestones and criteria. We say a process is visible
to the extent that it can be effectively monitored.
Successful completion of a planned activity must be distinguished from mere termination, as
otherwise it is too tempting to meet an impending deadline by omitting some planned work.
Skipping planned verification activities or addressing them superficially can seem to accelerate a
late project, but the boost is only apparent; the real effect is to postpone detection of more faults
to later stages in development, where their detection and removal will be far more threatening to
project success.
For example, suppose a developer is expected to deliver unit test cases as part of a work unit. If
project deadlines are slipping, the developer is tempted to scrimp on designing unit tests and
writing supporting code, perhaps dashing off a few superficial test cases so that the unit can be
committed to the code base. The rushed development and inadequate unit testing are nearly
guaranteed to leave bugs that surface later, perhaps in integration or system testing, where they
will have a far greater impact on project schedule. Worst of all, they might be first detected in
operational use, reducing the real and perceived quality of the delivered product. In monitoring
progress, therefore, it is essential to include appropriate metrics of the thoroughness or
completeness of the activity.
Monitoring produces a surfeit of detail about individual activities. Managers need to make
decisions based on an overall understanding of project status, so raw monitoring information
must be aggregated in ways that provide an overall picture.
Risk Management in the Quality Plan: Risks Generic to Process Management
The quality plan must identify potential risks and define appropriate control tactics. Some risks
and control tactics are generic to process management, while others are specific to the quality
process. Here we provide a brief overview of some risks generic to process management. Risks
specific to the quality process are summarized in the sidebar on page 391
Personnel Risks Example Control Tactics
A staff member is lost (becomes ill, Cross train to avoid overdependence on individuals;
changes employer, etc.) or is encourage and schedule continuous education; provide
underqualified for task (the project open communication with opportunities for staff self-
plan assumed a level of skill or assessment and identification of skills gaps early in the
familiarity that the assigned member project; provide competitive compensation and promotion
did not have). policies and a rewarding work environment to retain staff;
include training time in the project schedule.
Technology Risks Example Control Tactics

RANGANATHA 64
 SOFTWARE TESTING 17IS63

Many faults are introduced Anticipate and schedule extra time for testing unfamiliar
interfacing to an unfamiliar interfaces; invest training time for COTS components and
commercial off-the-shelf (COTS) for training with new tools; monitor, document, and
component. publicize common errors and correct idioms; introduce
new tools in lower-risk pilot projects or prototyping
exercises.
Test and analysis automation tools Introduce new tools in lower-risk pilot projects or
do not meet expectations. prototyping exercises; anticipate and schedule time for
training with new tools.
COTS components do not meet Include COTS component qualification testing early in
quality expectations. project plan; introduce new COTS components in lower-
risk pilot projects or prototyping exercises.
Schedule Risks Example Control Tactics
Inadequate unit testing leads to Track and reward quality unit testing as evidenced by low-
unanticipated expense and delays in fault densities in integration.
integration testing.
Difficulty of scheduling meetings Set aside times in a weekly schedule in which inspections
makes inspection a bottleneck in take precedence over other meetings and other work; try
development. distributed and asynchronous inspection techniques, with
a lower frequency of face-to-face inspection meetings.
Open table as spreadsheet

Risk Management in the Quality Plan: Risks Specific to Quality Management

Here we provide a brief overview of some risks specific to the quality process. Risks generic to
process management are summarized in the sidebar at page 390.
Development Risks Example Control Tactics
Poor quality software delivered to Provide early warning and feedback; schedule
testing group or inadequate unit test and inspection of design, code and test suites; connect
analysis before committing to the code development and inspection to the reward system;
base. increase training through inspection; require coverage
or other criteria at unit test level.
Executions Risks Example Control Tactics
Execution costs higher than planned; Minimize parts that require full system to be executed;
scarce resources available for testing inspect architecture to assess and improve testability;
(testing requires expensive or complex increase intermediate feedback; invest in scaffolding.
machines or systems not easily
available.)

RANGANATHA 65
 SOFTWARE TESTING 17IS63

Requirements Risks Example Control Tactics

High assurance critical requirements. Compare planned testing effort with former projects
with similar criticality level to avoid underestimating
testing effort; balance test and analysis; isolate critical
parts, concerns and properties.
Open table as spreadsheet

One key aggregate measure is the number of faults that have been revealed and removed, which
can be compared to data obtained from similar past projects. Fault detection and removal can be
tracked against time and will typically follow a characteristic distribution similar to that shown in
Figure 20.3. The number of faults detected per time unit tends to grow across several system
builds, then to decrease at a much lower rate (usually half the growth rate) until it stabilizes

Figure 8.3: A typical distribution of faults for system builds through time.
An unexpected pattern in fault detection may be a symptom of problems. If detected faults stop
growing earlier than expected, one might hope it indicates exceptionally high quality, but it
would be wise to consider the alternative hypothesis that fault detection efforts are ineffective. A
growth rate that remains high through more than half the planned system builds is a warning that
quality goals may be met late or not at all, and may indicate weaknesses in fault removal or lack
of discipline in development (e.g., a rush to add features before delivery, with a consequent
deemphasis on quality control).
A second indicator of problems in the quality process is faults that remain open longer than
expected. Quality problems are confirmed when the number of open faults does not stabilize at a
level acceptable to stakeholders.
The accuracy with which we can predict fault data and diagnose deviations from expectation
depends on the stability of the software development and quality processes, and on availability of
data from similar projects. Differences between organizations and across application domains are
wide, so by far the most valuable data is from similar projects in one's own organization.
The faultiness data in Figure 8.3 are aggregated by severity levels. This helps in better
understanding the process. Growth in the number of moderate faults late in the development

RANGANATHA 66
 SOFTWARE TESTING 17IS63

process may be a symptom of good use of limited resources concentrated in removing critical
and severe faults, not spent solving moderate problems.
Accurate classification schemata can improve monitoring and may be used in very large projects,
where the amount of detailed information cannot be summarized in overall data. The orthogonal
defect classification (ODC) approach has two main steps: (1) fault classification and (2) fault
analysis.
ODC fault classification is done in two phases: when faults are detected and when they are fixed.
At detection time, we record the activity executed when the fault is revealed, the trigger that
exposed the fault, and the perceived or actual impact of the fault on the customer. A possible
taxonomy for activities and triggers is illustrated in the sidebar at page 395. Notice that triggers
depend on the activity. The sidebar at page 396 illustrates a possible taxonomy of customer
impacts.
At fix time, we record target, type, source, and age of the software. The target indicates the
entity that has been fixed to remove the fault, and can be requirements, design, code,
build/package,or documentation/development. The type indicates the type of the fault.
Taxonomies depend on the target. The sidebar at page 396 illustrates a taxonomy of types of
faults removed from design or code. Fault types may be augmented with an indication of the
nature of the fault, which can be: missing, that is, the fault is to due to an omission, as in a
missing statement; incorrect, as in the use of a wrong parameter; or extraneous, that is, due to
something not relevant or pertinent to the document or code, as in a section of the design
document that is not pertinent to the current product and should be removed. The source of the
fault indicates the origin of the faulty modules: in-house, library, ported from other platforms,or
outsourced code.
The age indicates the age of the faulty element - whether the fault was found in new, old (base),
rewritten,or re-fixed code.
The detailed information on faults allows for many analyses that can provide information on the
development and the quality process. As in the case of analysis of simple faultiness data, the
interpretation depends on the process and the product, and should be based on past experience.
The taxonomy of faults, as well as the analysis of faultiness data, should be refined while
applying the method.
When we first apply the ODC method, we can perform some preliminary analysis using only part
of the collected information:
Distribution of fault types versus activities Different quality activities target different classes
of faults. For example, algorithmic (that is, local) faults are targeted primarily by unit testing,
and we expect a high proportion of faults detected by unit testing to be in this class. If the
proportion of algorithmic faults found during unit testing is unusually small, or a larger than
normal proportion of algorithmic faults are found during integration testing, then one may
reasonably suspect that unit tests have not been well designed. If the mix of faults found during
integration testing contains an unusually high proportion of algorithmic faults, it is also possible
that integration testing has not focused strongly enough on interface faults.

RANGANATHA 67
 SOFTWARE TESTING 17IS63

Distribution of triggers over time during field test Faults corresponding to simple usage
should arise early during field test, while faults corresponding to complex usage should arise
late. In both cases, the rate of disclosure of new faults should asymptotically decrease.
Unexpected distributions of triggers over time may indicate poor system or acceptance test. If
triggers that correspond to simple usage reveal many faults late in acceptance testing, we may
have chosen a sample that is not representative of the user population. If faults continue growing
during acceptance test, system testing may have failed, and we may decide to resume it before
continuing with acceptance testing.
Age distribution over target code Most faults should be located in new and rewritten code,
while few faults should be found in base or re-fixed code, since base and re-fixed code has
already been tested and corrected. Moreover, the proportion of faults in new and rewritten code
with respect to base and re-fixed code should gradually increase. Different patterns may indicate
holes in the fault tracking and removal process or may be a symptom of inadequate test and
analysis that failed in revealing faults early (in previous tests of base or re-fixed code). For
example, an increase of faults located in base code after porting to a new platform may indicate
inadequate tests for portability.
Distribution of fault classes over time The proportion of missing code faults should gradually
decrease, while the percentage of extraneous faults may slowly increase, because missing
functionality should be revealed with use and repaired, while extraneous code or documentation
may be produced by updates. An increasing number of missing faults may be a symptom of
instability of the product, while a sudden sharp increase in extraneous faults may indicate
maintenance problems.

Improving the Process

Many classes of faults that occur frequently are rooted in process and development flaws. For
example, a shallow architectural design that does not take into account resource allocation can
lead to resource allocation faults. Lack of experience with the development environment, which
leads to misunderstandings between analysts and programmers on rare and exceptional cases,
can result in faults in exception handling. A performance assessment system that rewards faster
coding without regard to quality is likely to promote low quality code.
The occurrence of many such faults can be reduced by modifying the process and environment.
For example, resource allocation faults resulting from shallow architectural design can be
reduced by introducing specific inspection tasks. Faults attributable to inexperience with the
development environment can be reduced with focused training sessions. Persistently poor
programming practices may require modification of the reward system.
Often, focused changes in the process can lead to product improvement and significant cost
reduction. Unfortunately, identifying the weak aspects of a process can be extremely difficult,

RANGANATHA 68
 SOFTWARE TESTING 17IS63

and often the results of process analysis surprise even expert managers. The analysis of the fault
history can help software engineers build a feedback mechanism to track relevant faults to their
root causes, thus providing vital information for improving the process. In some cases,
information can be fed back directly into the current product development, but more often it
helps software engineers improve the development of future products. For example, if analysis of
faults reveals frequent occurrence of severe memory management faults in C programs, we
might revise inspection checklists and introduce dynamic analysis tools, but it may be too late to
change early design decisions or select a different programming language in the project
underway. More fundamental changes may be made in future projects.
Root cause analysis (RCA) is a technique for identifying and eliminating process faults. RCA
was first developed in the nuclear power industry and later extended to software analysis. It
consists of four main steps to select significant classes of faults and track them back to their
original causes: What, When, Why, and How.
What are the faults? The goal of this first step is to identify a class of important faults. Faults
are categorized by severity and kind. The severity of faults characterizes the impact of the fault
on the product. Although different methodologies use slightly different scales and terms, all of
them identify a few standard levels, described in Table 8.1
Table 8.1: Standard severity levels for root cause analysis (RCA).
Open table as spreadsheet
Level Description Example
Critical The product is unusable. The fault causes the program to crash.
Severe Some product features cannot be The fault inhibits importing files saved with a
used, and there is no workaround. previous version of the program, and there is no
way to convert files saved in the old format to the
new one.
Moderate Some product features require The fault inhibits exporting in Postscript format.
workarounds to use, and reduce Postscript can be produced using the printing
efficiency, reliability, or facility, but the process is not obvious or
convenience and usability. documented (loss of usability) and requires extra
steps (loss of efficiency).
Cosmetic Minor inconvenience. The fault limits the choice of colors for
customizing the graphical interface, violating the
specification but causing only minor
inconvenience.
The RCA approach to categorizing faults, in contrast to ODC, does not use a predefined set of
categories. The objective of RCA is not to compare different classes of faults over time, or to
analyze and eliminate all possible faults, but rather to identify the few most important classes of
faults and remove their causes. Successful application of RCA progressively eliminates the

RANGANATHA 69
 SOFTWARE TESTING 17IS63

causes of the currently most important faults, which lose importance over time, so applying a
static predefined classification would be useless. Moreover, the precision with which we identify
faults depends on the specific project and process and varies over time.
ODC Classification of Triggers Listed by Activity
Design Review and Code Inspection
Design Conformance A discrepancy between the reviewed artifact and a prior-stage artifact that
serves as its specification.
Logic/Flow An algorithmic or logic flaw.
Backward Compatibility A difference between the current and earlier versions of an artifact
that could be perceived by the customer as a failure.
Internal Document An internal inconsistency in the artifact (e.g., inconsistency between code
and comments).
Lateral Compatibility An incompatibility between the artifact and some other system or
module with which it should interoperate.
Concurrency A fault in interaction of concurrent processes or threads.
Language Dependency A violation of language-specific rules, standards, or best practices.
Side Effects A potential undesired interaction between the reviewed artifact and some other part
of the system.
Rare Situation An inappropriate response to a situation that is not anticipated in the artifact.
(Error handling as specified in a prior artifact design conformance, not rare situation.)
Structural (White-Box) Test
Simple Path The fault is detected by a test case derived to cover a single program element.
Complex Path The fault is detected by a test case derived to cover a combination of program
elements.
Functional (Black-Box) Test
Coverage The fault is detected by a test case derived for testing a single procedure (e.g., C
function or Java method), without considering combination of values for possible parameters.
Variation The fault is detected by a test case derived to exercise a particular combination of
parameters for a single procedure.
Sequencing The fault is detected by a test case derived for testing a sequence of procedure calls.
Interaction The fault is detected by a test case derived for testing procedure interactions.
System Test
Workload/Stress The fault is detected during workload or stress testing.
Recovery/Exception The fault is detected while testing exceptions and recovery procedures.
Startup/Restart The fault is detected while testing initialization conditions during start up or
after possibly faulty shutdowns.
Hardware Configuration The fault is detected while testing specific hardware configurations.
Software Configuration The fault is detected while testing specific software configurations.
Blocked Test Failure occurred in setting up the test scenario.

RANGANATHA 70
 SOFTWARE TESTING 17IS63

ODC Classification of Customer Impact

Installability Ability of the customer to place the software into actual use. (Usability of the
installed software is not included.)
Integrity/Security Protection of programs and data from either accidental or malicious
destruction or alteration, and from unauthorized disclosure.
Performance The perceived and actual impact of the software on the time required for the
customer and customer end users to complete their tasks.
Maintenance The ability to correct, adapt, or enhance the software system quickly and at
minimal cost.
Serviceability Timely detection and diagnosis of failures, with minimal customer impact.
Migration Ease of upgrading to a new system release with minimal disruption to existing
customer data and operations.
Documentation Degree to which provided documents (in all forms, including electronic)
completely and correctly describe the structure and intended uses of the software.
Usability The degree to which the software and accompanying documents can be understood
and effectively employed by the end user.
Standards The degree to which the software complies with applicable standards.
Reliability The ability of the software to perform its intended function without unplanned
interruption or failure.
Accessibility The degree to which persons with disabilities can obtain the full benefit of the
software system.
Capability The degree to which the software performs its intended functions consistently with
documented system requirements.
Requirements The degree to which the system, in complying with document requirements,
actually meets customer expectations
ODC Classification of Defect Types for Targets Design and Code
Assignment/Initialization A variable was not assigned the correct initial value or was not
assigned any initial value.
Checking Procedure parameters or variables were not properly validated before use.
Algorithm/Method A correctness or efficiency problem that can be fixed by reimplementing a
single procedure or local data structure, without a design change.
Function/Class/Object A change to the documented design is required to conform to product
requirements or interface specifications.
Timing/Synchronization The implementation omits necessary synchronization of shared
resources, or violates the prescribed synchronization protocol.
Interface/Object-Oriented Messages Module interfaces are incompatible; this can include
syntactically compatible interfaces that differ in semantic interpretation of communicated data.
Relationship Potentially problematic interactions among procedures, possibly involving
different assumptions but not involving interface incompatibility.

RANGANATHA 71
 SOFTWARE TESTING 17IS63

A good RCA classification should follow the uneven distribution of faults across categories. If,
for example, the current process and the programming style and environment result in many
interface faults, we may adopt a finer classification for interface faults and a coarse-grain
classification of other kinds of faults. We may alter the classification scheme in future projects as
a result of having identified and removed the causes of many interface faults.
Classification of faults should be sufficiently precise to allow identifying one or two most
significant classes of faults considering severity, frequency, and cost of repair. It is important to
keep in mind that severity and repair cost are not directly related. We may have cosmetic faults
that are very expensive to repair, and critical faults that can be easily repaired. When selecting
the target class of faults, we need to consider all the factors. We might, for example, decide to
focus on a class of moderately severe faults that occur very frequently and are very expensive to
remove, investing fewer resources in preventing a more severe class of faults that occur rarely
and are easily repaired.
When did faults occur, and when were they found? It is typical of mature software processes
to collect fault data sufficient to determine when each fault was detected (e.g., in integration test
or in a design inspection). In addition, for the class of faults identified in the first step, we
attempt to determine when those faults were introduced (e.g., was a particular fault introduced in
coding, or did it result from an error in architectural design?).
Why did faults occur? In this core RCA step, we attempt to trace representative faults back to
causes, with the objective of identifying a "root" cause associated with many faults in the class.
Analysis proceeds iteratively by attempting to explain the error that led to the fault, then the
cause of that error, the cause of that cause, and so on. The rule of thumb "ask why six times"
does not provide a precise stopping rule for the analysis, but suggests that several steps may be
needed to find a cause in common among a large fraction of the fault class under consideration.

The 80/20 or Pareto Rule

Fault classification in root cause analysis is justified by the so-called 80/20 or Pareto rule. The
Pareto rule is named for the Italian economist Vilfredo Pareto, who in the early nineteenth
century proposed a mathematical power law formula to describe the unequal distribution of
wealth in his country, observing that 20% of the people owned 80% of the wealth.
Pareto observed that in many populations, a few (20%) are vital and many (80%) are trivial. In
fault analysis, the Pareto rule postulates that 20% of the code is responsible for 80% of the faults.
Although proportions may vary, the rule captures two important facts:
3. Faults tend to accumulate in a few modules, so identifying potentially faulty modules can
improve the cost effectiveness of fault detection.
4. Some classes of faults predominate, so removing the causes of a predominant class of
faults can have a major impact on the quality of the process and of the resulting product.
The predominance of a few classes of faults justifies focusing on one class at a time.

RANGANATHA 72
 SOFTWARE TESTING 17IS63

Tracing the causes of faults requires experience, judgment, and knowledge of the development
process. We illustrate with a simple example. Imagine that the first RCA step identified memory
leaks as the most significant class of faults, combining a moderate frequency of occurrence with
severe impact and high cost to diagnose and repair. The group carrying out RCA will try to
identify the cause of memory leaks and may conclude that many of them result from forgetting to
release memory in exception handlers. The RCA group may trace this problem in exception
handling to lack of information: Programmers can't easily determine what needs to be cleaned
up in exception handlers. The RCA group will ask why once more and may go back to a design
error: The resource management scheme assumes normal flow of control and thus does not
provide enough information to guide implementation of exception handlers. Finally, the RCA
group may identify the root problem in an early design problem: Exceptional conditions were an
afterthought dealt with late in design.
Each step requires information about the class of faults and about the development process that
can be acquired through inspection of the documentation and interviews with developers and
testers, but the key to success is curious probing through several levels of cause and effect.
How could faults be prevented? The final step of RCA is improving the process by removing
root causes or making early detection likely. The measures taken may have a minor impact on
the development process (e.g., adding consideration of exceptional conditions to a design
inspection checklist), or may involve a substantial modification of the process (e.g., making
explicit consideration of exceptional conditions a part of all requirements analysis and design
steps). As in tracing causes, prescribing preventative or detection measures requires judgment,
keeping in mind that the goal is not perfection but cost-effective improvement.
ODC and RCA are two examples of feedback and improvement, which are an important
dimension of most good software processes. Explicit process improvement steps are, for
example, featured in both SRET (sidebar on page 380) and Cleanroom (sidebar on page 378).

The Quality Team

The quality plan must assign roles and responsibilities to people. As with other aspects of
planning, assignment of responsibility occurs at a strategic level and a tactical level. The tactical
level, represented directly in the project plan, assigns responsibility to individuals in accordance
with the general strategy. It involves balancing level of effort across time and carefully managing
personal interactions. The strategic level of organization is represented not only in the quality
strategy document, but in the structure of the organization itself.
The strategy for assigning responsibility may be partly driven by external requirements. For
example, independent quality teams may be required by certification agencies or by a client
organization. Additional objectives include ensuring sufficient accountability that quality tasks
are not easily overlooked; encouraging objective judgment of quality and preventing it from
being subverted by schedule pressure; fostering shared commitment to quality among all team
members; and developing and communicating shared knowledge and values regarding quality.

RANGANATHA 73
 SOFTWARE TESTING 17IS63

Measures taken to attain some objectives (e.g., autonomy to ensure objective assessment) are in
tension with others (e.g., cooperation to meet overall project objectives). It is therefore not
surprising to find that different organizations structure roles and responsibilities in a wide variety
of different ways. The same individuals can play the roles of developer and tester, or most testing
responsibility can be assigned to members of a distinct group, and some may even be assigned to
a distinct organization on a contractual basis. Oversight and accountability for approving the
work product of a task are sometimes distinguished from responsibility for actually performing a
task, so the team organization is somewhat intertwined with the task breakdown.
Each of the possible organizations of quality roles makes some objectives easier to achieve and
some more challenging. Conflict of one kind or another is inevitable, and therefore in organizing
the team it is important to recognize the conflicts and take measures to control adverse
consequences. If an individual plays two roles in potential conflict (e.g., a developer responsible
for delivering a unit on schedule is also responsible for integration testing that could reveal faults
that delay delivery), there must be countermeasures to control the risks inherent in that conflict.
If roles are assigned to different individuals, then the corresponding risk is conflict between the
individuals (e.g., if a developer and a tester do not adequately share motivation to deliver a
quality product on schedule).
An independent and autonomous testing team lies at one end of the spectrum of possible team
organizations. One can make that team organizationally independent so that, for example, a
project manager with schedule pressures can neither bypass quality activities or standards, nor
reallocate people from testing to development, nor postpone quality activities until too late in the
project. Separating quality roles from development roles minimizes the risk of conflict between
roles played by an individual, and thus makes most sense for roles in which independence is
paramount, such as final system and acceptance testing.
An independent team devoted to quality activities also has an advantage in building specific
expertise, such as test design. The primary risk arising from separation is in conflict between
goals of the independent quality team and the developers.
When quality tasks are distributed among groups or organizations, the plan should include
specific checks to ensure successful completion of quality activities. For example, when module
testing is performed by developers and integration and system testing is performed by an
independent quality team, the quality team should check the completeness of module tests
performed by developers, for example, by requiring satisfaction of coverage criteria or inspecting
module test suites. If testing is performed by an independent organization under contract, the
contract should carefully describe the testing process and its results and documentation, and the
client organization should verify satisfactory completion of the contracted tasks.
Existence of a testing team must not be perceived as relieving developers from responsibility for
quality, nor is it healthy for the testing team to be completely oblivious to other pressures,
including schedule pressure. The testing team and development team, if separate, must at least
share the goal of shipping a high-quality product on schedule.

RANGANATHA 74
 SOFTWARE TESTING 17IS63

Independent quality teams require a mature development process to minimize communication

and coordination overhead. Test designers must be able to work on sufficiently precise
specifications and must be able to execute tests in a controllable test environment. Versions and
configurations must be well defined, and failures and faults must be suitably tracked and
monitored across versions.
It may be logistically impossible to maintain an independent quality group, especially in small
projects and organizations, where flexibility in assignments is essential for resource
management. Aside from the logistical issues, division of responsibility creates additional work
in communication and coordination. Finally, quality activities often demand deep knowledge of
the project, particularly at detailed levels (e.g., unit and early integration test). An outsider will
have less insight into how and what to test, and may be unable to effectively carry out the crucial
earlier activities, such as establishing acceptance criteria and reviewing architectural design for
testability. For all these reasons, even organizations that rely on an independent verification and
validation (IV&V) group for final product qualification allocate other responsibilities to
developers and to quality professionals working more closely with the development team.
At the polar opposite from a completely independent quality team is full integration of quality
activities with development, as in some "agile" processes including XP.
Communication and coordination overhead is minimized this way, and developers take full
responsibility for the quality of their work product. Moreover, technology and application
expertise for quality tasks will match the expertise available for development tasks, although the
developer may have less specific expertise in skills such as test design.
The more development and quality roles are combined and intermixed, the more important it is
to build into the plan checks and balances to be certain that quality activities and objective
assessment are not easily tossed aside as deadlines loom. For example, XP practices like "test
first" together with pair programming (sidebar on page 381) guard against some of the inherent
risks of mixing roles. Separate roles do not necessarily imply segregation of quality activities to
distinct individuals.
It is possible to assign both development and quality responsibility to developers, but assign two
individuals distinct responsibilities for each development work product. Peer review is an
example of mixing roles while maintaining independence on an item-by-item basis. It is also
possible for developers and testers to participate together in some activities.
Many variations and hybrid models of organization can be designed. Some organizations have
obtained a good balance of benefits by rotating responsibilities. For example, a developer may
move into a role primarily responsible for quality in one project and move back into a regular
development role in the next. In organizations large enough to have a distinct quality or testing
group, an appropriate balance between independence and integration typically varies across
levels of project organization. At some levels, an appropriate balance can be struck by giving
responsibility for an activity (e.g., unit testing) to developers who know the code best, but with a
separate oversight responsibility shared by members of the quality team. For example, unit tests
may be designed and implemented by developers, but reviewed by a member of the quality team

RANGANATHA 75
 SOFTWARE TESTING 17IS63

for effective automation (particularly, suitability for automated regression test execution as the
product evolves) as well as thoroughness. The balance tips further toward independence at higher
levels of granularity, such as in system and acceptance testing, where at least some tests should
be designed independently by members of the quality team.
Outsourcing test and analysis activities is sometimes motivated by the perception that testing is
less technically demanding than development and can be carried out by lower-paid and lower-
skilled individuals. This confuses test execution, which should in fact be straightforward, with
analysis and test design, which are as demanding as design and programming tasks in
development. Of course, less skilled individuals can design and carry out tests, just as less skilled
individuals can design and write programs, but in both cases the results are unlikely to be
satisfactory.
Outsourcing can be a reasonable approach when its objectives are not merely minimizing cost,
but maximizing independence. For example, an independent judgment of quality may be
particularly valuable for final system and acceptance testing, and may be essential for measuring
a product against an independent quality standard (e.g., qualifying a product for medical or
avionic use). Just as an organization with mixed roles requires special attention to avoid the
conflicts between roles played by an individual, radical separation of responsibility requires
special attention to control conflicts between the quality assessment team and the development
team.
The plan must clearly define milestones and delivery for outsourced activities, as well as checks
on the quality of delivery in both directions: Test organizations usually perform quick checks to
verify the consistency of the software to be tested with respect to some minimal "testability"
requirements; clients usually check the completeness and consistency of test results. For
example, test organizations may ask for the results of inspections on the delivered artifact before
they start testing, and may include some quick tests to verify the installability and testability of
the artifact. Clients may check that tests satisfy specified functional and structural coverage
criteria, and may inspect the test documentation to check its quality. Although the contract
should detail the relation between the development and the testing groups, ultimately,
outsourcing relies on mutual trust between organizations

Documenting Analysis and Test

Mature software processes include documentation standards for all the activities of the software
process, including test and analysis activities. Documentation can be inspected to verify progress
against schedule and quality goals and to identify problems, supporting process visibility,
monitoring, and replicability
Overview
Documentation is an important element of the software development process, including the
quality process. Complete and well-structured documents increase the reusability of test suites

RANGANATHA 76
 SOFTWARE TESTING 17IS63

within and across projects. Documents are essential for maintaining a body of knowledge that
can be reused across projects. Consistent documents provide a basis for monitoring and assessing
the process, both internally and for external authorities where certification is desired. Finally,
documentation includes summarizing and presenting data that forms the basis for process
improvement. Test and analysis documentation includes summary documents designed primarily
for human comprehension and details accessible to the human reviewer but designed primarily
for automated analysis.
Documents are divided into three main categories: planning, specification, and reporting.
Planning documents describe the organization of the quality process and include strategies and
plans for the division or the company, and plans for individual projects. Specification documents
describe test suites and test cases. A complete set of analysis and test specification documents
include test design specifications, test case specification, checklists, and analysis procedure
specifications. Reporting documents include details and summary of analysis and test results.
Organizing Documents
In a small project with a sufficiently small set of documents, the arrangement of other project
artifacts (e.g., requirements and design documents) together with standard content (e.g., mapping
of subsystem test suites to the build schedule) provides sufficient organization to navigate
through the collection of test and analysis documentation. In larger projects, it is common
practice to produce and regularly update a global guide for navigating among individual
documents.
Mature processes require all documents to contain metadata that facilitate their management.
Documents must include some basic information about its context in order to make the document
self-contained, approval indicating the persons responsible for the document and document
history.
Naming conventions help in quickly identifying documents. A typical standard for document
names would include keywords indicating the general scope of the document, its nature, the
specific document, and its version, as in Figure 8.4

RANGANATHA 77
 SOFTWARE TESTING 17IS63

Chipmunk Document Template Document Title

Approvals
issued by name signature date
approved by name signature date
distribution status (internal use only, restricted, …)
distribution list (people to whom the document must be sent)
Open table as spreadsheet
History
version description

Open table as spreadsheet

Table of Contents
 List of sections.
Summary
 Summarize the contents of the document. The summary should clearly explain the
relevance of the document to its possible uses.

RANGANATHA 78
 SOFTWARE TESTING 17IS63

Goals of the document

 Describe the purpose of this document: Who should read it, and why?
Required documents and references
 Provide a reference to other documents and artifacts needed for understanding and
exploiting this document. Provide a rationale for the provided references.
Glossary
 Provide a glossary of terms required to understand this document.
Section 1
 …
Section N
 …

Test Strategy Document

Analysis and test strategies describe quality guidelines for sets of projects, usually for an entire
company or organization. Strategies, and therefore strategy documents, vary widely among
organizations, but we can identify a few key elements that should be included in almost any
well-designed strategy document. These are illustrated in the document excerpt on page 459.
Strategy documents indicate common quality requirements across products. Requirements may
depend on business conditions. For example, a company that produces safety-critical software
may need to satisfy minimum dependability requirements defined by a certification authority,
while a department that designs software embedded in hardware products may need to ensure
portability across product lines. Some requirements on dependability and usability may be
necessary to maintain brand image and market position. For example, a company might decide to
require conformance to W3C-WAI accessibility standards uniformly across the product line.
The strategy document sets out requirements on other quality documents, typically including an
analysis and test plan, test design specifications, test case specifications, test logs, and test
summary reports. Basic document requirements, such as naming and versioning, follow
standards for other project documentation, but quality documents may have additional,
specialized requirements. For example, testing logs for avionics software may be required to
contain references to the version of the simulator used for executing the test before installing the
software on board the aircraft.

Analysis and Test Plan

While the format of an analysis and test strategy vary from company to company, the structure of
an analysis and test plan is more standardized. A typical structure of a test and analysis plan
includes information about items to be verified, features to be tested, the testing approach, pass
and fail criteria, test deliverables, tasks, responsibilities and resources, and environment
constraints. Basic elements are described in the sidebar on page 461.

RANGANATHA 79
 SOFTWARE TESTING 17IS63

The overall quality plan usually comprises several individual plans of limited scope. Each test
and analysis plan should indicate the items to be verified through analysis or testing. They may
include specifications or documents to be inspected, code to be analyzed or tested, and interface
specifications to undergo consistency analysis. They may refer to the whole system or part of it -
like a subsystem or a set of units. Where the project plan includes planned development
increments, the analysis and test plan indicates the applicable versions of items to be verified.
For each item, the plan should indicate any special hardware or external software required for
testing. For example, the plan might indicate that one suite of subsystem tests for a security
package can be executed with a software simulation of a smart card reader, while another suite
requires access to the physical device. Finally, for each item, the plan should reference related
documentation, such as requirements and design specifications, and user, installation, and
operations guides.
An Excerpt of the Chipmunk Analysis and Test Strategy
Document CP05-14.03: Analysis and Test Strategy
…
Applicable Standards and Procedures
Artifact Applicable Standards and Guidelines
Web application Accessibility: W3C-WAI …
Reusable component (internally developed) Inspection procedure: [WB12-03.12]
External component Qualification procedure: [WB12-22.04]
Open table as spreadsheet
…
Documentation Standards
Project documents must be archived according to the standard Chipmunk archive procedure
[WB02-01.02]. Standard required documents include
Document Content & Organization Standard
Quality plan [WB06-01.03]
Test design specifications [WB07-01.01] (per test suite)
Test case specifications [WB08-01.07] (per test suite)
Test logs [WB10-02.13]
Test summary reports [WB11-01.11]
Inspection reports [WB12-09.01]
Open table as spreadsheet
Analysis and Test Activities
Tools

RANGANATHA 80
 SOFTWARE TESTING 17IS63

The following tools are approved and should be used in all development projects. Exceptions
require configuration committee approval and must be documented in the project plan.
Fault logging Chipmunk BgT [WB10-23.01]
…
Open table as spreadsheet
Staff and Roles
A development work unit consists of unit source code, including unit test cases, stubs, and
harnesses, and unit test documentation. A unit may be committed to the project baseline when
the source code, test cases, and test results have passed peer review.
References
[WB02-01.02] Archive Procedure [WB06-01.03] Quality Plan Guidelines
[WB07-01.01] Test Design Specifications Guidelines [WB08-01.07] Test Case Specifications
Guidelines
[WB11-01.11] Summary Reports Template [WB10-02.13] Test Log Template
[WB11-09.01] Inspection Report Template [WB12-03.12] Standard Inspection
Procedures
[WB12-22.04] Quality Procedures for Software [WB12-23.01] BgT Installation Manual
Developed by Third Parties and User Guide
Open table as spreadsheet
…

A test and analysis plan may not address all aspects of software quality and testing activities. It
should indicate the features to be verified and those that are excluded from consideration (usually
because responsibility for them is placed elsewhere). For example, if the item to be verified
includes a graphical user interface, the test and analysis plan might state that it deals only with
functional properties and not with usability, which is to be verified separately by a usability and
human interface design team.
Explicit indication of features not to be tested, as well as those included in an analysis and test
plan, is important for assessing completeness of the overall set of analysis and test activities.
Assumption that a feature not considered in the current plan is covered at another point is a
major cause of missing verification in large projects.
The quality plan must clearly indicate criteria for deciding the success or failure of each planned
activity, as well as the conditions for suspending and resuming analysis and test.
Plans define items and documents that must be produced during verification. Test deliverables
are particularly important for regression testing, certification, and process improvement.
The core of an analysis and test plan is a detailed schedule of tasks. The schedule is usually
illustrated with GANTT and PERT diagrams showing the relation among tasks as well as their

RANGANATHA 81
 SOFTWARE TESTING 17IS63

relation to other project milestones. The schedule includes the allocation of limited resources
(particularly staff) and indicates responsibility for reresources and responsibilities sults.
A quality plan document should also include an explicit risk plan with contingencies. As far as
possible, contingencies should include unambiguous triggers (e.g., a date on which a contingency
is activated if a particular task has not be completed) as well as recovery procedures.
Finally, the test and analysis plan should indicate scaffolding, oracles, and any other software or
hardware support required for test and analysis activities

Test Design Specification Documents

Design documentation for test suites and test cases serve essentially the same purpose as other
software design documentation, guiding further development and preparing for maintenance.
Test suite design must include all the information needed for initial selection of test cases and
maintenance of the test suite over time, including rationale and anticipated evolution.
Specification of individual test cases includes purpose, usage, and anticipated changes.
A Standard Organization of an Analysis and Test Plan
Analysis and test items:
 The items to be tested or analyzed. The description of each item indicates version and
installation procedures that may be required.

Features to be tested:
 The features considered in the plan.
Features not to be tested:
 Features not considered in the current plan.
Approach:
 The overall analysis and test approach, sufficiently detailed to permit identification of the
major test and analysis tasks and estimation of time and resources.
Pass/Fail criteria:
 Rules that determine the status of an artifact subjected to analysis and test.
Suspension and resumption criteria:
 Conditions to trigger suspension of test and analysis activities (e.g., an excessive failure
rate) and conditions for restarting or resuming an activity.
Risks and contingencies:
 Risks foreseen when designing the plan and a contingency plan for each of the identified
risks.
Deliverables:
 A list all A&T artifacts and documents that must be produced.
Task and schedule:

RANGANATHA 82
 SOFTWARE TESTING 17IS63

 A complete description of analysis and test tasks, relations among them, and relations
between A&T and development tasks, with resource allocation and constraints. A task
schedule usually includes GANTT and PERT diagrams.
Staff and responsibilities:
 Staff required for performing analysis and test activities, the required skills, and the
allocation of responsibilities among groups and individuals. Allocation of resources to
tasks is described in the schedule.
Environmental needs:
 Hardware and software required to perform analysis or testing activities.

Test design specification documents describe complete test suites (i.e., sets of test cases that
focus on particular aspects, elements, or phases of a software project). They may be divided into
unit, integration, system, and acceptance test suites, if we organize them by the granularity of the
tests, or functional, structural, and performance test suites, if the primary organization is based
on test objectives. A large project may include many test design specifications for test suites of
different kinds and granularity, and for different versions or configurations of the system and its
components. Each specification should be uniquely identified and related to corresponding
project documents, as illustrated in the sidebar on page 463.
Test design specifications identify the features they are intended to verify and the approach used
to select test cases. Features to be tested should be cross-referenced to relevant parts of a
software specification or design document.
A test design specification also includes description of the testing procedure and pass/fail
criteria. The procedure indicates steps required to set up the testing environment and perform the
tests, and includes references to scaffolding and oracles. Pass/fail criteria distinguish success
from failure of a test suite as a whole. In the simplest case a test suite execution may be
determined to have failed if any individual test case execution fails, but in system and acceptance
testing it is common to set a tolerance level that may depend on the number and severity of
failures.
A test design specification logically includes a list of test cases. Test case specifications may be
physically included in the test design specification document, or the logical inclusion may be
implemented by some form of automated navigation. For example, a navigational index can be
constructed from references in test case specifications.
Individual test case specifications elaborate the test design for each individual test case, defining
test inputs, required environmental conditions and procedures for test execution, as well as
expected outputs or behavior. The environmental conditions may include hardware and software
as well as any other requirements. For example, while most tests should be executed
automatically without human interaction, intervention of personnel with certain special skills
(e.g., a device operator) may be an environmental requirement for some.
A test case specification indicates the item to be tested, such as a particular module or product
feature. It includes a reference to the corresponding test design document and describes any

RANGANATHA 83
 SOFTWARE TESTING 17IS63

dependence on execution of other test cases. Like any standard document, a test case
specification is labeled with a unique identifier.

Test and Analysis Reports

Reports of test and analysis results serve both developers and test designers. They identify open
faults for developers and aid in scheduling fixes and revisions. They help test designers assess
and refine their approach, for example, noting when some class of faults is escaping early test
and analysis and showing up only in subsystem and system testing.

Functional Test Design Specification of check configuration

Test Suite Identifier
WB07-15.01
Features to Be Tested
Functional test for check configuration, module specification WB02-15.32.[a]
Approach
Combinatorial functional test of feature parameters, enumerated by category- partition method
over parameter table on page 3 of this document.[b]
Procedure
Designed for conditional inclusion in nightly test run. Build target T02 15 32 11 includes JUnit
harness and oracles, with test reports directed to standard test log. Test environment includes
table MDB 15 32 03 for loading initial test database state.
Test cases[c]
WB07-15.01.C01 malformed model number
WB07-15.01.C02 model number not in DB
… …
WB07- valid model number with all legal required slots and some legal optional
15.01.C09[d] slots
… …
WB07-15.01.C19 empty model DB
WB07-15.01.C23 model DB with a single element
WB07-15.01.C24 empty component DB
WB07-15.01.C29 component DB with a single element
[d]
See sample test case specification, page 464.
Open table as spreadsheet
Pass/Fail Criterion

RANGANATHA 84
 SOFTWARE TESTING 17IS63

Successful completion requires correct execution of all test cases with no violations in test log.
Test Case Specification for check configuration
Test Case Identifier
WB07-15.01.C09[a]
Test items
Module check configuration of the Chipmunk Web presence system, business logic subsystem.
Input specification
Test Case Specification:
Model No. valid
No. of required slots for selected model (#SMRS) many
No. of optional slots for selected model (#SMOS) many
Correspondence of selection with model slots complete
No. of required components with selection ≠ empty = No. of required slots
No. of optional components with select ≠ empty < No. of optional slots
Required component selection all valid
Optional component selection all valid
No. of models in DB many
No. of components in DB many
Open table as spreadsheet
Test case:
Model number Chipmunk C20
#SMRS 5
Screen 13"
Processor Chipmunk II plus
Hard disk 30 GB
RAM 512 MB
OS RodentOS 3.2 Personal Edition
#SMOS 4
External storage device DVD player
Open table as spreadsheet

Output Specification
 return value valid

RANGANATHA 85
 SOFTWARE TESTING 17IS63

Environment Needs
Execute with ChipmunkDBM v3.4 database initialized from table MDB 15 32 03.
Special Procedural Requirements
 none
Intercase Dependencies
 none

A prioritized list of open faults is the core of an effective fault handling and repair procedure.
Failure reports must be consolidated and categorized so that repair effort can be managed
systematically, rather than jumping erratically from problem to problem and wasting time on
duplicate reports. They must be prioritized so that effort is not squandered on faults of relatively
minor importance while critical faults are neglected or even forgotten.
Other reports should be crafted to suit the particular needs of an organization and project,
including process improvement. Summary reports serve primarily to track progress and status.
They may be as simple as confirmation that the nightly build-and-test cycle ran successfully with
no new failures, or they may provide somewhat more information to guide attention to potential
trouble spots. Detailed test logs are designed for selective reading, and include summary tables
that typically include the test suites executed, the number of failures, and a breakdown of failures
into those repeated from prior test execution, new failures, and test cases that previously failed
but now execute correctly.
In some domains, such as medicine or avionics, the content and form of test logs may be
prescribed by a certifying authority. For example, some certifications require test execution logs
signed by both the person who performed the test and a quality inspector, who ascertains
conformance of the test execution with test specifications.

RANGANATHA 86