SOFTWARE QUALITY ASSURANCE

3. TEST GENERATION
3.1 SOFTWARE TESTING
Software testing is a process of executing a program or application with the intent of finding the
software bugs.
It can also be stated as the process of validating and verifying that a software program or
application or product:
Meets the business and technical requirements that guided its design and
development
Works as expected
Can be implemented with the same characteristic.

The definition of Software testing is broken into the following parts:

1) Process: Testing is a process rather than a single activity.
2) All Life Cycle Activities: Testing is a process thats take place throughout the Software
Development Life Cycle (SDLC).

The process of designing tests early in the life cycle can help to prevent defects from
being introduced in the code. Sometimes its referred as verifying the test basis via the
test design.
The test basis includes documents such as the requirements and design specifications.

3) Static Testing: It can test and find defects without executing code. Static Testing is done
during verification process. This testing includes reviewing of the documents (including source
code) and static analysis. This is useful and cost effective way of testing. For example:
reviewing, walkthrough, inspection, etc.
4) Dynamic Testing: In dynamic testing the software code is executed to demonstrate the result
of running tests. Its done during validation process. For example: unit testing, integration
testing, system testing, etc.
5) Planning: We need to plan as what we want to do. We control the test activities, we report
on testing progress and the status of the software under test.
6) Preparation: We need to choose what testing we will do, by selecting test conditions and
designing test cases.
1

unit III

SOFTWARE QUALITY ASSURANCE

7) Evaluation: During evaluation we must check the results and evaluate the software under
test and the completion criteria, which helps us to decide whether we have finished testing and
whether the software product has passed the tests.
8) Software products and related work products: Along with the testing of code the testing
of requirement and design specifications and also the related documents like operation, user and
training material is equally important.
Software Testing has different goals and objectives. The major objectives of Software testing are
as follows:

Finding defects which may get created by the programmer while developing the software.
Gaining confidence in and providing information about the level of quality.
To prevent defects.
To make sure that the end result meets the business and user requirements.
To ensure that it satisfies the BRS that is Business Requirement Specification and SRS
that is System Requirement Specifications.
To gain the confidence of the customers by providing them a quality product.

Software testing is very important because of the following reasons:

1. Software testing is really required to point out the defects and errors that were made
during the development phases.
2. Its essential since it makes sure of the Customers reliability and their satisfaction in the
application.
3. It is very important to ensure the Quality of the product. Quality product delivered to the
customers helps in gaining their confidence.
4. Testing is necessary in order to provide the facilities to the customers like the delivery of
high quality product or software application which requires lower maintenance cost and
hence results into more accurate, consistent and reliable results.
5. Testing is required for an effective performance of software application or product.
6. Its important to ensure that the application should not result into any failures because it
can be very expensive in the future or in the later stages of the development.
7. Its required to stay in the business.

3.2. VALIDATION

Determining if the system complies with the requirements and performs functions for
which it is intended and meets the organizations goals and user needs.
Validation is done at the end of the development process and takes place after
verifications are completed.
It answers the question like: Am I building the right product?
unit III

SOFTWARE QUALITY ASSURANCE

Am I accessing the right data (in terms of the data required to satisfy the requirement).
It is a High level activity.
Performed after a work product is produced against established criteria ensuring that the
product integrates correctly into the environment.
Determination of correctness of the final software product by a development project with
respect to the user needs and requirements.

3.2.1. Software verification and validation

According to the Capability Maturity Model we can also define validation as The process of
evaluating software during or at the end of the development process to determine whether it
satisfies specified requirements.
A product can pass while verification, as it is done on the paper and no running or functional
application is required. But, when same points which were verified on the paper is actually
developed then the running application or product can fail while validation. This may happen
because when a product or application is build as per the specification but these specifications
are not up to the mark hence they fail to address the user requirements.
3.2.2. Advantages of Validation:
1. During verification if some defects are missed then during validation process it can be
caught as failures.
2. If during verification some specification is misunderstood and development had
happened then during validation process while executing that functionality the difference
between the actual result and expected result can be understood.
3. Validation is done during testing like feature testing, integration testing, system testing,
load testing, compatibility testing, stress testing, etc.
4. Validation helps in building the right product as per the customers requirement and helps
in satisfying their needs.
Validation is basically done by the testers during the testing. While validating the product if
some deviation is found in the actual result from the expected result then a bug is reported or an
3

unit III

SOFTWARE QUALITY ASSURANCE

incident is raised. Not all incidents are bugs. But all bugs are incidents. Incidents can also be of
type Question where the functionality is not clear to the tester.
Hence, validation helps in unfolding the exact functionality of the features and helps the testers
to understand the product in much better way. It helps in making the product more user friendly.
Validation testing provides answers to questions such as: Does the software fulfill its
intended use? Is the company building the right product? Can the project be properly
implemented?
Are the documents in line with the development process?
3.2.3. Validation Testing Process:
The aim of software testing is to measure the quality of software in terms of number of
defects found in it, the number of tests run, and the system covered by the tests. When bugs or
defects are found with the help of testing, the bugs are logged and the development team fixes
them. Once the bugs are fixed, testing is carried out again to ensure that they are indeed fixed,
and no new defects have been introduced in the software. With the entire cycle, the quality of the
software increases. The software validation process can be described as follows:
Validation Planning
To plan all the activities that needs to be included while testing.

Define Requirements
To set goals and define the requirements for testing

Select Appropriate Team

To select a skilled and knowledgeable development team (third party included)

Develop Documents
To develop a user specifications document describing the operating conditions

Evaluation
To evaluate the software as per the specifications and submit a validation report

Incorporating Changes
4

unit III

SOFTWARE QUALITY ASSURANCE

To change the software so as to remove any errors found during evaluation.

3.2.4. Techniques
There are various validation techniques like fault injection, dependability analysis, etc.,
that are commonly used for software testing. Here are a few more.
1. Formal Methods:
This technique makes use of mathematical and logical techniques to analyze the input
specifications, the product design, the supporting documents, and the behavior of the product
under test.
2. Fault Injection
As the name suggests, faults or bugs are intentionally added to software so as to check its
working/functionality under the said conditions. These are of two types - hardware fault injection
and software fault injection. In the former, faults in physical hardware are injected, while in the
latter, errors are injected in the software or in the system's memory.
3. Dependability Analysis
This technique is used to increase the dependability of the product. The hazards in the
software are identified, and methods are suggested to reduce theses potential hazards.
4. Hazard Analysis
Every software has its own standard tests to identify hazards. This type of analysis
follows the said procedure to identify the hazard and find out its countermeasures.
5. Risk Analysis
This takes the hazard analysis further by identifying countermeasures for each type of
hazard that is identified.

3.3. TEST PLANS

Test plan is the project plan for the testing work to be done. It is not a test design
specification, a collection of test cases or a set of test procedures; in fact, most of our test plans
do not address that level of detail. Many people have different definitions for test plans.
There are three main reasons to write the test plans:
First, by writing a test plan it guides our thinking. Writing a test plan forces us to
confront the challenges that await us and focus our thinking on important topics.

unit III

SOFTWARE QUALITY ASSURANCE

Fred Brooks explains the importance of careful estimation and planning for testing in one of his
book as follows:
Failure to allow enough time for system test, in particular, is peculiarly disastrous. Since
the delay comes at the end of the schedule, no one is aware of schedule trouble until almost the
delivery date [and] delay at this point has unusually severe financial repercussions. The project is
fully staffed, and cost-per day is maximum [as are the associated opportunity costs], It is
therefore very important to allow enough system test time in the original schedule.
Second, the test planning process and the plan itself serve as the means of
communication with other members of the project team, testers, peers, managers and other
stakeholders. This communication allows the test plan to influence the project team and the
project team to influence the test plan, especially in the areas of organization-wide testing
policies and motivations; test scope, objectives and critical areas to test; project and product
risks, resource considerations and constraints; and the testability of the item under test. We can
complete this communication by circulating one or two test plan drafts and through review
meetings. Such a draft will include many notes such as the following:
Third, the test plan helps us to manage change. During early phases of the project, as we
gather more information, we revise our plans. As the project evolves and situations change, we
adapt our plans. By updating the plan at major milestone helps us to keep testing aligned with
project needs. As we run the tests, we make final adjustments to our plans based on the results.
You might not have the time or the energy to update your test plans every time a change is
made in the project, as some projects can be quite dynamic.
At times it is better to write multiple test plans in some situations. For example, when we
manage both integration and system test levels, those two test execution periods occur at
different points in time and have different objectives. For some systems projects, a hardware test
plan and a software test plan will address different techniques and tools as well as different
audiences. However, there are chances that these test plans can get overlapped, hence, a master
test plan should be made that addresses the common elements of both the test plans can reduce
the amount of redundant documentation.
3.3.1. IEEE 829 STANDARD TEST PLAN TEMPLATES:

Test plan identifier.

Test deliverables Introduction.
Test tasks.
Test items.
Environmental needs.
Features to be tested.
Responsibilities.
Features not to be tested.
unit III

SOFTWARE QUALITY ASSURANCE

Staffing and training needs.

Approach Schedule.
Item pass/fail criteria.
Risks and contingencies.
Suspension and resumption criteria Approvals.

3.4. TEST CASE

A test case, in software engineering, is a set of conditions or variables under which a
tester will determine whether an application, software systemor one of its features is working as
it was originally established for it to do. The mechanism for determining whether a software
program or system has passed or failed such a test is known as a test oracle. In some settings, an
oracle could be a requirement or use case, while in others it could be a heuristic. It may take
many test cases to determine that a software program or system is considered sufficiently
scrutinized to be released. Test cases are often referred to as test scripts, particularly when
written - when they are usually collected into test suites.
3.4.1. Formal test cases
In order to fully test that all the requirements of an application are met, there must be at
least two test cases for each requirement: one positive test and one negative test. If a requirement
has sub-requirements, each sub-requirement must have at least two test cases. Keeping track of
the link between the requirement and the test is frequently done using a traceability matrix.
Written test cases should include a description of the functionality to be tested, and the
preparation required to ensure that the test can be conducted.
A formal written test-case is characterized by a known input and by an expected output,
which is worked out before the test is executed. The known input should test a precondition and
the expected output should test a post condition.
3.4.2. Informal test cases
For applications or systems without formal requirements, test cases can be written based
on the accepted normal operation of programs of a similar class. In some schools of testing, test
cases are not written at all but the activities and results are reported after the tests have been run.
In scenario testing, hypothetical stories are used to help the tester think through a
complex problem or system. These scenarios are usually not written down in any detail. They
can be as simple as a diagram for a testing environment or they could be a description written in
prose. The ideal scenario test is a story that is motivating, credible, complex, and easy to
evaluate. They are usually different from test cases in that test cases are single steps while
scenarios cover a number of steps of the key.
In its most basic form, a test case is a simple pair of <input, expected outcome>. If a
program under test is expected to compute the square root of nonnegative numbers, then four
examples of test cases are as shown in Figure 2.
In stateless systems, where the outcome depends solely on the current input, test cases are
very simple in structure, as shown in Figure 2.
7

unit III

SOFTWARE QUALITY ASSURANCE

A program to compute the square root of nonnegative numbers is an example of a

stateless system. A compiler for the C programming language is another example of a stateless
system. A compiler is a stateless system because to compile a program it does not need to know
about the programs it compiled previously.
TB1: <0, 0>,
TB2: <25, 5>,
TB3: <40, 6.3245553>,
TB4: <100.5, 10.024968>.

Figure 2: Examples of basic test cases.

TS1: <check balance, $500.00>, <withdraw, amount?>,
<$200.00, $200.00>, <check balance, $300.00>.

Figure 3: Example of a test case with a sequence of <input, expected outcome>.

In state-oriented systems, where the program outcome depends both on the current state
of the system and the current input, a test case may consist of a sequence of <input, expected
outcome>pairs. A telephone switching system and an automated teller machine (ATM) are
examples of state-oriented systems. For an ATM machine, a test case for testing the withdraw
function is shown in Figure 3 Here, we assume that the user has already entered validated inputs,
such as the cash card and the personal identification number (PIN).
In the test case TS1, check balance and withdraw in the first, second, and fourth
tuples represent the pressing of the appropriate keys on the ATM keypad. It is assumed that the
user account has $500.00 on it, and the user wants to withdraw an amount of $200.00. The
expected outcome $200.00 in the third tuple represents the cash dispensed by the ATM. After
the withdrawal operation, the user makes sure that the remaining balance is $300.00.
For state-oriented systems, most of the test cases include some form of decision and
timing in providing input to the system. A test case may include loops and timers, which we do
not show at this moment.

3.5. TEST GENERATION

Any form of test generation uses a source document. In the most informal of test
methods, the source document resides in the mind of the tester who generates tests based on the
knowledge of the requirements. In several commercial environments, the process is a bit more
formal. The tests are generated using a mix of formal and informal methods either directly from
the requirements document serving as the source. In more advanced test processes, requirements
serve as a source for the development of formal models.

unit III

SOFTWARE QUALITY ASSURANCE

3.5.1. Test generation strategies

Model based: require that a subset of the requirements be modeled using a formal
notation (usually graphical). Models: Finite State Machines, Timed automata, Petri net, etc.
Specification based: require that a subset of the requirements be modeled using a formal
mathematical notation. Examples: B, Z, and Larch.
Code based: generate tests directly from the code.

3.5.2. Test generation techniques

Four techniques are considered:

Equivalence partitioning
Boundary value analysis
Cause effect graphing and
Predicate based test generation.

Each of these test generation techniques is a black-box technique and useful for generating test
cases during functional testing.

unit III

SOFTWARE QUALITY ASSURANCE

3.5.3. The test selection problem

Requirements and test generation
Requirements serve as the starting point for the generation of tests. During the initial
phases of development, requirements may exist only in the minds of one or more people. These
requirements, more aptly ideas, are then specified rigorously using modeling elements such as
use cases, sequence diagrams, and state charts in UML.
Rigorously specified requirements are often transformed into formal requirements using
requirements specification languages such as Z, S, and RSML.

3.6. EQUIVALENCE CLASS PARTITIONING

An input domain may be too large for all its elements to be used as test input. However,
the input domain can be partitioned into a finite number of sub domains for selecting test inputs.
Each sub domain is known as an equivalence class (EC), and it serves as a source of at least one
test input. The objective of equivalence partitioning is to divide the input domain of the system
under test into classes, or groups, of inputs. All the inputs in the same class have a similar effect
on the system under test [14, 15]. An EC is a set of inputs that the system treats identically when
10

unit III

SOFTWARE QUALITY ASSURANCE

the system is tested. It represents certain conditions, or predicates, on the input domain. An input
condition on the input domain is a predicate over the values of the input domain. A valid input to
a system is an element of the input domain that is expected to return a non error value. An
invalid input is an input that is expected to return an error value. Input conditions are used to
partition the input domain into ECs for the purpose of selecting inputs.
Guidelines for EC Partitioning Equivalence classes can be derived from an input domain
by a heuristic technique. One can approximate the ECs by identifying classes for which different
program behaviors are specified. Identification of ECs becomes easier with experience. Myers
suggests the following guidelines to identify ECs [16].
1. An input condition specifies a range [a, b]: Identify one EC for a X b and two other
classes for X <a and X>b to test the system with invalid inputs.
2. An input condition specifies a set of values: Create one EC for each element of the set
and one EC for an invalid member. For example, if the input is selected from a set of N items,
then N + 1 ECs are created:
(i) One EC for each element of the set {M1}, {M2}, . . . , {MN}
(ii) One EC for elements outside the set {M1,M2, . . .,MN}.
3. Input condition specifies for each individual value: If the system handles each valid
input differently, then create one EC for each valid input. For example, if the input is from a
menu, then create one EC for each menu item.

4. An input condition specifies the number of valid values (say N): Create one EC for the
correct number of inputs and two ECs for invalid inputsone for zero values and one for more
than N values. For example, if a program can accept 100 natural numbers for sorting, then three
ECs are created:
11

unit III

SOFTWARE QUALITY ASSURANCE

(i) One for 100 valid input of natural numbers.

(ii) One for no input value.
(iii) One for more than 100 natural numbers.
5. An input condition specifies a must-be value: Create one EC for a must-be value and
one EC for something that is not a must-be value.
For example, if the first character of a password must be a numeric character, then we are
required to generate two ECs:
(i)
(ii)

one for valid values, {pswd | the first character ofpswd has a numeric
value},
(ii) one for invalid values, {pswd | the first character of pswd is not
numeric}.

6. Splitting of EC: If elements in a partitioned EC are handled differently by the system,

then split the EC into smaller ECs. Identification of Test Cases from ECs Having identified the
ECs of an input domain of a program, test cases for each EC can be identified by the following:
Step 1: Assign a unique number to each EC.
Step 2: For each EC with valid input that has not been covered by test cases yet, write a new test
case covering as many uncovered ECs as possible.
Step 3: For each EC with invalid input that has not been covered by test cases, write a new test
case that covers one and only one of the uncovered ECs.
In summary, the advantages of EC partitioning are as follows:
A small number of test cases are needed to adequately cover a large input domain.
One gets a better idea about the input domain being covered with theselected test cases.
The probability of uncovering defects with the selected test cases based on EC
partitioning is higher than that with a randomly chosen test suite of the same size.
The EC partitioning approach is not restricted to input conditions alone;the technique
may also be used for output domains.
Example: Adjusted Gross Income. Consider a software system that computes
income tax based on adjusted gross income (AGI) according to the following rules:
If AGI is between $1 and $29,500, the tax due is 22% of AGI.
If AGI is between $29,501 and $58,500, the tax due is 27% of AGI.
If AGI is between $58,501 and $100 billion, the tax due is 36% of AGI.

12

unit III

SOFTWARE QUALITY ASSURANCE

In this case, the input domain is from $1 to $100 billion. There are three input conditions in the
example:
1. $1 AGI $29,500.
2. $29,501 AGI $58,500.
3. $58,501 AGI $100 billion.
First we consider condition 1, namely, $1 AGI $29,500, to derive two ECs:
EC1: $1 AGI $29,500; valid input.
EC2: AGI < 1; invalid input.
Then, we consider condition 2, namely, $29,501 AGI $58,500, to derive one EC:
EC3: $29,501 AGI $58,500; valid input.
Finally, we consider condition 3, namely, $58,501 AGI $100 billion, to derive two ECs:
EC4: $58,501 AGI $100 billion; valid input.
EC5: AGI > $100 billion; invalid input.
Note that each condition was considered separately in the derivation of ECs. Conditions are not
combined to select ECs. Five test cases are generated to cover the five ECs, as shown in Table
9.10.
In the EC partition technique, a single test input is arbitrarily selected to cover a specific
EC. We need to generate specific test input by considering the extremes, either inside or outside
of the defined EC partitions. This leads us to the next technique, known as boundary value
analysis, which focuses on the boundary of the ECs to identify test inputs.

3.7. BOUNDARY VALUE ANALYSIS

Boundary value analysis is a test selection technique that targets faults in applications at
the boundaries of equivalence classes. While equivalence partitioning selects tests from within
13

unit III

SOFTWARE QUALITY ASSURANCE

equivalence classes, boundary value analysis focuses on tests at and near the boundaries of
equivalence classes. Certainly, tests derived using either of the two techniques may overlap.
The central idea in boundary value analysis (BVA) is to select test data near the boundary
of a data domain so that data both within and outside an EC are selected. It produces test inputs
near the boundaries to find failures caused by incorrect implementation of the boundaries.
Boundary conditions are predicates that apply directly on and around the boundaries of input
ECs and output ECs. In practice, designers and programmers tend to overlook boundary
conditions.
Consequently, defects tend to be concentrated near the boundaries between ECs.
Therefore, test data are selected on or near a boundary. In that sense, the BVA technique is an
extension and refinement of the EC partitioning technique. In the BVA technique, the boundary
conditions for each EC are analyzed in order to generate test cases.
3.7.1. BVA: Procedure
1. Partition the input domain using uni-dimensional partitioning. This leads to as many
partitions as there are input variables. Alternately, a single partition of an input domain can be
created using multidimensional partitioning. We will generate several sub-domains in this step.
2. Identify the boundaries for each partition. Boundaries may also be identified using
special relationships amongst the inputs.
3. Select test data such that each boundary value occurs in at least one test input.
3.7.2. Guidelines for BVA:
As in the case of EC partitioning, the ability to develop high-quality effective test cases
using BVA requires experience. The guidelines discussed below are applicable to both input
conditions and output conditions. The conditions are useful in identifying high-quality test cases.
By high-quality test cases we mean test cases that can reveal defects in a program.
1. The EC specifies a range:
If an EC specifies a range of values, then construct test cases by considering the boundary
points of the range and points just beyond the boundaries of the range. For example, let an EC
specify the range of 10.0 X 10.0. This would result in test data {9.9 10.0,10.1} and
{9.9, 10.0, 10.1}.
2. The EC specifies a number of values:
If an EC specifies a number of values, then construct test cases for the minimum and the
maximum value of the number. In addition, select a value smaller than the minimum and a value
14

unit III

SOFTWARE QUALITY ASSURANCE

larger than the maximum value. For example, let the EC specification of a student dormitory
specify that a housing unit can be shared by one to four students; test cases that include 1, 4, 0,
and 5 students would be developed.
3. The EC specifies an ordered set:
If the EC specifies an ordered set, such as a linear list, table, or sequential file, then focus
attention on the first and last elements of the set.
Example:
Let us consider the five ECs identified in our previous example to compute income tax based
on AGI. The BVA technique results in test as follows for each EC. The redundant data points
may be eliminated.
EC1: $1 AGI $29,500; this would result in values of $1, $0, $1, $1.50 and $29,499.50,
$29,500, $29,500.50.
EC2: AGI <1; This would result in values of $1, $0, $1, $100 billion.
EC3: $29,501 AGI $58,500; this would result in values of $29,500, $29,500.50, $29,501,
$58,499, $58,500, $58,500.50, $58,501.
EC4: $58,501 AGI $100 billion; this would result in values of $58,500, $58,500.50,
$58,501, $100 billion, $101 billion.
EC5: AGI >$100 billion; this would result in $100 billion, $101 billion, $10000 billion.
Remark:
Should we test for an AGI value of $29,500.50 (i.e., between the partitions), and if so,
what should be the result? Since we have not been told whether the decimal values are actually
possible, the best decision to make is to test for this value and report the result.

3.8. CATEGORY PARTITION METHOD

The category partition method (CPM) is a generalization and formalization of a classical
functional testing approach. The reader is reminded of the two steps of the classical functional
testing approach:
(i) Partition the input domain of the functional unit to be tested into equivalence classes.
(ii) Select test data from each EC of the partition.
The CPM is a systematic, specification-based methodology that uses an informal
functional specification to produce formal test specification. The test designers key job is to
develop categories, which are defined to be the major characteristics of the input domain of the
function under test. Each category is partitioned into ECs of inputs called choices. The choices in
15

unit III

SOFTWARE QUALITY ASSURANCE

each category must be disjoint, and together the choices in each category must cover the input
domain. Grochtmann and Grimm extend this approach by capturing the constraints using a tree
structure to reduce the number of impossible test cases.
The main advantage of this approach is the creation of a formal test specification written
in languages such as test specification language (TSL), Z or testing and test control notation
(TTCN) to represent an informal or natural language formal specification. The formal test
specification gives the test engineers a logical way to control test generation and is easily
modified to accommodate changes or mistakes in the functional specification. The use of Zallows
more flexibility in the specification of constraints and more formalities in the representation
The category partition testing method is comprised of the following steps:
Step 1: Analyze the Specification
The method begins with the decomposition of a functional specification into functional
units that can be separately tested. For each functional unit, identify the following:
Parameters of the functional unit
Characteristics of each parameter, that is, the elementary characteristics of the
parameters that affect the execution of the unit
Objects in the environment whose state might affect the operation of the functional unit
Characteristics of each environment object Parameters are the explicit input to a
functional unit, and environment conditions are the state characteristics of the system at the time
of execution of a functional specification.
Step 2: Identify Categories
A category is a classification of the major properties(or characteristics) of a parameter or
an environmental condition. Consider, for instance, a program PSORT that reads an input file F
containing a variable-length array of values of arbitrary type. The expected output is a
permutation of input with values sorted according to some total ordering criterion. The
environmental condition for PSORT is status of F, which can be classified into three categories,
namely Status of F = Does Not Exist, Status of F = Exists But Empty, and Status of F = Exists
and Nonempty. The properties of the input parameter categories for the PSORT are as follows:
the array size, the type of elements, the minimum element value, the maximum element value,
and the positions in the array of the maximum and minimum values. Categories can be derived
directly from the functional specification, implicit design in formation, or intuition of the test
engineer. Often categories are derived from preconditions and type information about the input
parameters and system state components.
Step 3: Partition Categories into Choices
Partition each category into distinct choices that include all the different kinds of values
that are possible for the category. Each choice is an equivalent class of values assumed to have
identical properties as far as testing and error detection capability are concerned. The choices
16

unit III

SOFTWARE QUALITY ASSURANCE

must be disjoint and cover all the categories. While the categories are derived from a functional
specification, the choices can be based on specification and the test engineers past experience of
designing effective test cases, such as error guessing. In the sorting program PSORT example,
one possible way to partition the category array size is size = 0, size = 1, 2 size 100, and
size >100. These choices are based primarily on experience with likely errors.
The selection of a single choice from each category determines a test frame, which is the
basis for constructing the actual test cases. A test frame consists of a set of choices from the
specification, with each category contributing either zero or one choice. Since the choices in
different categories frequently interact with each other in ways that affect the resulting test cases,
the choices can be annotated with constraints (see next step) to indicate these relations. In the
absence of constraints the number of potential test frames is the product of the number of choices
in each categoryand this is likely to be very large.
Step 4: Determine Constraints among Choices
Constraints are restrictions among the choices within different categories that can interact
with one another. A typical constraint specifies that a choice from one category cannot occur
together in a test frame with certain choices from another category. Choices and constraints are
derived from the natural language functional specifications but can often be specified by formal
methods, thus making their analysis easier to automate. With a careful specification of
constraints, the number of potential test frames can be reduced to a manageable number.
Step 5: Formalize and Evaluate Test Specification
Specify the categories, choices, and constraints using a specification technique that is
compatible with the test generation tool, such as TSL, Z, or TTCN, that produces test frames.
Most test generation tools also provide automated techniques for evaluating the internal
consistency of the formal specification. This
evaluation often discovers errors or inconsistencies in the specification of constraints and
sometimes leads to discovery of errors in the source functional specification.
Step 6: Generate and Validate Test Cases
The final step in the test production process is to transform the generated test frames into
executable test cases. If the test specification includes post conditions that must be satisfied, then
the tool verifies the post conditions. In case a reference Implementation is available, the test
cases can be validated by executing them and checking their results against the reference
implementation. The validation of test cases is a labor-intensive process in the absence of a
reference implementation.

3.9. COMBINATORIAL TESTING

3.9.1. Combinatorial testing: Basic idea
Identify distinct attributes that can be varied in the data, environment, or configuration
17

unit III

SOFTWARE QUALITY ASSURANCE

Example: browser could be IE or Firefox, operating system could be Vista,

XP, or OSX
Systematically generate combinations to be tested
Example: IE on Vista, IE on XP, Firefox on Vista, Firefox on OSX.
Rationale: Test cases should be varied and include possible corner cases
3.9.2. Key ideas in combinatorial approaches
Category-partition testing
Separate (manual) identification of values that characterize the input space from
(automatic) generation of combinations for test cases
Pair wise testing
Systematically test interactions among attributes of the program input space with a
relatively small number of test cases
Catalog-based testing
Aggregate and synthesize the experience of test designers in a particular organization or
application domain, to aid in identifying attribute values
Category partition (manual steps)
1. Decompose the specification into independently testable features. For each feature identify
Parameters
Environment elements
For each parameter and environment element identify elementary characteristics (categories)
2. Identify relevant values for each characteristic (category) identify (classes of) values
Normal values
Boundary values
Special values
Error values
3. Introduce constraints
18

unit III

SOFTWARE QUALITY ASSURANCE

An informal specification: check configuration

Check Configuration
Check the validity of a computer configuration
The parameters of check-configuration are:
Model
Set of components
An informal specification: Parameter model:
A model identifies a specific product and determines a set of constraints on available
components. Models are characterized by logical slots for components, which may or may not be
implemented by physical slots on a bus. Slots may be required or optional. Required slots must
be assigned with a suitable component to obtain a legal configuration, while optional slots may
be left empty or filled depending on the customers' needs.
An informal specification of parameter set of components:
A set of (slot, component) pairs, corresponding to the required andoptional slots of the
model. A component is a choice that can be varied within a model, and which is not designed to
be replaced by the end user. Available components and a default for each slot is determined by
the model. The special value empty is allowed (and may be the default selection) for optional
slots. In addition to being compatible or incompatible with a particular model and slot, individual
components may be compatible or incompatible with each other.
Step1: Identify independently testable units and categories
Choosing categories
No hard-and-fast rules for choosing categories
Not a trivial task!
Categories reflect test designer's judgment
Regarding which classes of values may be treated differently by an implementation
Choosing categories well requires experience and knowledge
Of the application domain and product architecture.
19

unit III

SOFTWARE QUALITY ASSURANCE

The test designer must look under the surface of the specification and identify hidden
characteristics
Identify independently testable units
Parameter Model
Model number
Number of required slots for selected model (#SMRS)
Number of optional slots for selected model (#SMOS)
Parameter Components
Correspondence of selection with model slots

Required component selection

Optional component selection

Environment element: Product database
Number of models in database (#DBM)
Number of components in database (#DBC)
Step 2: Identify relevant values
Identify (list) representative classes of values for each of the categories
Ignore interactions among values for different categories (considered in the next step)
Representative values may be identified by applying Boundary value testing
select extreme values within a class
select values outside but as close as possible to the class
select interior (non-extreme) values of the class - Erroneous condition testing
select values outside the normal domain of the program
Identify relevant values of Model
Model number, malformed, not in database, valid values are identified.
20

unit III

SOFTWARE QUALITY ASSURANCE

Number of required slots for selected model (#SMRS)

0
1
Many
Number of optional slots for selected model (#SMOS)
0
1
Many
Identify relevant values: Component
Correspondence of selection with model slots
Omitted slots
Extra slots
Mismatched slots
Complete correspondence
Number of required components with non empty selection
0
Less than number required slots
Equal to number required slots
Required component selection
Some defaults
All valid

Number of optional components with non empty selection

21

unit III

SOFTWARE QUALITY ASSURANCE

0
<
Less than #SMOS
Equal to #SMOS
Optional component selection
Some defaults
All valid

selection

Identify relevant values: Database

Number of models in database (#DBM)
0
1
Many
Number of components in database (#DBC)
0
1
Many
Note 0 and 1 are unusual (special) values. They might cause unanticipated behavior alone or in
combination with particular values of other parameters
Step 3: Introduce constraints
A combination of values for each category corresponds to a test case specification
Example: zero slots and at least one incompatible slot

22

unit III

SOFTWARE QUALITY ASSURANCE

Introduce constraints to rule out impossible combinations and to reduce the size of the test suite
if too large.
Error constraint
[error] indicates a value class that
corresponds to a erroneous values
need be tried only once
Example: Model number: Malformed and Not in database
Error value classes
No need to test all possible combinations of errors
One test is enough (we assume that handling an error case bypasses other program
logic)
Property constraints
Constraint [property] [if-property] rule out invalid combinations of values [property] groups
values of a single parameter to identify subsets of values with common properties [if-property]
bounds the choices of values for a category that can be combined with a particular value selected
for a different category
Single constraints
[Single] indicates a value class that test designers choose to test only once to reduce the
number of test cases
Example value some default for required component selection and optional component selection
may be tested only once despite not being an erroneous condition
Example - Single constraints
Number of required slots for selected model (#SMRS)

0 [single]

1 [property RSNE] [single]

Number of optional slots for selected model (#SMOS)

23

0 [single]
unit III

SOFTWARE QUALITY ASSURANCE

1 [single] [property OSNE]

Number of models in database (#DBM)

1 [single]

Number of components in database (#DBC)

1 [single]

Category partition testing gave us

Systematic approach: Identify characteristics and values (the creative step), generate
combinations (the mechanical step) but Test suite size grows very rapidly with number of
categories
Pairwise (and n-way) combinatorial testing do
Combine values systematically but not exhaustively
Rationale: Most unplanned interactions are among just two or a few parameters or
parameter characteristics
Pairwise combinatorial testing
Category partition works well when intuitive constraints reduce the number of combinations to
a small amount of test cases without many constraints, the number of combinations may be
unmanageable.
Pairwise combination (instead of exhaustive) Generate combinations that efficiently cover all
pairs (triples) of classes
Rationale: most failures are triggered by single values or combinations of a few values.
Covering pairs (triples,) reduces the number of test cases, but reveals most faults
Category-partition approach gives us,
Separation between (manual) identification of parameter characteristics and values and
(automatic) generation of test cases that combine them
Constraints to reduce the number of combinations

24

unit III

SOFTWARE QUALITY ASSURANCE

3.10. DECISION TABLES

A major limitation of the EC-based testing is that it only considers each input separately.
The technique does not consider combining conditions. Different combinations of equivalent
classes can be tried by using a new technique based on the decision table to handle multiple
inputs. Decision tables have been used for many years as a useful tool to model software
requirements and design decisions. It is a simple, yet powerful notation to describe complex
systems from library information management systems to embedded real-time systems.
The general structure of a decision table comprises a set of conditions (or causes) and a
set of effects (or results) arranged in the form of a column on the left of the table. In the second
column, next to each condition, we have its possible values: yes (Y), no (N), and dont care
(dash). To the right of the values column, we have a set of rules. For each combination of the
three conditions {C1, C2, C3}, there exists a rule from the set {R1, R2 R8}. Each rule
comprises a yes, no, or dont care response and contains an associated list of effects {E1, E2,
E3}. Then, for each relevant effect, an effect sequence number specifies the order in which the
effect should be carried out if the associated sets of conditions are satisfied. For example, if C1
and C2 are true but C3 is not true, then E3 should be followed by E1. The checksum is used for
verification of the combinations the decision table represents.

Test data are selected so that each rule in a table is exercised and the actual results are
verified with the expected results. In other words, each rule of a decision table represents a test
case. The steps in developing test cases using the decision table technique are as follows:
Step 1:
The test designer needs to identify the conditions and the effects for each specification
unit. A condition is a distinct input condition or an EC of input conditions. An effect is an output
condition. Determine the logical relationship between the conditions and the effects.
25

unit III

SOFTWARE QUALITY ASSURANCE

Step 2:
List all the conditions and effects in the form of a decision table. Write down the values
the condition can take.
Step 3:
Calculate the number of possible combinations. It is equal to the number of different
values raised to the power of the number of conditions.
Step 4:
Fill the columns with all possible combinationseach column corresponds to one
combination of values. For each row (condition) do the following:
1. Determine the repeating factor (RF): divide the remaining number of combinations by
the number of possible values for that condition.
2. Write RF times the first value, then RF times the next, and so forth until the row is full.
Step 5:
Reduce combinations (rules). Find indifferent combinationsplace a dash and join
column where columns are identical. While doing this, ensure that effects are the same.
Step 6:
Check covered combinations (rules). For each column calculate the combinations it
represents. A dash represents as many combinations as the condition has. Multiply for each dash
down the column. Add up total and compare with step 3. It should be the same.
Step 7:
Add effects to the column of the decision table. Read column by column and determine
the effects. If more than one effect can occur in a single combination, then assign a sequence
number to the effects, there by specifying the order in which the effects should be performed.
Check the consistency of the decision table.
Step 8:
The columns in the decision table are transformed into test cases. Decision tablebased
testing is effective under certain conditions as follows:
The requirements are easily mapped to a decision table.
The resulting decision table should not be too large. One can break down a large
decision table into multiple smaller tables.
Each column in a decision table is independent of the other columns.
A decision table is a good way to deal with combinations of things (e.g. inputs). This
technique is sometimes referred to as a cause-effect table. The reason for this is that there is an
associated logic diagramming technique called cause-effect graphing which was sometimes
used to help derive the decision table.
26

unit III

SOFTWARE QUALITY ASSURANCE

Decision tables provide a systematic way of stating complex business rules, which is
useful for developers as well as for testers.

Decision tables can be used in test design whether or not they are used in specifications,
as they help testers explore the effects of combinations of different inputs and other
software states that must correctly implement business rules.

It helps the developers to do a better job can also lead to better relationships with them.
Testing combinations can be a challenge, as the number of combinations can often be
huge. Testing all combinations may be impractical if not impossible. We have to be
satisfied with testing just a small subset of combinations but making the choice of which
combinations to test and which to leave out is also important. If you do not have a
systematic way of selecting combinations, an arbitrary subset will be used and this may
well result in an ineffective test effort.

How to Use decision tables for test designing?

The first task is to identify a suitable function or subsystem which reacts according to a
combination of inputs or events. The system should not contain too many inputs otherwise the
number of combinations will become unmanageable. It is better to deal with large numbers of
conditions by dividing them into subsets and dealing with the subsets one at a time. Once you
have identified the aspects that need to be combined, then you put them into a table listing all the
combinations of True and False for each of the aspects.
Let us consider an example of a loan application, where you can enter the amount of the
monthly repayment or the number of years you want to take to pay it back (the term of the loan).
If you enter both, the system will make a compromise between the two if they conflict. The two
conditions are the loan amount and the term, so we put them in a table (see Table 4.2).
TABLE 3.2 Empty decision table:
Conditions

Rule 1

Rule 2

Rule 3

Rule 4

Repayment amount
has been entered:
Term of loan has been
Entered:

Next we will identify all of the combinations of True and False (see Table 4.3). With two
conditions, each of which can be True or False, we will have four combinations (two to the
power of the number of things to be combined). Note that if we have three things to combine, we
will have eight combinations, with four things, there are 16, etc. This is why it is good to tackle
small sets of combinations at a time. In order to keep track of which combinations we have, we
will alternate True and False on the bottom row, put two Trues and then two Falses on the row
27

unit III

SOFTWARE QUALITY ASSURANCE

above the bottom row, etc., so the top row will have all Trues and then all Falses (and this
principle applies to all such tables).

TABLE 3.3 Decision table with input combinations:

Conditions

Rule 1

Rule 2

Rule 3

Rule 4

Repayment amount has

been entered:

Term of loan has

been entered:

In the next step we will now identify the correct outcome for each combination (see
Table 4.4). In this example, we can enter one or both of the two fields. Each combination is
sometimes referred to as a rule.

TABLE 3.4 Decision table with combinations and outcomes:

Conditions

Rule 1

Rule 2

Rule 3

Rule 4

Repayment amount has

been entered:

Term of loan has been

entered:

Actions/Outcomes
Process loan amount:

Process term:

Y
Y

At this point, we may realize that we hadnt thought about what happens if the customer
doesnt enter anything in either of the two fields. The table has highlighted a combination that
was not mentioned in the specification for this example. We could assume that this combination
should result in an error message, so we need to add another action (see Table 4.5). This
highlights the strength of this technique to discover omissions and ambiguities in specifications.
It is not unusual for some combinations to be omitted from specifications; therefore this is also a
valuable technique to use when reviewing the test basis.
28

unit III

SOFTWARE QUALITY ASSURANCE

TABLE 3.5 Decision table with additional outcomes:

Conditions

Rule 1

Rule 2

Rule 3

Rule 4

Repayment amount
has been entered:

Term of loan has been

entered:

Actions/Outcomes
Process loan amount:

Process term:

Y
Y

Error message:

Now, we make slight change in this example, so that the customer is not allowed to enter
both repayment and term. Now the outcome of our table will change, because there should also
be an error message if both are entered, so it will look like Table 4.6.

TABLE 3.6 Decision table with changed outcomes:

Conditions

Rule 1

Rule 2

Rule 3

Rule 4

Repayment amount has

been entered:

Term of loan has been

entered:

Actions/Outcomes
Process loan amount:

Process term:
Error message:

Y
Y

You might notice now that there is only one Yes in each column, i.e. our actions are
mutually exclusive only one action occurs for each combination of conditions. We could
29

unit III

SOFTWARE QUALITY ASSURANCE

represent this in a different way by listing the actions in the cell of one row, as shown in Table
4.7. Note that if more than one action results from any of the combinations, then it would be
better to show them as separate rows rather than combining them into one row.

TABLE 3.7 Decision table with outcomes in one row:

Conditions

Rule 1

Rule 2

Rule 3

Rule 4

Repayment amount has

been entered:

Term of loan has been

entered:

Actions/Outcomes:
Result: Error Process loan Process Error message amount term message. The final step
of this technique is to write test cases to exercise each of the four rules in our table.
Credit card example:
Lets take another example. If you are a new customer and you want to open a credit card
account then there are three conditions first you will get a 15% discount on all your purchases
today, second if you are an existing customer and you hold a loyalty card, you get a 10%
discount and third if you have a coupon, you can get 20% off today (but it cant be used with the
new customer discount). Discount amounts are added, if applicable. This is shown in Table 4.8.
TABLE 3.8 Decision table for credit card example

In Table 4.8, the conditions and actions are listed in the left hand column. All the other
columns in the decision table each represent a separate rule, one for each combination of
conditions. We may choose to test each rule/combination and if there are only a few this will
usually be the case. However, if the number of rules/combinations is large we are more likely to
sample them by selecting a rich subset for testing.
Now lets see the decision table for credit card shown above:

30

Note that we have put X for the discount for two of the columns (Rules 1 and 2) this
means that this combination should not occur. You cannot be both a new customer and
unit III

SOFTWARE QUALITY ASSURANCE

also holding a loyalty card as per the conditions mentioned above. Hence there should be
an error message stating this.

We have made an assumption in Rule 3. Since the coupon has a greater discount than the
new customer discount, we assume that the customer will choose 20% rather than 15%.
We cannot add them, since the coupon cannot be used with the new customer discount
as stated in the condition above. The 20% action is an assumption on our part, and we
should check that this assumption (and any other assumptions that we make) is correct,
by asking the person who wrote the specification or the users.

For Rule 5, however, we can add the discounts; since both the coupon and the loyalty
card discount should apply (thats our assumption).

Rules 4, 6 and 7 have only one type of discount and Rule 8 has no discount, so 0%.

If we are applying this technique thoroughly, we would have one test for each column or rule of
our decision table. The advantage of doing this is that we may test a combination of things that
otherwise we might not have tested and that could find a defect. However, if we have a lot of
combinations, it may not be possible or sensible to test every combination. If we are timeconstrained, we may not have time to test all combinations. Dont just assume that all
combinations need to be tested. It is always better to prioritize and test the most important
combinations. Having the full table helps us to decide which combinations we should test and
which not to test this time. In the example above all the conditions are binary, i.e. they have only
two possible values: True or False (or we can say Yes or No).
?

31

unit III