UNIT-V

Efficient Test Suit Management

Test case design does a test suite grow, Minimizing the test suite and its benefits, test
suite prioritization, Types of test case prioritization, prioritization techniques, measuring
the effectiveness of a prioritized test suite.
Software Quality Management: Software Quality metrics, SQA models Debugging:
process, techniques, correcting bugs, Basics of testing management tools, test link and
Jira.

Software testing is a continuous process that takes place throughout the life cycle of a
project. Test cases in an existing test suite can often be used to test a modifi ed program.
However, if the test suite is inadequate for retesting, new test cases may be developed and
added to the test suite. Thus, the size of a test suite grows as the software evolves. Due to
resource constraints, it is important to prioritize the execution of test cases so as to
increase the chances of early detection of faults. A reduction in the size of the test suite
decreases both the overhead of maintaining the test suite and the number of test cases that
must be rerun after changes are made to the software. This chapter discusses the
techniques of minimizing a test suite.
12.1 WHY DOES A TEST SUITE GROW?
A testing criterion is a rule or collection of rules that imposes requirements on a set of test
cases. Test engineers measure the extent to which a criterion is satisfied in terms of
coverage; a test set achieves 100% coverage if it completely satisfies the criterion.
Coverage is measured in terms of the requirements that are imposed; partial coverage is
defined as the percentage of requirements that are satisfied.
Test requirements are specific things that must be satisfied or covered; e.g. for statement
coverage, each statement is a requirement; in mutation, each mutant is a requirement; and
in data flow testing, each DU pair is a requirement.
Coverage criteria are used as a stopping point to decide when a program is sufficiently
tested. In this case, additional tests are added until the test suite has achieved a specified
coverage level according to a specific adequacy criterion. For example, to achieve
statement coverage adequacy for a program, one would add additional test cases to the
test suite until each statement in that program is executed by at least one test case.
Test suites can also be reused later as the software evolves. Such test suite reuse, in the
form of regression testing, is pervasive in the software industry. These tests account for as
much as one half of the cost of software maintenance. Running all the test cases in a test
suite, however, can require a lot of effort.
 There may be unnecessary test cases in the test suite including both obsolete and
redundant test cases. A change in a program causes a test case to become obsolete by
removing the reason for the test case’s inclusion in the test suite. A test case is redundant
if other test cases in the test suite provide the same coverage of the program. Thus, due to
obsolete and redundant test cases, the size of a test suite continues to grow unnecessarily.
12.2 MINIMIZING THE TEST SUITE AND ITS BENEFITS
A test suite can sometimes grow to an extent that it is nearly impossible to execute. In this
case, it becomes necessary to minimize the test cases such that they are executed for
maximum coverage of the software. Following are the reasons why minimization is
important: „
 Release date of the product is near. „
 Limited staff to execute all the test cases. „
 Limited test equipments or unavailability of testing tools.
When test suites are run repeatedly for every change in the program, it is of enormous
advantage to have as small a set of test cases as possible. Minimizing a test suite has the
following benefits:
 Sometimes, as the test suite grows, it can become prohibitively expensive to execute on
new versions of the program. These test suites will often contain test cases that are no
longer needed to satisfy the coverage criteria, because they are now obsolete or
redundant. Through minimization, these redundant test cases will be eliminated.
 The sizes of test sets have a direct bearing on the cost of the project. Test suite
minimization techniques lower costs by reducing a test suite to a minimal subset.
 Reducing the size of a test suite decreases both the overhead of maintaining the test
suite and the number of test cases that must be rerun after changes are made to the
software, thereby reducing the cost of regression testing.
Thus, it is of great practical advantage to reduce the size of test cases.
12.3 DEFINING TEST SUITE MINIMIZATION PROBLEM
Harrold et al. [51] have defined the problem of minimizing the test suite as given below.
Given A test suite TS; a set of test case requirements r1, r 2 ,….., r n that must be satisfied
to provide the desired testing coverage of the program; and subsets of TS, T1 , T2 , …. ,
Tn , one associated with each of the ri ’s such that any one of the test cases tj belonging to
Ti can be used to test ri .
Problem Find a representative set of test cases from TS that satisfies all the ri ’s.
The ri ’s can represent either all the test case requirements of a program or those
requirements related to program modifications. A representative set of test cases that
satisfies the ri ’s must contain at least one test case from each Ti . Such a set is called a
hitting set of the group of sets, T1, T2, ... Tn. Maximum reduction is achieved by finding
the smallest representative of test cases. However, this subset of the test suite is the
minimum cardinality hitting set of the Ti ’s and the problem of finding the minimum
cardinality hitting set is NP complete. Thus, minimization techniques resort to heuristics.
12.4 TEST SUITE PRIORITIZATION
The reduction process can be best understood if the cases in a test suite are prioritized in
some order. The purpose of prioritization is to reduce the set of test cases based on some
rational, non-arbitrary criteria, while aiming to select the most appropriate tests. For
example, the following priority categories can be determined for the test cases:
Priority 1 The test cases must be executed, otherwise there may be worse consequences
after the release of the product. For example, if the test cases for this category are not
executed, then critical bugs may appear.
Priority 2 The test cases may be executed, if time permits.
Priority 3 The test case is not important prior to the current release. It may be tested shortly
after the release of the current version of the software.
Priority 4 The test case is never important, as its impact is nearly negligible
In the prioritization scheme, the main guideline is to ensure that low-priority test cases do
not cause any severe impact on the software. There may be several goals of prioritization.
These goals can become the basis for prioritizing the test cases. Some of them are
discussed here:
 Testers or customers may want to get some critical features tested and presented in
the fi rst version of the software. Thus, the important features become the criteria
for prioritizing the test cases. But the consequences of not testing some low-priority
features must be checked. Therefore, risk factor should be analysed for every
feature in consideration. „
 Prioritization can be on the basis of the functionality advertised in the market. It
becomes important to test those functionalities on a priority basis, which the
company has promised to its customers. „
 The rate of fault detection of a test suite can reveal the likelihood of faults earlier. „
 Increase the coverage of coverable code in the system under test at a faster rate,
allowing a code coverage criterion to be met earlier in the test process. „
 Increase the rate at which high-risk faults are detected by a test suite, thus locating
such faults earlier in the testing process. „
 Increase the likelihood of revealing faults related to specific code changes, earlier
 in the regression testing process.

12.5 TYPES OF TEST CASE PRIORITIZATION

General Test Case Prioritization In this prioritization, we prioritize the test cases that will
be useful over a succession of subsequent modified versions of P, without any knowledge
of the modified versions. Thus, a general test case prioritization can be performed
following the release of a program version during off-peak hours, and the cost of
performing the prioritization is amortized over the subsequent releases.
Version-Specific Test Case Prioritization
Here, we prioritize the test cases such that they will be useful on a specific version P’ of
P. Version-specific prioritization is performed after a set of changes have been made to P
and prior to regression testing of P’, with the knowledge of the changes that have been
made.
12.6 PRIORITIZATION TECHNIQUES
Prioritization techniques schedule the execution of test cases in an order that attempts to
increase their effectiveness at meeting some performance goal. Therefore, given any
prioritization goal, various prioritization techniques may be applied to a test suite with the
aim of meeting that goal. For example, in an attempt to increase the rate of fault-detection
of test suites, we might prioritize test cases in terms of the extent to which they execute
modules that, measured historically, tend to fail. Alternatively, we might prioritize the
test cases in terms of their increasing cost-per-coverage of code components, or in terms
of their increasing cost-per-coverage of features listed in a requirement specifi cation. In
any case, the intent behind the choice of a prioritization technique is to increase the
likelihood that the prioritized test suite can better meet the goal than an ad hoc or random
ordering of test cases.
Prioritization can be done at two levels, as discussed below.
Prioritization for regression test suite
This category prioritizes the test suite of regression testing. Since regression testing is
performed whenever there is a change in the software, we need to identify the test cases
corresponding to modified and affected modules.
Prioritization for system test suite
This category prioritizes the test suite of system testing. Here, the consideration is not the
change in the modules. The test cases of system testing are prioritized based on several
criteria: risk analysis, user feedback, fault-detection rate, etc.
12.6.1 COVERAGE-BASED TEST CASE PRIORITIZATION
 This type of prioritization [58] is based on the coverage of codes, such as statement
coverage, branch coverage, etc. and the fault exposing capability of the test cases. Test
cases are ordered based on their coverage. For example, count the number of statements
covered by the test cases. The test case that covers the highest number of statements will
be executed first. Some of the techniques are discussed below.
Total Statement Coverage Prioritization
This prioritization orders the test cases based on the total number of statements covered.
It counts the number of statements covered by the test cases and orders them in a
descending order. If multiple test cases cover the same number of statements, then a
random order may be used. For example, if T1 covers 5 statements, T2 covers 3, and T3
covers 12 statements; then according to this prioritization, the order will be T3, T1, T2.
Additional Statement Coverage Prioritization
Total statement coverage prioritization schedules the test cases based on the total
statements covered. However, it will be useful if it can execute those statements as well
that have not been covered yet. Additional statement coverage prioritization iteratively
selects a test case T1, that yields the greatest statement coverage, then selects a test case
which covers a statement uncovered by T1. Repeat this process until all statements
covered by at least one test case have been covered.
For example, if we consider Table 12.1, according to total statement coverage criteria, the
order is 2,1,3. But additional statement coverage selects test case 2 fi rst and next, it
selects test case 3, as it covers statement 4 which has not been covered by test case 2.
Thus, the order according to addition coverage criteria is 2,3,1.

Total Branch Coverage Prioritization

In this prioritization, the criterion to order is to consider condition branches in a program
instead of statements. Thus, it is the coverage of each possible outcome of a condition in a
predicate. The test case which will cover maximum branch outcomes will be ordered fi rst. For
example, see Table 12.2. Here, the order will be 1, 2, 3.

Additional Branch Coverage Prioritization

Here, the idea is the same as in additional statement coverage of first selecting the test
case with the maximum coverage of branch outcomes and then, selecting the test case
which covers the branch outcome not covered by the previous one.
Total Fault-Exposing-Potential ( FEP) Prioritization
Statement and branch coverage prioritization ignore a fact about test cases and faults:
 Some bugs/faults are more easily uncovered than other faults. „
 Some test cases have the proficiency to uncover particular bugs as compared to
other test cases.
Thus, the ability of a test case to expose a fault is called the fault exposing potential. It
depends not only on whether test cases cover a faulty statement, but also on the
probability that a fault in that statement will also cause a failure for that test case.
To obtain an approximation of the FEP of a test case, an approach was adopted using
mutation analysis [137,138]. This approach is discussed below.
Given a program P and a test suite T,
 First, create a set of mutants N = {n1, n2,…..nm} for P, noting which
statement sj in P contains each mutant. „
 Next, for each test case ti T, execute each mutant version nk of P on ti , noting
whether ti kills that mutant. „
 Having collected this information for every test case and mutant, consider
each test case ti , and each statement sj in P, and calculate FEP(s,t) of ti on sj
as the ratio:
 Mutants of sj killed / Total number of mutants of sj

If ti does not execute sj , this ratio is zero.

To perform total FEP prioritization, given these (FEP)(s,t) values, calculate an award value for
each test case ti T, by summing the (FEP)(sj , ti ) values for all statements sj in P. Given these
award values, we prioritize the test cases by sorting them in order of descending award value.

Table 12.3 shows FEP estimates of a program. Total FEP prioritization outputs the test case
order as (2, 3, 1).

12.6.2 RISK-BASED PRIORITIZATION

Risk-based prioritization [22] is a well-defined process that prioritizes modules for testing. It uses
risk analysis to highlight potential problem areas, whose failures have adverse consequences.
The testers use this risk analysis to select the most crucial tests. Thus, risk-based technique is to
prioritize the test cases based on some potential problems which may occur during the project.

 Probability of occurrence/fault likelihood It indicates the probability of occurrence of a

problem.
 Severity of impact/ failure impact If the problem has occurred, how much impact does
it have on the software.

Risk analysis uses these two components by first listing the potential problems and then,
assigning a probability and severity value for each identified problem, as shown in Table 12.4. By
ranking the results in this table in the form of risk exposure, testers can identify the potential
 problems against which the software needs to be tested and executed first. For example, the
problems in the given table can be prioritized in the order of P5, P4, P2, P3, P1.

A risk analysis table consists of the following columns:

Problem ID A unique identifier to facilitate referring to a risk factor. „

Potential problem Brief description of the problem. „

Uncertainty factor It is the probability of occurrence of the problem. Probability values are on a
scale of 1 (low) to 10 (high). „

Severity of impact Severity values on a scale of 1 (low) to 10 (high). „

Risk exposure Product of probability of occurrence and severity of impact.

12.6.3 PRIORITIZATION BASED ON OPERATIONAL PROFILES

This is not a prioritization in true sense, but the system is developed in such a way that only useful test
cases are designed. So there is no question of prioritization. In this approach, the test planning is done
based on the operation profiles [128, 129] of the important functions which are of use to the customer.
An operational profile is a set of tasks performed by the system and their probabilities of occurrence.
After estimating the operational profiles, testers decide the total number of test cases, keeping in view
the costs and resource constraints.

12.6.4 PRIORITIZATION USING RELEVANT SLICES

During regression testing, the modified program is executed on all existing regression test cases to check
that it still works the same way as the original program, except where a change is expected. But re-
running the test suite for every change in the software makes regression testing a time-consuming
process. If we can find the portion of the software which has been affected with the change in software,
then we can prioritize the test cases based on this information. This is called the slicing technique
[130,131,132]. The various definitions related to this technique have been defined in literature. Let us
discuss them.

Execution Slice The set of statements executed under a test case is called the execution slice of the
program. For example, see the following program (Fig. 12.1):

T 1 and T 2 produce correct results. On the other hand, T 3 produces an incorrect result. Syntactically, it
is correct, but an employee with the empid ‘0’ will not get any salary, even if his basic salary is read as
input. So it has to be modified.
Suppose S 3 is modified as [if (basic>5000 && empid>0)]. It means for T 1, T 2, and T 3, the program
would be rerun to validate whether the change in S3 has introduced new bugs or not. But if there is a
change in S 7 statement [da = (basic*25)/100; instead of da = (basic*15)/100;], then only T 2 will be
rerun. So in the execution slice, we will have less number of statements. The execution slice is
highlighted in the given code segment.
Instead of computing an execution slice at the statement level, we may also compute it at the function
or the module level. That is, instead of determining which program statements are executed under each
test case, we may simply determine which functions or modules are invoked under each test case. Then,
we need to execute the new program on a test case only if a modified function or module was invoked
during the original program’s execution on that test case.
Dynamic Slice
The set of statements executed under a test case having an effect on the program output is called the
dynamic slice of the program with respect to the output variables. For example, see the following
program (Fig. 12.2).

Begin S1: read (a,b);

S2: sum=0;
S2.1: I=0;
S3: if (a==0)
{
S4: print(b);
S5: sum+=b;
}
S6: else if(b==0)
{
S7: print(a);
S8: sum+=a;
}
S9: else
{
S10: sum=a+b+sum;
S10.1 I=25;
S10.2 print(I);
}
S11:endif
S12:print(sum);
End
Figure 12.2 Example program for dynamic slice

T 1, T 2, and T 3 will run correctly but if some modification is done in S 10.1 [say I = 50], then this change
will not affect the output variable. Therefore, there is no need to rerun any of the test cases. On the
other hand, if S 10 is changed [say, sum=a*b+sum], then this change will affect the output variable
‘sum’, so there is a need to rerun T 3. The dynamic slice is highlighted in the code segment.
Begin
S1: read (a,b);
S2: sum = 0;
S2.1: I=0;
S3: if (a==0)
{
S4: print(b);
S5: sum+=b;
}
S6: elseif(b==0)
{
S7: print(a);
S8: sum+=a;
}
S9: else
{
S10: sum=a+b+sum;
S10.1 I=25;
S10.2 print(I);
}
S11: endif
S12: print(sum);
End
Relevant Slice

The set of statements that were executed under a test case and did not affect the output, but have the
potential to affect the output produced by a test case, is known as the relevant slice of the program. It
contains the dynamic slice and in addition, includes those statements which, if corrected, may modify
the variables at which the program failure has manifested. For example, consider the example of Fig.
12.2, statements S 3 and S 6 have the potential to affect the output, if modified.

If there is a change in any statement in the relevant slice, we need to rerun the modified software on
only those test cases whose relevant slices contain a modified statement. Thus, on the basis of relevant
slices, we can prioritize the test cases. This technique is helpful for prioritizing the regression test suite
which saves time and effort for regression testing.

Jeffrey and Gupta [139] enhanced the approach of relevant slicing and stated: ‘If a modification in the
program has to affect the output of a test case in the regression test suite, it must affect some
computation in the relevant slice of the output for that test case’. Thus, they applied the heuristic for
prioritizing test cases such that the test case with larger number of statements must get higher weight
and will get priority for execution.

12.6.5 PRIORITIZATION BASED ON REQUIREMENTS

This technique is used for prioritizing the system test cases. The system test cases also become too large
in number, as this testing is performed on many grounds. Since system test cases are largely dependent
on the requirements, the requirements can be analysed to prioritize the test cases.

This technique does not consider all the requirements on the same level. Some requirements are more
important as compared to others. Thus, the test cases corresponding to important and critical
requirements are given more weight as compared to others, and these test cases having more weight
are executed earlier.

Hema Srikanth et al. [136] have applied requirement engineering approach for prioritizing the system
test cases. It is known as PORT (prioritization of requirements for test). They have considered the
following four factors for analysing and measuring the criticality of requirements:

Customer-assigned priority of requirements Based on priority, the customer assigns a weight (on a
scale of 1 to 10) to each requirement. Higher the number, higher is the priority of the requirement.
Requirement volatility This is a rating based on the frequency of change of a requirement. The
requirement with a higher change frequency is assigned a higher weight as compared to the stable
requirements.

Developer-perceived implementation complexity All the requirements are not equal on a

implementation level. The developer gives more weight to a requirement which he thinks is more
difficult to implement.

Fault proneness of requirements This factor is identified based on the previous versions of system. If a
requirement in an earlier version of the system has more bugs, i.e. it is error-prone, then this
requirement in the current version is given more weight. This factor cannot be considered for a new
software. Based on these four factor values, a prioritization factor value (PFV) is computed as given
below. PFV is then used to produce a prioritized list of system test cases.

12.7 MEASURING THE EFFECTIVENESS OF A PRIORITIZED TEST SUITE

When a prioritized test suite is prepared, how will we check its effectiveness? We need one metric
which can tell us the effectiveness of one prioritized test suite. For this purpose, the rate of fault-
detection criterion can be taken. Elbaum et al. [133,134] developed APFD (average percentage of faults
detected) metric that measures the weighted average of the percentage of faults detected during the
execution of a test suite. Its value ranges from 0 to 100, where a higher value means a faster fault-
detection rate. Thus, APFD is a metric to detect how quickly a test suite identifies the faults. If we plot
percentage of test suite run on the x-axis and percentage of faults detected on the y-axis, then the area
under the curve is the APFD value, as shown in Fig. 12.3.

Example 12.1
All the bugs detected are not of the same level of severity. One bug may be more critical as compared to
others. Moreover, the cost of executing the test cases also differs. One test case may take more time as
compared to others. Thus, APFD does not consider the severity level of the bugs and the cost of
executing the test cases in a test suite. Elbaum et al. [135] modifi ed their APFD metric and considered
these two factors to form a new metric which is known as cost-cognizant APFD and denoted as APFDc.
In APFDc, the total cost incurred in all the test cases is represented on x-axis and the total fault severity
detected is taken on y-axis. Thus, it measures the unit of fault severity detected per unit test case cost.

SOFTWARE QUALITY METRICS

Software quality metrics are a subset of software metrics that focus on the quality aspects of the
product, process, and project. In general, software quality metrics are more closely associated with
process and product metrics than with project metrics. Nonetheless, the project parameters such as the
number of developers and their skill levels, the schedule, the size, and the organization structure
certainly affect the quality of product. Software quality metrics can be divided further into end-product
quality, and based on the findings, to engineer the improvements in both process and product quality.
Moreover, we should view quality from the entire software life cycle perspective and in this regard, we
should include metrics that measure the quality level of the maintenance process as another category of
software quality metrics. Thus, software quality metrics can be grouped into the following three
categories in accordance with the software lifecycle: (i) product quality metrics, in-process quality
metrics, and metrics for software maintenance.

Product Quality Metrics

In product quality, the following metrics are considered [42]:

Mean-time to failure ( MTTF) MTTF metric is an estimate of the average or mean time until a product’s
fi rst failure occurs. It is most often used with safety critical systems such as the airline traffi c control
system.

It is calculated by dividing the total operating time of the units tested by the total number of failures
encountered.

Defect density metrics It measures the defects relative to the software size.

Defect density = Number of defects / Size of product

Thus, it is the defect rate metrics or the volume of defects generally used for commercial software
systems. It is important for cost and resource estimate of the maintenance phase of the software
lifecycle.
Customer problem metrics This metric measures the problems which customers face while using the
product. The problems may be valid defects or usability problems, unclear documentation, etc. The
problem metrics is usually expressed in terms of problems per user month ( PUM), as given below:

PUM is usually calculated for each month after the software is released. Basically, this metric relates
with the problem of software usage. The goal is to achieve a low PUM by:

(a) Improving the development process and reducing the product defects.

(b) Reducing the non-defect-oriented problems by improving all the aspects of a product.

(c) Increasing the number of installed licenses of the product.

Customer satisfaction metrics Customer satisfaction is usually measured through various methods of
customer surveys via the five-point scale.

1. Very satisfied

2. Satisfied

3. Neutral

4. Dissatisfied

5. Very dissatisfied

Based on this scale, several metrics can be prepared like the percentage of completely satisfied
customers or the percentage of satisfied customers.

In-process Quality Metrics

In-process quality metrics are less formally defined than the end-product metrics. Following are some
metrics [42]:

Defect-density during testing

Higher defect rates found during testing is an indicator that the software has experienced higher error
injection during its development process. This metric is a good indicator of quality while the software is
still being tested. It is especially useful to monitor subsequent releases of a product in the same
development organization.
Defect-arrival pattern during testing The pattern of defect arrivals or the time between consecutive
failures gives more information. Different patterns of defect arrivals indicate different quality levels in
the field. The objective is to have defect arrivals that stabilize at a very low level or times between
failures that are far apart.

Metrics for Software Maintenance In this phase, the goal is to fi x the defects as soon as possible and
with excellent quality. The following metrics are important in software maintenance:

Fix backlog and backlog management index

Fix backlog metrics is the count of reported problems that remain open at the end of a month or a week.
This metric can provide meaningful information for managing the maintenance process.

Backlog management index ( BMI) is also the metric to manage the backlog of open unresolved
problems.

If BMI is larger than 100, it means the backlog is reduced; if BMI is less than 100, the backlog is
increased. Therefore, the goal is to have a BMI larger than 100.

Fix response time and fix responsiveness While fixing the problems, time-limit also matters according to
the severity of the problems. There may be some critical situations in which the customers feel the risk
due to defects in the software product; therefore high-severity defects require the response in terms of
fixing them as early as possible. The fi x response time metric is used for this purpose and is calculated
as mean-time of high severity defects from open to close. Thus, a short fix response time leads to
customer satisfaction.

This metric can be enhanced from the customers’ viewpoint. Sometimes, the customer sets a time-limit
to fix the defects. This agreed-to fix time is known as fi x responsiveness metric. The goal is to get high
fix responsiveness to meet customer expectations and have highly satisfied customers.

Percent delinquent fixes for each fix, if the turn-around time greatly exceeds the required response
time, then it is classified as delinquent.
Fix quality It is the metric to measure the number of fixes that turn out to be defective. A fix is defective
if it did not fix the reported problems or injected a new defect as a consequence of fixing. For real-time
software, this metric is important for customer satisfaction. Fix quality is measured as a percent
defective fixes. Percent defective fixes is the percentage of all fixes in a time interval that are defective.

SQA MODELS

Organizations must adopt a model for standardizing their SQA activities. Many SQA models have been
developed in recent years. Some of them are discussed below.

13.11.1 ISO 9126

This is an international software quality standard from ISO. It aims to eliminate any misunderstanding
between customers and developers. There may be a case that the customer is not able to communicate
the requirements for the product to be developed. Moreover, the developer should also be able to
understand the requirement and assess with confidence whether it is possible to provide the product
with the right level of software quality. Thus, there may be problems with both customers and
developers which may affect the delivery and quality of the software project. ISO 9126 targets the
common understanding between customers and developers.

ISO 9126 provides a set of six quality characteristics with their definitions and associated quality
evaluation process to be used when specifying the requirements, and evaluating the quality of software
products throughout their life cycle. The quality characteristics are shown in Fig. 13.6. The quality
characteristics are accompanied by guidelines for their use. Each attribute is associated with one or
more metrics, which allow a value for that attribute to be determined for a particular system.

ISO 9126 defines the following guidelines for implementation of this model:
Quality requirements definition A quality requirement specification is prepared with implied needs,
technical documentation of the project, and the ISO standard.

Evaluation preparation This stage involves the selection of appropriate metrics, a rating-level definition,
and the definition of assessment criteria. Metrics are quantifiable measures mapped onto scales. The
rating levels definition determines what ranges of values on these scales count as satisfactory or
unsatisfactory. The assessment criteria definition takes as input a management criterion specific to the
project and prepares a procedure for summarizing the results of the evaluation.

Evaluation procedure The selected metrics are applied to the software product and values are taken on
the scales. Subsequently, for each measured value, the rating level is determined. After this, a set of
rated levels are summarized, resulting in a summary of the software product quality. This summary is
then compared with other aspects such as time and cost, and the final managerial decision is taken,
based on managerial criteria.

13.11.2 CAPABILITY MATURITY MODEL (CMM)

It is a framework meant for software development processes. This model is aimed to measure the
quality of a process by recognizing it on a maturity scale. In this way, matured organizations can be
distinguished from immature ones. The maturity of a process directly affects the progress of a project
and its results. Therefore, for a high-quality software product, the process adopted for development
must be a matured one.

The development of CMM started in 1986 by the Software Engineering Institute (SEI) with MITRE
Corporation. They developed a process maturity framework for improving software processes in
organizations. After working for four years on this framework, the SEI developed the CMM for software.
Later on, it was revised and is continuing to evolve.

CMM Structure

The structure of CMM consists of five maturity levels. These levels consist of key process areas (KPAs)
which are organized by common features. These features in turn consist of key practices, decided in
KPAs depending on these common features, as shown in Fig. 13.7. The maturity levels indicate the
process capability. Key practices describe the activities to be done. Common features address
implementations. In this way, by working on the activities described in key practices under some
common feature, we achieve a goal under a KPA. In other words, each KPA identifies a cluster of related
activities that, when performed collectively, achieve a set of goals considered important for enhancing
the process capability.
The structure of CMM is hierarchical in nature such that if one organization wishes to be on maturity
level 3, then it must first achieve maturity level 2, and then work according to the goals of the next
higher level. It means that we need to attain maturity in a sequenced order of all levels, as shown in Fig.
13.8. The process capability at each maturity level is shown in Table 13.1.

Maturity Levels

A brief description of the five maturity levels is given below.

Initial At this level, the process is chaotic or ad-hoc, where there is no control on development progress
and success is purely dependent on individual efforts. There is no stable environment for development.
The team members may not know the components or goals of development processes. The projects
exceed the budget and schedule.

Repeatable Earlier project successes are used here and the lessons learned from past projects are
incorporated in the current project. Basic project management processes are established so that cost,
schedule, and other parameters can be tracked. However, organization-wide processes still do not exist.

Defined The management scope is widened to the organization here. Project management as well as
process management starts. The standard processes are defined with documentations.

Managed A quantitative understanding of the processes is established in order to monitor and control
them. Quantitative quality goals are also set for the processes. It means the processes are predictable
and controlled.

Optimizing The process is continually improved through incremental and innovative technological
changes or improvements. In this way, quantitative process improvement objectives for the organization
is identified, evaluated, and deployed.

Key Process Areas

The KPAs and their corresponding goals at each maturity level are discussed in Tables 13.2, 13.3, 13.4,
and 13.5.
Common Features

As discussed above, all KPAs are organized by common features. The common features are attributes
that indicate what to implement according to a KPA goal. The common features have been listed in
Table 13.6

Assessment of Process Maturity

The assessment is done by an assessment team consisting of some experienced team members. These
members start the process of assessment by collecting information about the process being assessed.
The information about the process can be collected in the form of maturity questionnaires,
documentation, or interviews. For assessment, some representative projects of the organization are
selected.

13.11.3 SOFTWARE TOTAL QUALITY MANAGEMENT ( STQM)

Total quality management ( TQM) is a term that was originally coined in 1985 by the Naval Air Systems
Command to describe its Japanese-style management approach to quality improvement. TQM is a
philosophical management strategy to have quality culture in the entire organization. TQM means that
the organization’s culture is defined by the constant attainment of satisfaction through an integrated
system of tools, techniques, and training. Thus, TQM is defined as a quality-centered, customer-focused,
fact-based, team-driven, senior-management-led process to achieve an organization’s strategic
imperative through continuous process improvement

T = Total = everyone in the organization

Q = Quality = customer satisfaction

M = Management = people and processes

Total quality here means to provide quality product and quality services to the customer, i.e. quality in
all aspects. TQM is based on the participation of all the members of an organization to improve
processes, products, services, and the culture they work in. TQM is a customer-driven approach,
wherein the quality is defined and judged by the customer. It focuses on continuous process
improvement to achieve high quality of product (or service). Moreover, the emphasis is to achieve total
quality throughout the entire business, not just in the product. Thus, quality is a part of every function in
all the phases of a product’s lifecycle.

The elements of a TQM system can be summarized as follows:

1. Customer focus The goal is to attain total customer satisfaction. It includes studying customers’
needs, gathering customer requirements, and measuring and managing customer satisfaction.

2. Process The goal is to achieve continuous process improvement. Process includes both the business
process and the product development process. Through process improvement, product quality will be
enhanced.

3. Human-side of quality The goal is to create a company-wide quality culture. Focus areas include
management commitment, total participation, employee empowerment, and other social,
psychological, and human factors.

4. Measurement and analysis The goal is to drive continuous improvement in all quality parameters by
the goal-oriented measurements

TQM has a few short-term advantages. Most of its benefits are long-term and come into effect only after
running smoothly for some time. In large organizations, it may take several years before long-term
benefits are realized. Long-term benefits that may be expected from TQM are: higher productivity,
increased morale, reduced costs, and greater customer commitment.

TQM originated in the manufacturing sector, but it has been successfully adopted by almost every type
of organization. So it can also be applied to software development because we need to improve our
software processes. Remember, we can’t test quality in our software, we design in it. And the only way
we can design quality in it is by continuously monitoring and improving our processes. As we studied in
the previous sections, the software quality is maintained by the SQA team. However, SQA is not enough
to achieve the quality standards demanded by the customer. It becomes important to adopt the TQM
method to the entire software development organization. This is known as software total quality
management (STQM). The STQM is also a fundamental cultural change that incorporates quality
improvements in every aspect of the software organization. Therefore, SQA provides the methodology
to assure quality, while STQM provides the framework to continually improve it. STQM fits well in the
components of TQM discussed above.

Customer focus in software development

As we know, requirement bugs constitute a large portion of the software bugs. Therefore, gathering
customers’ needs has become increasingly important in the software industry.
Process, technology, and development quality

Given the customers’ requirements, the central question is how to develop the software effectively so
that it can meet the criterion of ‘conformance to customers’ requirements’. For this, a matured process
and updated technologies are needed to achieve product quality.

Human-side of software quality

In software development, at the operational level, TQM can be viewed as the integration of project,
process, and quality management with the support of every development team member. During
software development, managers must manage in-process quality in the way they manage schedule for
quality improvement. Quality, process, and schedule management must be totally integrated for
software development to be effective.

Measurement and analysis

Software development must be controlled in order to have product delivery on time and within budget,
which are the prime concerns for every software development. Software metrics help in measuring
various factors during development and thereby, in monitoring and controlling them.

13.11.4 SIX SIGMA

Six sigma is a quality model originally developed for manufacturing processes. It was developed by
Motorola. Six sigma derives its meaning from the field of statistics. Sigma is the standard deviation for a
statistical population. Six sigma means, if one has six standard deviations between the mean of a
process and the nearest specification limit, there will be practically no items that fail to meet the
specifications. Therefore, the goal of this model is to have a process quality of that level. Eventually, six
sigma evolved and was applied to other non-manufacturing processes. Today, you can apply six sigma to
many fields like services, medical, and insurance procedures, call centers including software.

Six sigma is a way to achieve strategic business results emphasizing on lower costs with less number of
defects. Six sigma processes will produce less than 3.4 defects per million opportunities. To achieve this
target, it uses a methodology known as DMAIC with the following steps:

 „ Define opportunities „
 Measure performance „
 Analyse opportunity „
 Improve performance „
 Control performance

This methodology improves any existing business process by constantly reviewing and improving the
process.

Six sigma is not just all about statistics. It can be applied to software problems which affects its quality.
Six sigma can be applied to analyse the customer requirements and defi ne business goals
correspondingly. Its approach to customer requirements, if applied to software development projects, is
fundamentally different than those typically practiced in software deployment efforts. It does not start
by asking the customers about the requirements first. But it begins by analysing what we need to learn.
There are six sigma tools which help in identifying the prioritization of functionalities to be delivered.

Debugging

Debugging is not a part of the testing domain. Therefore, debugging is not testing. It is a separate
process performed as a consequence of testing. But the testing process is considered a waste if
debugging is not performed after the testing. Testing phase in the SDLC aims to find more and more
bugs, remove the errors, and build confidence in the software quality. In this sense, testing phase can be
divided into two parts:

1. Preparation of test cases, executing them, and observing the output. This is known as testing.

2. If output of testing is not successful, then a failure has occurred. Now the goal is to find the bugs that
caused the failure and remove the errors present in the software. This is called debugging.

Debugging is the process of identification of the symptoms of failures, tracing the bug, locating the
errors that caused the bug, and correcting these errors. Describing it in more concrete terms, debugging
is a two-part process. It begins with some indication of the existence of an error. It is the activity of [2]:

1. Determining the exact nature of the bug and location of the suspected error within the program.

2. Fixing or repairing the error.

18.2 DEBUGGING PROCESS

As discussed, the goal of debugging process is to determine the exact nature of failure with the help of
symptoms identified, locate the bugs and errors, and finally correct it. The debugging process (see Fig.
18.1) is explained in the following steps:
 Check the result of the output produced by executing test cases prepared in the testing process.
If the actual output matches with the expected output, it means that the results are successful.
Otherwise, there is failure in the output which needs to be analysed. „
 Debugging is performed for the analysis of failure that has occurred, where we identify the
cause of the problem and correct it. It may be possible that symptoms associated with the
present failure are not sufficient to find the bug. Therefore, some additional testing is required
so that we may get more clues to analyse the causes of failures.
 If symptoms are sufficient to provide clues about the bug, then the cause of failure is identified.
The bug is traced to find the actual location of the error. „
 Once we find the actual location of the error, the bug is removed with corrections. „
 Regression testing is performed as bug has been removed with corrections in the software.
Thus, to validate the corrections, regression testing is necessary after every modification.

DEBUGGING TECHNIQUES

18.4.1 DEBUGGING WITH MEMORY DUMP

In this technique, a printout of all registers and relevant memory locations is obtained and studied.
The storage locations are in octal or hexadecimal format. The relevant data of the program is
observed through these memory locations and registers for any bug in the program. However, this
method is inefficient. This method should be used as the last option. Following are some drawbacks
of this method [2]:

1. There is difficulty of establishing the correspondence between storage locations and the variables
in one’s source program.

2. The massive amount of data with which one is faced, most of which is irrelevant.

3. It is limited to static state of the program as it shows the state of the program at only one instant
of time.

18.4.2 DEBUGGING WITH WATCH POINTS

The program’s final output failure may not give sufficient clues about the bug. It is possible that
intermediate execution, at some points in the program, may provide sufficient cause of the problem.
At a particular point of execution in the program, value of variables or other actions can be verified.
These particular points of execution are known as watch points. Debugging with watch points can be
implemented with the following methods:

Output statements In this method, output statements can be used to check the state of a condition
or a variable at some watch point in the program. Therefore, output statements are inserted at
various watch points; program is complied and executed with these output statements. Execution of
output statements may give some clues to find the bug. This method displays the dynamics of a
program and allows one to examine information related to the program failure. This method has the
following drawbacks:
  It may require many changes in the code. These changes may mask an error or introduce
new errors in the program.
 After analysing the bug, we may forget to remove these added statements which may cause
other failures or misinterpretations in the result.

Breakpoint execution This is the advanced form of the watch point used with an automated
debugger program. Breakpoint is actually a watch point inserted at various places in the
program. But these insertions are not placed in the actual user program and therefore need not
be removed manually like output statements.

The other difference in this method is that program is executed up to the breakpoint inserted.
At that point, you can examine whatever is desired. Afterwards, the program will resume and
will be executed further for the next breakpoint. Thus, breakpoints allow the programmer to
control execution of the program and specify how and where the application will stop to allow
further examination. Since there can be many breakpoints in the same program, all breakpoints
can also be viewed by the programmer. Breakpoints displayed in the view show the source
location of each breakpoint as well as its status.

Breakpoints have an obvious advantage over output statements. Some are discussed here:

(a) There is no need to compile the program after inserting breakpoints, while this is necessary
after inserting output statements.

(b) Removing the breakpoints after their requirement is easy as compared to removing all
inserted output statements in the program.

(c) The status of program variable or a particular condition can be seen after the execution of a
breakpoint as the execution temporarily stops after the breakpoint. On the other hand in case of
output statements, the full program is executed and output is viewed after the total execution
of the program.

Breakpoints can be categorized as follows:

(i) Unconditional breakpoint: It is a simple breakpoint without any condition to be evaluated. It is

simply inserted at a watch point and its execution stops the execution of the program.
(ii) Conditional breakpoint: On the activation of this breakpoint, one expression is evaluated for its
Boolean value. If true, the breakpoint will cause a stop; otherwise, execution will continue.
(iii) Temporary breakpoint: This breakpoint is used only once in the program. When it is set, the
program starts running, and once it stops, the temporary breakpoint is removed. The temporary
nature is one of its attributes. In other respects, it is just like other breakpoints.
(iv) Internal breakpoint: These are invisible to the user but are key to debugger’s correct handling of
its algorithms. These are the breakpoints set by the debugger itself for its own purposes.
Single stepping The idea of single stepping is that the users should be able to watch the execution of the
program after every executable instruction. After every instruction execution, the users can watch the
condition or status of variable. Single stepping is implemented with the help of internal breakpoints.
Step-into It means execution proceeds into any function in the current source statement and stops at
the first executable source line in that function.

Step-over It is also called skip, instead of step. It treats a call to a function as an atomic operation and
proceeds past any function calls to the textually succeeding source line in the current scope.

18.4.3 BACKTRACKING

This is a logical approach for debugging a program. The following are the steps for backtracking process
(see Fig. 18.2):

(a) Observe the symptom of the failure at the output side and reach the site where the symptom can be
uncovered. For example, suppose you are not getting display of a value on the output. After examining
the symptom of this problem, you uncover that one module is not sending the proper message to
another module. It may also be the case that function X is not working. This can be uncovered using the
above mentioned techniques, e.g. using breakpoints.

(b) Once you have reached the site where symptom has been uncovered, trace the source code starting
backwards and move to the highest level of abstraction of design. The bug will be found in this path. For
tracing backwards, collect source code of all the modules which are related at this point.

(c) Slowly isolate the module through logical backtracking using data fl ow diagrams ( DFDs) of the
modules wherein the bug resides.

(d) Logical backtracking in the isolated module will lead to the actual bug and error can thus be
removed.

Backtracking requires that the person debugging must have knowledge regarding the design of the
system so that he can understand the logical flow of the program through DFDs. This method also
requires the manual observation of the source code. This method is very effective as compared to the
other methods in pinpointing the error location quickly, if you understand the logic of the program.
18.5 CORRECTING THE BUGS

The second phase of the debugging process is to correct the error when it has been uncovered. But it is
not as easy as it seems. The design of the software should not be affected by correcting the bug or due
to new modifications. Before correcting the errors, we should concentrate on the following points:

(a) Evaluate the coupling of the logic and data structure where corrections are to be made. Highly
coupled module correction can introduce many other bugs. That is why low-coupled module is easy to
debug.

(b) After recognizing the influence of corrections on other modules or parts, plan the regression test
cases to perform regression testing as discussed earlier.

(c) Perform regression testing with every correction in the software to ensure that the corrections have
not introduced bugs in other parts of the software.

18.5.1 DEBUGGING GUIDELINES

Fresh thinking leads to good debugging Don’t spend continuous hours and hours on debugging a
problem. The continuous involvement in the problem will collapse your effi ciency. So rest in between
and start with a fresh mind to concentrate on the problem.

Don’t isolate the bug from your colleagues

It has been observed that bugs cannot be solved in isolation. Show the problem to other colleagues,
explain to them what is happening, and discuss the solution. There will be a lot of clues that have not
been tried yet. It has also been observed that by just discussing the problem with others may suddenly
lead you to come up with some solution. Therefore, it is always better to share the problem to others.
Don’t attempt code modifications in the first attempt Debugging always starts with the analysis of
clues. If you don’t analyse the failures and clues and simply change the code randomly with the view
—‘let’s see what happens with this change’, this thinking will lead you nowhere but with junk code
added to the software.

Additional test cases are a must if you don’t get the symptom or clues to solve the problem

Design test cases with the view to execute those parts of the program which are causing the problem.
This test case execution will provide symptoms to be analysed.

Regression testing is a must after debugging Don’t simply fi x the error and forget about it. The process
of debugging does not end with fixing the error. After fixing the error, you need to realize the effects of
this change on other parts. Test cases must be designed to uncover any effects on bugs produced on
other parts. That is why regression testing is necessary after fixing the errors.

Design should be referred before fixing the error Any change in the code to fi x the bug should be
according to pre-specified design of the software. While making corrections, that design should not be
violated by any means.