Chapter - 12

Software Quality and Reliability

 Introduction

• Once software is tested, it is assumed that it is defect free and

it will perform according to the needs of the customer.
• As a software product is to be used for a long time, it is
important to measure its quality for better reliability and
durability.
• Measuring the reliability of software products has been a
major issue for the developer and the customer.
• A good quality product satisfies the customer needs, is
constructed as per the standards and norms, has sound
internal design, and is developed within an optimized cost and
schedule.
• The internal design of software is much more important than
the external design. 2
 Software Quality Concept

• The quality of a software product is defined in terms of its

characteristics or attributes.
• The values of attributes vary from product to product.
• A product can be of good quality or bad quality. Weaker values
of attributes define a bad-quality product whereas higher
values of attributes define a good-quality product.
• The aim of software development organizations is to produce a
high-quality product.
• The external quality of a product does not always ensure the
internal quality.
• It is analogous to the quality of a building.

3
 Software Quality Concept

• A good quality product aims to satisfy the customer requirements,

developed under the guidelines of software quality standards.
• Cost and schedule also play important roles in defining the quality of
software.
• The process of development can affect the product quality.
• A standard process provides a systematic approach to development and
leads to a high-quality product.
• Quality of software can be defined in terms of following points.
 satisfies customer requirements
 possesses higher values of its characteristics
 has sound internal design along with external design
 is developed within the budget and cost
 follows the development standards
4
 Software Quality Factors

• McCall, Richards, and Walters have proposed certain factors

that affect the quality of software:
• These factors are classified into the following categories:
– Product operational factors: Correctness, reliability,
usability, integrity, and efficiency
– Product revision factors: Maintainability, flexibility, and
testability
– Product adaptability factors: Portability, reusability, and
interoperability

5
 Classification of Software quality factors

Software quality factors

Product operation Product revision Product adoptability

Maintainability Flexibility Testability Portability Reusability Interoperability

Correctness Reliability Usability Integrity Efficiency

Figure 12.1: Software quality factors

6
 Software quality factors
 Correctness: A program is correct if it performs according to
the specifications of functions it should provide.
 Reliability is the extent to which a program performs its
intended functions satisfactorily with required precision
without failure in a specified duration.
 Usability is the extent of effort required to learn, operate, and
use a product.
 Integrity is the extent of effort to control illegal access to data
and program by unauthorized people.
 Efficiency is the volume of computing resourses (e.g.,
processor time, memory space, bandwidth in communication
devices, etc.) and code required to perform software
functions.
7
 Software quality factors
 Maintainability is the ease to locate and correct errors.
Maintainability of a software is measured through mean time to
change.
 Flexibility is the cost required to modify an operational program.
 Testability is the effort required to test a program to ensure that it
performs its intended function.
 Portability is the effort required for transferring software products
to various hardware and software environments.
 Reusability is the extent to which software or its parts can be reused
in the development of some other software.
 Interoperability is the effort required to couple one system to
another. Strong coupling and loose coupling are the approaches used
in interoperability.

8
 Verification and Validation (V&V)

• The software product being developed must be checked during

development and post-development phases.
• Verification and Validation (V&V) is the process of checking a
software product to ensure that it meets the specifications and
satisfies the intended purpose.
• The V&V activities begin with requirements specification
review and continue to design reviews to code inspection
during testing.
• Verification is the process of evaluating work products in
software development phases to assess whether the work
product meets the specifications as intended for the purpose.

9
 Verification and Validation (V&V)

• IEEE defines, “verification is the act of reviewing, inspecting,

testing, checking, auditing, or otherwise establishing and
documenting whether product, process, service, or documents
conform to the specified requirements.”
• Validation is the process of evaluating software during or at
the end of the development process to determine whether it
satisfies the specified stated requirements.
• IEEE defines, “validation in the process of evaluating software
at the end of the software development process to determine
compliance with the requirement.” Validation is the end-to-end
verification.

10
 Verification and Validation (V&V)

• Berry Boehm defines and differentiates verification and

validation as:
 Verification: “Are we building the product right?”
 Validation: “Are we building the right product?”
• Verification and validation ensure that the software performs
no unintended functions and provides information about its
quality and reliability .
• The ultimate goal of verification and validation is to establish
confidence that the software system is “fit-for purpose.”

11
 The Cost of Quality
• The cost of quality is the cost related to achieving quality and
performing quality-related activities.
• In 1979, Philip Crosby stated that quality is free.
• He demonstrated that the cost of producing high-quality
products does not take extra cost than producing low-cost
products.
• The cost of quality is divided into the cost of conformance and
the cost of nonconformance.
• The cost of conformance is the cost incurred before the
delivery of a product, i.e., in identifying bugs, locating, and
correcting bugs, etc.
• The cost of nonconformance comes after the product is
released.
12
 The Cost of Quality

• The cost of quality is categorized into three classes: prevention

cost, appraisal cost, and failure cost
 Prevention cost is the cost incurred before performing quality
assurance activities. That is, it is the cost spent on quality
planning, formal technical reviews, test equipments, and
training.
 Appraisal cost is the cost incurred in performing quality
assurance activities, viz., inspection, testing, equipment
calibration and maintenance, etc.
 Failure cost is the cost of conformance and the cost of
nonconformance.

13
 The Cost of Quality

120
100
80
Cost ($)

60
40
20
0
Analysis Design Coding Tetsing Release &
operation
Software life cycle phases

Figure 12.2: Increase in the cost of quality

14
 Software Quality Assurance (SQA)

• “assurance” means “guarantee”. So, software quality assurance

(SQA) guarantees high quality of software.
• IEEE defines quality assurance as a planned and systematic
pattern of all actions necessary to produce adequate
confidence that the item or product conforms to established
technical requirements .
• Quality assurance activities are different from development
and maintenance activities.
• Software quality assurance is a preventive approach of quality.
• Its goal is to provide and implement activities within the
quality system that assist in getting confidence to ensure that
the work product will fulfill the requirements of quality.

15
 SQA Activities

• The quality assurance activities are performed at two levels,

namely, the activities performed by the software development
team and the activities performed by the quality assurance
team.
• The software development team focuses on applying technical
methods and tools in achieving product quality.
• Following methods are performed by the development team to
ensure product quality:
 Applying technical methods and tools
 Conducting verification and validation
 Performing measurements
 Performing testing

16
 SQA Activities

• The SQA team performs the following activities:

 Prepares SQA plan
 Participates in process execution for development
 Reviews software engineering activities to verify
compliance with the defined software process
 Software auditing
 Record keeping
 Reporting

17
 The ISO Quality Standard
• International Organization for Standardization (ISO) is a nonprofit
and worldwide federation of national standards bodies from several
countries.
• ISO provides state of the art specifications for products, services,
and good practices, helping to make industry more efficient and
effective.
• A quality management system provides a framework that is used to
monitor and improve performance, improve customer satisfaction,
staff motivation, and continual improvement.
• ISO 9000 is applicable to various industries other than the software
industry.
• ISO has several versions and it has gone through several revisions.
• ISO 9001:2008 is the latest ISO standard which is most widely used
by the software companies.
18
 ISO 9000 Standard

• It is based on eight quality management principles that senior

management applies for organizational improvement.
 Customer focus: As organizations depend on their customers,
they should understand current and future customers.
 Leadership: Leadership establishes unity of purpose and
direction of the organization. Organizations should create and
maintain an internal environment in which people can become
fully involved in achieving the organization's objectives.
 Involvement of people: People at all levels are the assets of an
organization and their full involvement enables their abilities
to be used for the organization's benefits.
 Process approach: The desired result is achieved more
efficiently when activities and related resources are managed
as a process.

19
 ISO 9000 Standard
 System approach to management: Identifying, understanding,
and managing interrelated processes as a system contributes to
the organization's effectiveness and efficiency in achieving its
objectives.
 Continual improvement: Continual improvement of the
organization's overall performance should be a permanent
objective of the organization.
 Factual approach to decision making: Effective decisions are
based on the analysis of data and information.
 Mutually beneficial supplier relationships: An organization
and its suppliers are interdependent and a mutually beneficial
relationship enhances the ability of both to create value.

20
 Process-Based Quality Management System Approach

Continual improvement of the quality management system

Customer and other Customer and

parties Management other parties
responsibility

Measurement,
Resource analysis and Satisfaction
management improvement

Product
Requirements Product
realization
Input Output

Value adding activities Information flow

Figure 12.5: Process-based quality management system approach

21
 ISO 9000 Certification Process
• Organizations willing to achieve ISO certification go through
the following ISO 9000 certification steps and apply to the
ISO 9000 registrar.
– Proposal stage
– Pre-assessment stage
– Document review and adequacy audit
– Compliance audit
– Registration
– Continued surveillance

22
 Advantages and Disadvantage of ISO 9000

• The ISO 9000 standards improve operating procedure and

reduce cost.
• When companies advertise the fact that they are ISO certified,
it proves to be a very powerful marketing tool for them.
• Also, ISO 9000 helps companies to have a competitive
advantage, especially in the ever-changing software market.
• The ISO international standards ensure that products and
services are safe, reliable, and have good quality.

23
 Advantages and Disadvantage of ISO 9000

 Sometimes, changes in the operation of an organization are

very expensive. The designated people have to provide
training for ISO 9000.
 ISO provides the standard for quality assurance but it does not
guarantee the process to be of high quality.
 There are times when companies feel that the existing set of
operational procedures is already working well and they do not
feel a change is necessary.
 The institutionalization of ISO standard might prejudice
would-be customers against non-ISO-compliant service
providers who might have something of value or of necessity
to offer.
 It is not cheap to become an ISO 9000 certified company.

24
 The Capability Maturity Model (CMM)

• The Capability Maturity Model (CMM) is an industry standard

model for defining and measuring the maturity of the
development process and for providing strategy for improving
the software processes toward achieving high-quality products.
• It was established by Software Engineering Institute (SEI) in
1986 at Carnegie Mellon University (CMU) at California,
U.S.A, under the direction of the U.S. Department of Defense.
• The CMM is involved in the process management process to
improve the software process whereas life cycle models are
used for the software development process.
• The SEI-CMM is a reference model for appraising the
maturity of the software process at different levels.

25
The Capability Maturity Model(CMM)
5 Focus on continuous
Optimizing improvement

4 Processes are measured and

Managed controlled

3 Process standardization for both

Defined management and development

2 Focus on project
Repeatable management process

1
Chaotic, ad hoc
Initial

Figure 12.6: The Capability Maturity Model (CMM)

26
 The Capability Maturity Model(CMM)

• The CMM provides a way to develop and refine an

organization's processes.
• A maturity model can be used as a benchmark for assessing
different organizations for equivalent comparison.
• It describes the maturity of the company in the above stated
levels based upon the project the company is dealing with and
the clients.
• Within each of these maturity levels, there are defined key
process areas (KPAs) which characterize that level.
• An organization willing to achieve a level has to demonstrate
all the KPAs in the corresponding level of CMM.

27
 CMM level Focus KPAs
KPAs
1. Initial Competent people and heroics Not applicable

2. Repeatable Disciplined process Requirement management

Project planning and tracking
Subcontractor management
Software quality assurance
Configuration management
3. Defined Process standardization Organization process focus
Organization process definition
Training program
Software product engineering
Integrated software development
Inter-group coordination
Peer reviews
4. Managed Measurable and controlled Quantitative process management
processes for quality Quality management

5. Optimized Continuous process improvement Defect prevention

Technology change management
Process change management
28
 SEI-CMM Vs. ISO Standard
ISO Standard CMM
1. ISO focuses on the customer-supplier relationship, 1. The CMM focuses on the software supplier to
attempting to reduce the customer's risk in choosing improve the internal processes of the software to
a supplier. achieve a higher quality product for the benefit of the
customer.

2. The ISO 9000 standard is intentionally written for 2. The CMM framework is developed for the
a wide range of industries other than the software software industry.
industry. It has a broader scope like hardware,
software, processed materials, and services.

3. ISO 9000 emphasizes following a set of standards 3. The CMM emphasizes the process of continuous
to make success repeatable. It addresses the improvement.
minimum criteria for an acceptable quality system.

4. Once an organization has met the criteria to be 4. The CMM is an ongoing process of evaluation and
ISO certified through an independent audit, the next improvement, moving from lower level to a higher
step is just to maintain that level of certification. level. Even at the highest level of maturity in CMM,
the focus is on continuous improvement.

29
 SEI-CMM Vs. ISO Standard
5. The ISO standard for peer review states 5. The CMM also states this but identifies the
that the items should be present at the time of purpose and focuses on how this activity will
reviews. benefit the organization.

6. ISO is awarded by an international body. 6. The SEI-CMM assessment is purely for

internal use.

7. ISO 9001 requires documentation that 7. The CMM shares this emphasis on processes
contains instructions or guidance on what that are documented and practiced as
should be done or how it should be done. documented. These are carried out according to
the KPAs in CMM.

8. It mostly encourages product design and 8. It encourages software product engineering.

development.

9. It requires yearly recertification. 9. There is no need for a yearly recertification

process.

10. ISO 9001 appears to use the more traditional 10. The CMM levels, specifically levels 4 and 5,
waterfall model of lifecycle development indicate a modified lifecycle model known as
IDEAL. IDEAL stands for initiating,
diagnosing, establishing, acting, and learning.
30
 Six Sigma

• Six Sigma was developed by Bill Smith, a senior quality

assurance manager at Motorola and initially it was
implemented at Motorola in 1987.
• General Electric (GE) Corporation started Six Sigma in 1995
and later it was appreciated by several companies worldwide.
• Six Sigma is a disciplined process that helps us focus on
developing and delivering near-perfect products and services.
• Sigma is a measurement that indicates how a process is
performing.
• Six Sigma stands for six standard deviations from mean.

31
 Key elements of Six Sigma

• It has following three key elements :

 Customer: Delighting a customer is crucial because customers
define quality. They expect product quality, competitive prices,
on-time delivery, services, etc.
 Processes: Defining processes and metrics and measures for
processes is the key element of Six Sigma. All these things are
thought from the customer's perspective and help identify the
weak areas within a process, which can be improved.
 Employees: Ultimately it is the responsibility of employees to
perform Six Sigma. Therefore, a company must provide
opportunities and incentives for employees to work toward
customer satisfaction.

32
 Key concepts of Six Sigma

• The key concepts of Six Sigma are as follows:

 Critical to quality: Attributes most important to the customer.
 Defect: Failing to deliver what the customer wants.
 Process capability: What your process can deliver.
 Variation: What the customer sees and feels.
 Stable operations: Ensuring consistent, predictable processes
to improve.
 Design for Six Sigma: Designing to meet the customer needs
and process capability.

33
 Six Sigma Methodologies

• Six Sigma implements two important methodologies, viz.,

DMAIC and DMADV.
• The Six Sigma DMAIC process (define, measure, analyze,
improve, and control) is an improvement system for existing
processes falling below specification and looking for
incremental improvement.
• The Six Sigma DMADV process (define, measure, analyze,
design, and verify) is an improvement system used to develop
new processes or products at Six Sigma quality levels.
• Both Six Sigma processes are executed by Six Sigma Green
Belts and Six Sigma Black Belts. Both belts are overseen by
Six Sigma Master Black Belts.

34
 Best Practices of Software Engineering

• Develop software iteratively.

• Effectively manage requirements.
• Use component-based architectures.
• Model visually to manage complexity.
• Verify software quality continuously throughout life cycle.
• Manage changes to software.

35
 Software Reliability

• A system is said to be reliable if it works correctly at all times

without failures.
• IEEE defines: “Software reliability is the probability of
failure-free operation of software over a given time interval
and under given conditions.”
• Reliability of software depends on the presence or absence of
defects in the system.
• As the system consists of hardware and software, its reliability
depends on the reliability of the hardware and the reliability of
software.

36
 Hardware reliability Vs. software reliability

Defect intensity
Defect intensity

Release Need to
reengineer
Maintenance

Useful Test Operational life Obsolete

Infant Wear out
life &
mortali Time
Time debug
ty
(a) Hardware reliability (b) Software reliability

37
 Reliability Metrics
• Probability of Failure on Demand (POFOD):
– POFOD measures the likelihood of system failure in a service
request. It is measured for systems where there is a long gap
between service requests. Service requests occur in an
infrequent way.
• ROCOF (Rate of Occurrence of Failure):
– It is used to measure the frequency occurrence of unexpected
failures. It is measured by running a system over a specified
time duration.
• Mean Time to Failure (MTTF):
– MTTF is the average time interval between the consecutive
failures observed over a large number of failures. It considers
only the run time of the system.

38
 Reliability Metrics
• Mean Time to Repair (MTTR):
– MTTR is the average time taken to repair defects in a system. Time
is considered for detecting and fixing defects over a specified time
interval.
• Mean Time between Failures (MTBF):
– MTBF is measured by combining MTTR and MTTF. That is,

MTBF = MTTR + MTTF

• System availability:
– It is the likelihood that the system will be available for use over a
given time duration.
– System availability measurement excludes MTTR and MTTF. The
measurement of availability is important for systems like server
machines, telecommunication systems, etc.

39
 Reliability Approaches

• Fault Avoidance
– It is the ability to overcome an undesirable outcome in a
specified circumstance.
• Fault Detection
– A fault detection mechanism is helpful to detect and isolate
faults as early as possible.
• Fault Tolerance
– Fault tolerance is the extent to which a system will continue
to operate at a defined performance even though there exist
some malfunctioning units or components.

40