Software Project Management

1
 Chapter Seven
Software Quality Assurance
and Reliability

2
 Software Reliability

l IEEE-Std-729-1991: “Software reliability is defined as the

probability of failure-free operation for a specified period of
time in a specified environment”.
l ISO9126: “Reliability is the capability of the software
product to maintain a specified level of performance when
used under specified conditions”.
l Informal: Reliability is a measure of how well the software
provides the services expected by the customer.
l Quantification: Number of failures, severity.

3
 Cont’d
l Reliability is one of the metrics that are used to measure
quality.
l It is a user-oriented quality factor relating to system
operation.
» Intuitively, if the users of a system rarely experience failure, the
system is considered to be more reliable than one that fails more
often.
l A system without faults is considered to be highly reliable.
» Constructing a correct system is a difficult task.
» Even an incorrect system may be considered to be reliable if the
frequency of failure is “acceptable.”

4
 Cont’d
l Key concepts in discussing reliability:
» Fault
» Failure
» Time
» Three kinds of time intervals: MTTR, MTTF, MTBF
l Failure
» A failure is said to occur if the observable outcome of a program
execution is different from the expected outcome.
l Fault
» The adjudged cause of failure is called a fault.
» Example: A failure may be cause by a defective block of code.

5
 Cont’d
l Time
» Time is a key concept in the formulation of reliability.
» If the time gap between two successive failures is short, we say that
the system is less reliable.
» Two forms of time are considered.
– Execution time ()
– Calendar time (t)
l MTTF: Mean Time To Failure
l MTTR: Mean Time To Repair
l MTBF: Mean Time Between Failures (= MTTF + MTTR)

6
 Cont’d

Relationship between MTTR, MTTF, and MTBF

7
 Measuring Software Reliability

l Two ways to measure reliability

» Counting failures in periodic intervals
– Observe the trend of cumulative failure count - µ().
» Failure intensity
– Observe the trend of number of failures per unit time – λ().
l µ()
» This denotes the total number of failures observed until execution
time  from the beginning of system execution.
l λ()
» This denotes the number of failures observed per unit time after 
time units of executing the system from the beginning. This is also
called the failure intensity at time .
l Relationship between λ() and µ()
» λ() = dµ()/d 8
 Software Reliability Metrics
l Reliability metrics are units of measure for system reliability.
l System reliability is measured by counting the number of
operational failures and relating these to demands made on
the system at the time of failure.
l A long-term measurement program is required to assess the
reliability of critical systems.
l Probability of Failure on Demand (POFOD)
» POFOD = 0.001 - means 1 out of 1000 service requests result in
failure (for one in every 1000 requests the service fails per time unit).
» Probability system will fail when a service request is made.
» Useful when requests are made on an intermittent or infrequent basis.
» Appropriate for protection systems service requests may be rare and
consequences can be serious if service is not delivered.
» Relevant for many safety-critical systems with exception handlers. 9
 Cont’d
l Rate of Fault Occurrence (ROCOF)
» ROCOF = 0.02 - two failures for each 100 operational time units of
operation.
» Reflects rate of failure in the system.
» Useful when system has to process a large number of similar requests
that are relatively frequent.
» Relevant for operating systems and transaction processing systems.
l Mean Time to Failure (MTTF)
» Average time between observed failures.
» Measures time between observable system failures.
» For stable systems MTTF = 1/ROCOF
» Relevant for systems when individual transactions take lots of
processing time (e.g. CAD or WP systems)
10
 Cont’d
l Availability
» Measures the fraction of time system is really available for use.
» Takes repair and restart times into account.
» Relevant for non-stop continuously running systems (e.g. traffic signal)
» MTBF = Mean Time Between Failure
» MTTR = Mean Time to Repair
» Availability = MTBF / (MTBF+MTTR)
l Reliability = MTBF / (1+MTBF)
l Time Units
» Raw Execution Time - non-stop system.
» Calendar Time - if the system has regular usage patterns.
» Number of Transactions - demand type transaction systems.

11
 Failure Classification
l Transient - only occurs with certain inputs
l Permanent - occurs on all inputs
l Recoverable - system can recover without operator help.
l Unrecoverable - operator has to help.
l Non-corrupting - failure does not corrupt system state or data.
l Corrupting - system state or data are altered.

Failure Class Example Metric

Permanent ATM fails to operate ROCOF = .0001
Non-corrupting with any card, must Time unit = days
restart to correct
Transient Magnetic stripe can't POFOD = .0001
Non-corrupting be read on Time unit = transactions
undamaged card

12
 Software Reliability Models
l Software developers are struggling to deliver reliable
software with acceptable level of quality, within given
budget and schedule.
l Therefore, researchers are focusing on the identification of
the number of fault presents in the software or identification
of program modules that are most likely to contain faults.
l Detecting software faults early during development will
definitely improve the reliability and quality in cost-
effective way.
l A number of analytical models are developed that are based
mainly on the failure history of software and can be
classified according to the nature of the failure process.
13
 Cont’d

l Software reliability models usually refer to estimating the

number of remaining errors in a partially debugged software.
l Software reliability models can be broadly categorized as
suggested by Pham (2006):
» Deterministic - used to study the number of distinct operators and
operands as well as the machine instructions in the program.
– Ex. Halstead’s software metric and McCabe’s cyclomatic complexity
metric.
» Probabilistic - describes the failure occurrence and\or fault removal
phenomenon of the testing process as probabilistic events with
respect to time and\or testing effort.
– Failure rate model (times between failure models),
– Failure or fault count model (NHPP models).
– Error or fault-seeding model.
– Reliability growth model, etc. 14
 Failure Rate Models
(Time between Failure)

l It assumes that the time between (i-1)th and the ith failures
follows a distribution whose parameters depend on the
number of faults remaining in the program during this interval.
l Estimates of the parameters are obtained from the observed
values of time between failures or failure count in intervals.
l The estimates of software reliability, mean time to next
failure, etc., are then obtained from the fitted model.
l List of some failure rate models (Goel 1985):
» Jelinski and Moranda Model.
» Schick and Wolverton Model.
» Goel and Okumoto Imperfect Debugging Model, etc.

15
 Failure or Fault Count Models
l The interest of this class of models is in the number of faults
or failures in specified time intervals rather than in times
between failures.
l The failure counts are assumed to follow a known stochastic
process with a time-dependent discrete or continuous failure
rate.
l Parameters of the failure rate can be estimated from the
observed values of failure counts or from failure times.
l Estimates of software reliability, mean time to next failure,
etc., can again be obtained from the relevant equations.

16
 Cont’d
l The key models in this class are Goel (1985):
» Shooman exponential model.
» Musa execution time model.
» Goel–Okumoto non-homogeneous Poisson process model (G–O-
NHPP).
» S-shape growth model.
» Discrete reliability growth model.
» Musa-Okumoto logarithmic Poisson execution model.
» Generalized NHPP, etc.

17
 Error or Fault-Seeding Models
l The basic approach in this class of models is to ‘‘seed’’
known number of faults in the program which is assumed to
have an unknown number of indigenous faults.
l The program is tested and observed for numbers of seeded
and indigenous faults.
l From these, an estimate of the indigenous fault content of
the program is obtained and used to assess software
reliability and other relevant measures.
l Examples:
» Mills Hyper geometric model

18
 Reliability Growth Models
l Software reliability growth is defined as the phenomenon
that the number of software errors remaining in the system
decreases with the progress of testing.
l It is applicable during the testing phase and quantifies the
software reliability in terms of estimated number of software
error remaining in software or estimated time intervals
between software failures.
l Software reliability growth models have been grouped into two
classes of models: concave and S-shaped.
l They have the same asymptotic behavior, that is, the defect
detection rate decreases as the number of defects detected
(and repaired) increases, and the total number of defects
detected approach a finite value asymptotically.
19
 Cont’d

Concave and S-shaped model

20
Software reliability growth models (Wood 1996)

21
 Software Quality Assurance
l What is quality?
» IEEE Glossary: Degree to which a system, component, or process
meets (1) specified requirements, and (2) customer or user needs or
expectations.
» ISO: the totality of features and characteristics of a product or service
that bear on its ability to satisfy specified or implied needs.
» Traditional definition: fitness of purpose, a quality product does
exactly what the users want it to do.
l Quality Criteria include:
» Correctness,
» efficiency,
» flexibility,
» integrity,
» interoperability,
» maintainability,
» portability, reliability, reusability, testability and usability 22
 Cont’d
l Software quality assurance (SQA)
» Consists of a means of monitoring the software engineering processes
and methods used to ensure quality.
» It does this by means of audits of the quality management system under
which the software system is created.
» These audits are backed by one or more standards, usually ISO 9000 or
CMMI.
l It is practically impossible to iron out every single bug before
releasing it both from a difficulty point of view and due to time
constraints.

23
 Quality Assurance (QA)
l Measures the quality of processes used to create a quality
product.
» A set of activities designed to ensure that the development and/or
maintenance process is adequate to ensure a system will meet its
objectives.
» QA activities ensure that the process is defined and appropriate.
» Methodology and standards development are examples of QA
activities.
» A QA review would focus on the process elements of a project

24
 Quality Control (QC)
l Measures the quality of a product.
» A set of activities (inspections, reviews, and test ) designed to evaluate
a developed work product.
» QC activities focus on finding defects in specific deliverables.
l Two types of quality control:
» Quality design - the characteristics that designers specify for an item.
– includes: requirements, specifications, and the design of the system.
» Quality of conformance - the degree to which the design
specification are followed.
– It focuses on implementation based on the design.
l Generally, QC compare the work products with the specified
and measurable standards.

25
 Cont’d
l Testing - Quality control
» The process of executing a system with the intent of finding defects.
» includes test planning prior to the execution of the test cases.
» Testing is one example of a QC activity, but there are others such as
inspections

26
 Quality Assurance Vs. Quality Control
l Both QA and QC activities are generally required for
successful software development.
l Software quality control is a control of products.
l Software quality assurance is a control of processes.
» Related to the practice of quality assurance in product manufacturing
l Software vs. manufactured product
» Manufactured product is physical and can be seen, Software product is
not visible.
» Manufactured product rolls off the assembly line, it is essentially a
complete, finished product , Software is never finished.
l The processes and methods to manage, monitor, and
measure software ongoing quality are as fluid and sometimes
elusive as are the defects that they are meant to keep in
check.

27
 Software Quality Assurance Jobs
l Includes:
» Reviewing requirements documents
» Software testing
l It also encompasses the entire software development process
» Software design
» Coding
» Source code control
» Code reviews
» Change management
» Configuration management, Release management.

28
 Software Quality Assurance Methodology
l Process and Product Quality Assurance (PPQA) audits:
» the activity of ensuring that the process and work product conform to
the agreed upon process.

29
 Quality Control Activities
l Peer Reviews:
» Peer reviews of a project's work products are the most efficient defect
removal (quality control) activity.
l Validation testing
» the act of entering data that the tester knows to be erroneous into an
application.
– Example, typing "Hello" into an edit box that is expecting to receive a
numeric entry.
l Data comparison
» Comparing the output of an application with specific parameters to a
previously created set of data with the same parameters that is known
to be accurate.

30
 Cont’d
l Stress testing:
» A stress test is when the software is used as heavily as possible for a
period of time to see whether it copes with high levels of load.
» Often used for server software that will have multiple users connected
to it simultaneously.
» Also known as Destruction testing.
l Usability testing
» Sometimes getting users who are unfamiliar with the software to try it
for a while and offer feedback to the developers about what they found
difficult to do is the best way of making improvements to a user
interface

31
 Advantages of Quality Assurance
l Improved customer satisfaction
l Reduced cost of maintenance
l Eliminate errors when they are still inexpensive to correct
l Improves the quality of the software
l Improving the process of creating software

32
 McCall’s Quality Factors
l McCall, Richards, and Walters [MCC77] propose a useful categorization of
factors that affect software quality.
l These software quality factors focus on three important aspects of a
software product:
» its operational characteristics,
» its ability to undergo change, and
» its adaptability to new environments.

33
 Cont’d
l Correctness - The extent to which a program satisfies its
specification and fulfills the customer’s mission objectives.
l Reliability - The extent to which a program can be expected
to perform its intended function with required precision.
l Efficiency - The amount of computing resources and code
required by a program to perform its function.
l Integrity - Extent to which access to software or data by
unauthorized persons can be controlled.
l Usability - Effort required to learn, operate, prepare input,
and interpret output of a program.

34
 Cont’d
l Maintainability - Effort required to locate and fix an error in a
program. [This is a very limited definition.]
l Flexibility - Effort required to modify an operational program.
l Testability - Effort required to test a program to ensure that it
performs its intended function.
l Portability - Effort required to transfer the program from one
hardware and/or software system environment to another.
l Reusability - Extent to which a program [or parts of a
program] can be reused in other applications—related to the
packaging and scope of the functions that the program
performs.
l Interoperability - Effort required to couple one system to
another. 35
 SQA Group

l Who involves quality assurance activities?

» Software engineers, project managers, customers, sale people, SQA
group
l Engineers involved the quality assurance work:
» apply technical methods and measures
» conduct formal technical review
» perform well-planned software testing
l The SQA group’s role
» serves as the customer’s in-house representative
» assist the software engineering team in achieving high-quality

36
 Cont’d
l The SQA group’s responsibility:
» quality assurance planning oversight, record keeping, analysis and
reporting
l The SQA group’s tasks:
» Prepare a SQA plan for a project.
» Participate in the development of the project’s software process
description
» Review engineering activities to verify compliance with the defined
process
» Audits designated software work products to verify compliance the
defined process
» Ensure the deviations in software work and products according to a
documented procedure
» Records any noncompliance and reports to senior management

37
 SQA Plan
l The SQA plan provides a road map for instituting software quality assurance.
l Below presents an outline for SQA plans by IEEE [IEEE94].
l Basic items:
» purpose of plan and its scope
» management
– organization structure, SQA tasks, their placement in the process
– roles and responsibilities related to product quality
» Documentation
– project documents, models, technical documents, user documents.
» standards, practices, and conventions
» reviews and audits
» test -
– test plan and procedure
» problem reporting, and correction actions
» Tools, code control, media control, supplier control,
» records collection, maintenance, and retention
» Training and risk management
38
 SEI Capability Maturity Model
l Developed by Software Engineering Institute (SEI) of the
Carnegie Mellon University, USA:
» to assist the U.S. Department of Defense (DoD) in software acquisition.
» The rationale was to include:
– likely contractor performance as a factor in contract awards.

l Major DoD contractors began CMM-based process

improvement initiatives:
» as they vied for DoD contracts.
l SEI CMM helped organizations:
» Improve quality of software they developed
» Realize adoption of SEI CMM model had significant business benefits.
l Other organizations adopted CMM.
39
 Cont’d

l In simple words,
» CMM is a model for apprising the software process maturity of a
contractor into different levels.
» Can be used to predict the most likely outcome to be expected from
the next project that the organization undertakes.
l Can be used in two ways:
» Capability evaluation
» Software process assessment.

40
 Cont’d
l Capability Evaluation
» Provides a way to assess the software process capability of an
organization
– Helps in selecting a contractor
– Indicates the likely contractor performance
l Software Process Assessment
» Used by an organization to assess its current process:
– Suggests ways to improve the process capability.
– This type of assessment is for purely internal use.

41
 Cont’d
l The SEI CMM classifies software development industries into:
 Five maturity levels.
 Stages are ordered so that improvements at one stage provide foundations for the next
 Based on the pioneering work of Philip Crosby

Optimizing (5)

Managed (4)
Defined (3)
Repeatable (2)

Initial (1)
42
 Level 1: (Initial)

l Organization operates without any formalized process or

project plans.
l An organization at this level is characterized by ad hoc and
often chaotic activities.
l Software production processes are not defined,
» different engineers follow their own process
» development efforts become chaotic.
» The success of projects depend on individual efforts and heroics.
l

43
 Level 2: (Repeatable)

l Basic project management practices

» tracking cost, schedule, and functionality are followed.
l Size and cost estimation techniques
» function point analysis, COCOMO, etc. used.
l Production process is ad hoc
» not formally defined
» also not documented.
l Process used for different projects might vary between projects:
» earlier success on projects with similar applications can be repeated.
» Opportunity to repeat process exist when a company produces a family
of products.

44
 Level 3: (Defined)

l Management and development activities:

» defined and documented.
» Common organization-wide understanding of activities, roles, and
responsibilities.
l The process though defined,
» process and product qualities are not measured.
l ISO 9001 aims at achieving this level.

45
 Level 4: (Managed)
l Quantitative quality goals for products are set.
l Software process and product quality are measured:
» The measured values are used to control the product quality.
» Results of measurement used to evaluate project performance
– rather than improve process.
l Organization sets quantitative quality goals
l World-wide about 100 organizations assessed at this level.

46
 Level 5: (Optimizing)

l Statistics collected from process and product measurements are

analysed:
» continuous process improvement based on the measurements.
– Known types of defects are prevented from recurring by tuning the process
– lessons learned from specific projects incorporated into the process
l Identify best software engineering practices and innovations:
» tools, methods, or process are identified
» transferred throughout the organization
l World-wide about 50 organizations have been assessed at this
level.

47
 Key Process Areas

l Each level is associated with a key process area (KPA)

identifies
» where an organization at the previous level must focus to reach this level
l Level 2 KPAs
» Software project planning
– Size, cost, schedule.
– project monitoring
» Configuration management
» Subcontract management
l Level 3 KPAs
» Process definition and documentation
» Reviews
» Training program 48
 Cont’d
l Level 4 KPAs
» Quantitative measurements
» Process management
l Level 5 KPAs
» Defect prevention
» Technology change management
» Process change management

49
 ISO 9000
l ISO (international Standards Organization):
» a consortium of 63 countries established to formulate and foster
standardization.
l ISO published its 9000 series of standards in 1987.
l ISO 9000 certification:
» serves as a reference for contract between independent parties.
l The ISO 9000 standard:
» specifies guidelines for maintaining a quality system.
l ISO 9000 specifies:
» guidelines for repeatable and high quality product development.
» Also addresses organizational aspects
– responsibilities, reporting, procedures, processes, and resources for
implementing quality management. 50
 Cont’d

l A set of guidelines for the production process.

» not directly concerned about the product it self.
» a series of three standards:
– ISO 9001, ISO 9002, and ISO 9003.
l Based on the premise:
» if a proper process is followed for production:
– good quality products are bound to follow.
l ISO 9001 - Applies to:
» organizations engaged in design, development, production, and
servicing of goods.
» applicable to most software development organizations.

51
 Cont’d

l ISO 9002 - applies to:

» organizations who do not design products:
– but are only involved in production.
l Examples of this category of industries:
» steel or car manufacturing industries
» buy the product and plant designs from external sources:
– only manufacture products.
» not applicable to software development organizations.
l ISO 9003 - applies to:
» organizations involved only in installation and testing of the products.

52
 ISO 9000 for Software Industry

l ISO 9000 is a generic standard:

» applicable to many industries,
– starting from a steel manufacturing industry to a service rendering company.

l Many clauses of ISO 9000 documents:

» use generic terminologies
» very difficult to interpret them in the context of software organizations.

53
 Software vs. Other Industries
l Very difficult to interpret many clauses for software industry:
» software development is radically different from development of other
products.
l Software is intangible
» therefore difficult to control.
– It is difficult to control anything that we cannot see and feel.
» In contrast, in a car manufacturing unit:
– we can see a product being developed through stages such as fitting engine,
fitting doors, etc.
– one can accurately tell about the status of the product at any time.
» Software project management is an altogether different ball game.

54
 Cont’d
l During software development:
» the only raw material consumed is data.
l For any other product development:
» Lot of raw materials consumed
– e.g. Steel industry consumes large volumes of iron ore, coal, limestone, etc.

l ISO 9000 standards have many clauses corresponding to raw

material control .
» not relevant to software organizations.
l Radical differences exist between software and other product
development,
» difficult to interpret various clauses of the original ISO standard in the
context of software industry.
55
 ISO 9000 Part-3

l ISO released a separate document called ISO 9000 part-3 in

1991
» to help interpret the ISO standard for software industry.
l At present,
» official guidance is inadequate

56
 Why Get ISO 9000 Certification?
l Several benefits:
» Confidence of customers in an organization increases
– if organization qualified for ISO 9001 certification.
– This is especially true in the international market.
l Many international software development contracts insist:
» development organization to have ISO 9000 certification.
l Requires:
» a well-documented software production process to be in place.
» contributes to repeatable and higher quality software.
l Makes development process:
» focussed, efficient, and cost-effective

57
 Cont’d
l Points out the weakness of an organizations:
» recommends remedial action.
l Sets the basic framework:
» for development of an optimal process and TQM (Total Quality
Management).
– TQM – Advocates continuous process improvements through process measurements.
– optimizes them through redesign.

58