SWE 439

Software Quality Engineering

Introduction
 Objectives

 Understand what is quality

 Learn the quality engineering process and activities
 Discuss Verification and Validation (V&V) and their relationship with quality
assurance

2
 What is software quality?

 Quality is fitness for use

 Quality is compliance to a standard
 Quality is a degree of excellence
 A product which fulfills and continues to meet the purpose for which it was produced is a
quality product
 Conformity to user expectations, conformity to user requirements, customer satisfaction,
reliability and level of defects present
 The degree to which a set of inherent characteristics fulfils requirements
 The totality of features and characteristics of a product or service that bear or its ability to
satisfy specified or implied needs. (ISO 9003-3-1991).
 The degree to which a system, component, or process meets–
 Specified requirements, and
 Customer or user needs or expectations. (IEEE std. 610.12-1990)

3
 High quality software

 It must be useful (to the original customer)

 It must be portable (work at all customer’s sites)
 It must be maintainable
 It must be reliable
 It must have integrity (produces correct results, with a high degree of
accuracy)
 It must be efficient
 It must have consistency of function (it does what the user expects it to do)
 It must be accessible (to the user)
 It must have good human engineering (easy to learn and easy to use)

4
 Quality control and quality assurance

 There are two ways of knowing whether good quality software has been
produced or NOT:
 Measuring the attributes of the software that has been developed (quality control)
 Monitoring and controlling the development process of the software (quality assurance)

Quality assurance: A planned and systematic pattern of all

actions necessary to provide adequate confidence that an item
or product conforms to established technical requirements

5
 Views for quality

 In the user view, quality is fitness for purpose or meeting user’s needs
 In the manufacturing view, quality means conformance to process
standards
 In the product view, the focus is on inherent characteristics in the product
itself in the hope that controlling these internal quality indicators (product
internal metrics) will result in improved external product behavior (quality in
use)
 In the value-based view, quality is the customers’ willingness to pay for a
software

6
 What is the benefit of software quality

 Quality provides:
 The potential for an increased profit
 Consumers with quality in return for the money they pay

 Better quality leads to:

 Increased production
 Lower warranty costs
 Lower marketing costs
 Fewer surprises

Lack of Quality is the Reason for Failure

7
 Main tasks for software quality engineering

SWE 387

 Quality planning
SWE 326

 Execution of Quality Assurance (QA) activities

SWE 439

 Measurement and analysis

 To provide convincing evidence to demonstrate software quality to all parties involved

8
 Quality engineering activities and process

Set quality goals

Entry Quality
Quality No Quality goals Yes
Assurance
Planning Selected QA Feedback & satisfied?
Activities Exit
activities adjustment

Measurements
Selected
measurements Analysis/
& models modeling
Quality
results
Assessment &
Improvement

Jeff Tian, “Software Quality Engineering: Testing, Quality Assurance, and Quantifiable Improvement”, Wiley-IEEE Computer Society, 2005. 9
 Quality engineering activities and process

Quality Planning:
Making adjustments based on
Setting quality goals feedback
Select quality assurance strategies

Quality Assurance:
QA Phase: Testing
Quality gates: at phase transition pairs, e.g., passing design reviews before coding
Other QA activities scattered over all phases, e.g., inspection specs/design/code/test cases
Quality Assessment and improvement

INPUT: measurement from QA and development activities

OUTPUT: quality assessment and other analysis results

Jeff Tian, “Software Quality Engineering: Testing, Quality Assurance, and Quantifiable Improvement”, Wiley-IEEE Computer Society, 2005. 10
 Quality engineering effort

Jeff Tian, “Software Quality Engineering: Testing, Quality Assurance, and Quantifiable Improvement”, Wiley-IEEE Computer Society, 2005. 11
 Quality engineering process

Pre-QA activities In-QA activities Post-QA activities

• Quality planning • Executing QA activities • Quality measurement,
assessment and improvement

12
 Pre-QA activities - Quality planning

 Activities that should be carried out before carrying the regular QA activities
 Pre-QA quality planning includes setting quality goals and forming a QA
strategy:
 Setting quality goals to manage the QA activities and stop when the quality goals are me
by matching the quality expectations with what can be economically achieved:
• Identify quality views and attributes meaningful to target customers and users
• Select direct quality measures that can be used to measure the selected quality attributes
• Quantify these quality measures to set quality goals
 Forming a QA strategy: should be selected, before we can carry out specific QA activities,
collect data, perform analysis, and provide feedback
• Select appropriate QA activities to perform
• Choose appropriate quality measurements and models to provide feedback, quality assessment
and improvement
• Map the external quality measures into internal ones via selected quality models
13
 Setting quality goals

 Different quality attributes may have different levels of importance to different

customers and users.

 Relevant quality views and attributes need to be identified first:

 For example, reliability is typically the primary concern for various business and
commercial software systems because of people’s reliance on such systems and the
substantial financial loss if they are malfunctioning.
 Similarly, if a software is used in various real-time control situations, such as air traffic
control software and embedded software in automobile, medical devices, etc., accidents
due to failures may be catastrophic. Therefore, safety is the major concern.
 On the other hand, for mass market software packages, such as various auxiliary utilities
for personal computers, usability, instead of reliability or safety, is the primary concern.

14
 Setting quality goals

 Once we obtain qualitative knowledge about quality expectations, we need to

quantify these quality expectations to set appropriate quality goals in two
steps:
 Select or define the quality measurements and models commonly accepted
• For example, reliability and safety are examples of correctness-centered quality measures that
are meaningful to customers and users, which can be related to various internal measures of
faults commonly used within software development organizations
 Find out the expected values or ranges of the corresponding quality measurements
• For example, different market segments might have different reliability expectations. Such
quality expectations are also influenced by the general market conditions and competitive
pressure
 Another practical concern with the proper setting of quality goals is the cost associated
with different levels of quality.

15
 Forming a QA strategy

 Once specific quality goals were set, we can select appropriate QA

alternatives as part of a QA strategy to achieve these goals

 The influence of quality perspectives and attributes

 For example, various usability testing techniques may be useful for ensuring the usability of a
software product but may not be effective for ensuring its functional correctness

 The influence of different quality levels

 For example, systems with various software fault tolerance features may incur substantially more
additional cost than the ones without them

16
 In-QA activities - Executing QA activities

 Execute the planned QA activities

 Handle the discovered defects
 Deal with the discovered problems

17
 Quality assurance

Rajiv Chopra, “Software Quality Assurance - A Self-Teaching Introduction”, 2018 18

 Quality expectations

 People’s quality expectations for software systems they use rely upon are:
 They must perform these specific tasks correctly or satisfactorily (things right) – Verification
 The software systems must do what they are supposed to do (right things) – Validation

 Verification requires that the software systems perform their intended functions without
problems.
 For example, in the airline reservation system, the system should help travel agents or individual travelers make
valid reservations within a pre-specified time limit, instead of making invalid ones, taking too long to make a
reservation, or refusing to make reservations without proper justification.
 The focus of the related activities is to verify that the implemented software functions operate as specified.
 Validation requires that the software be the “right software” or perform the right functions.
 For example, an airline reservation system is supposed to handle reservations, not intended to fly airplanes
automatically.
 The focus of the related activities is to validate the required software functions under their intended operational
environment.
19
 Verification and Validation (V&V)

 Verification is defined as “the process of evaluating a system or component to

determine whether the products of a given development phase satisfy the conditions
imposed at the start of that phase”.

 Validation is defined as “the process of evaluating a system or component during or

at the end of the development process to determine whether it satisfies specified
requirements.”

 V&V is “a systems engineering process employing a rigorous methodology for

evaluating the correctness and quality of software product through the software life
cycle.”
V&V activities are core actions of SQA, and the key means of
V&V activities are documental review and program testing
20
 Software Quality and V&V

 Before being executed, V&V should be carefully planned as different quality attributes can
be verified in different phases of the life cycle of a system or software
 For example, static (internal) quality can be only verified during the coding phase
 Dynamic (external) quality requires a running code, so the first moment any measure may be taken is in the
phase of system tests

21
Role of V&V in requirements

22
Role of V&V in design

23
Role of V&V in implementation

24
Role of V&V in testing

25
Role of V&V in maintenance

26
 Positive effects of Software V&V on software projects

 Better quality of software

 More stable requirements
 More rigorous development planning, at least to interface with the software
V&V organization
 Better adherence by the development organization to programming language
and development standards and configuration management practices
 Early error detection and reduced false starts
 Better schedule compliance and progress monitoring
 Greater project management visibility into interim technical quality and
progress
 Better criteria and results for decision making at formal reviews and audits
27
 Negative effects of Software V&V on a software project

 Additional project cost of software V&V (10–30% extra).

 Additional interface involving the development team, user, and software V&V organization.
 Additional documentation requirement beyond the deliverable products
 Need to share computing facilities with and access to classified data for the software V&V
organization.
 Lower development staff productivity if programmers and engineers spend time explaining
the system to software V&V analysts, especially if explanations are not documented.
 Increased paperwork to provide written responses to software V&V error reports and other
V&V data requirements.
 For example, notices of formal review and audit meetings, updates to software release schedule, and response
to anomaly reports.
 Productivity of development staff affected adversely in resolving invalid anomaly reports.

28
 Post-QA activities - Quality measurement, assessment and
improvement

 The primary purpose of these activities:

 Provide quality assessment and feedback so that various management decisions can be
made
 Possible quality improvement initiatives.

 The “post-QA” does not mean after the finish of QA activities

 Many of the measurement and analysis activities are carried out parallel to QA activities
after they are started

29
 Quality assessment and improvement

 Various parallel and post-QA activities are carried out to close the quality
engineering loop
 The primary purpose of these activities is to provide quality assessment and
feedback so that various management decisions, such as product release,
can be made and possible quality and process improvement initiatives can
be carried out
 The major activities in this category include:
 Measurement
 Analysis and modeling
 Providing feedback and identifying improvement potentials
 Follow-up activities

30
 Cost of software quality

 Cost of software quality (CoSQ) is defined as the sum of the following project
costs:
 Cost of planning and construction activities (e.g., planning, designing, development);
 Cost of prevention activities (process improvement, tools, training);
 Cost of appraisal activities for defect detection (e.g., reviews, audits, testing);
 Cost of nonconformance rework (internal failure cost and external failure cost).

 The CoSQ can be broken down into two top-level categories:

 Conformance cost is the total of all investments in error and defect detection (appraisal)
and prevention activities.
 Nonconformance cost is the total of all spending dealing with errors and defects that
have been detected.
31
 Summary

 Software quality may include many different attributes and may be defined and perceived differently
based on people’s different roles and responsibilities
 To ensure quality, perform many software QA and quality engineering activities
 To manage the quality assurance (QA) activities and to provide realistic opportunities of quantifiable
quality improvement, we need to go beyond QA to perform the following:
 Quality planning before specific QA activities are carried out, in the so-called pre-QA activities in software quality engineering.
• Set the overall quality goal.
• Select specific QA alternatives and techniques to implement as well as measurement and models
 Quality quantification and improvement through measurement, analysis, feedback, and follow-up activities.
• These activities need to be carried out after the start of specific QA activities,
• The analyses would provide us with quantitative assessment of product quality, and identification of improvement
opportunities
• The follow-up actions would implement these quality and process improvement initiatives and help us achieve quantifiable
quality improvement
 The integration of these activities with the QA activities forms our software quality engineering process

32
 Summary

 Verification is doing things right

 Validation is doing the right hings
 The purpose of V&V is to help the development organization build quality into
the software throughout the development life cycle.
 V&V activities are core actions of SQA, and the key means of V&V activities
are documental review and program testing

33
 Resources

 Jeff Tian, “Software Quality Engineering: Testing, Quality

Assurance, and Quantifiable Improvement”, Wiley-IEEE
Computer Society, 2005.

 Rajiv Chopra, “Software Quality Assurance - A Self-Teaching

Introduction”, 2018

 Witold Suryn, “Software Quality Engineering”, 2014

34