Software Quality

Assurance

1
What is Software Quality
Assurance?

2
What is Quality?

Quality – developed product meets it’s

specification
Problems:
• Development organization has requirements exceeding
customer's specifications (added cost of product
development)
• Certain quality characteristics can not be specified in

unambiguous terms (i.e. maintainability)

• Even if the product conforms to it’s specifications, users
may
not consider it to be a quality product (because users may not
be involved in the development of the requirements)
3
What is Quality
Management?

Quality Management – ensuring that required level of

product quality is achieved
• Defining procedures and standards
• Applying procedures and standards to the product and process
• Checking that procedures are followed
• Collecting and analyzing various quality data

Problems:
• Intangible aspects of software quality can’t be
standardized
(i.e elegance and readability)

4
What are SQA, SQP, SQC,
and SQM?
SQA includes all 4 elements…
• Software Quality Assurance – establishment of network of
organizational procedures and standards leading to high-
quality software
2. Software Quality Planning – selection of appropriate
procedures and standards from this framework and adaptation
of these to specific software project
3. Software Quality Control – definition and enactment of
processes that ensure that project quality procedures and
standards are being followed by the software development
team
4. Software Quality Metrics – collecting and analyzing quality
data to predict and control quality of the software product
being developed 5
Software Development
Standards

6
Why are Standards Important?

• Standards provide encapsulation of best, or at least most

appropriate, practice
• Standards provide a framework around which the quality
assurance process may be implemented
• Standards assist in continuity of work when it’s carried out by
different people throughout the software product lifecycle

Standards should not be avoided. If they are too extensive

for the task at hand, then they should be tailored.

7
SDS a Simplistic approach

In most mature organizations:

• ISO is not the only source of SDS
• Process and Product standards are derived independently
• Product standards are not created by SQA
8
 Process Standards

rocess Standards – standards that define the proc

hich should be followed during software developmen
ISO CMM CMMI

Organizational Organizational
IPDS
Quality Manual SD Process STD’s

Project Project SD Process STD’s Project

SQP (SDP, IP, Method Sheets) SCMP

9
 Product Standards

roduct Standards – standards that apply to softwa

roduct being developed

ISO MIL/ Industry

STD’s STD’s

Organizational
Product STD’s

COTS Project Product STD’s (SDP,

STD’s IP, Method Sheets)

10
Quality Models

11
ISO - 9001 Elements
 Quality System Requirements  Software Quality Responsibilities
 Management Responsibility  Management Responsibility
 Quality system  Quality system
 Contract review  Contract review
 Design Control  Design Control
 Document control  Document control
 Purchasing  Purchasing
 Purchaser supplied product  -
 Product identification and traceability  Product identification and traceability
 Process control  Process control
 Inspection and testing  Inspection and testing
 Inspection, measuring and test equipment  -
 Inspection and test status  Inspection and test status
 Control of non-conforming product  -
 Corrective action  Corrective action
 Handling, storage, preservation, packaging and shipping  -
 Quality records
 Internal quality audits  Quality records
 Training  Internal quality audits
 Servicing  Training
 Statistical techniques  -
 Statistical techniques
12
Capability Maturity Model
KPA’s

13
CMM Level’s 2-5

14
CMM Integration Model
Architecture

15
CMM to CMMI Mapping

16
Documentation

17
Documentation Hierarchy

• Documents are not the only tangible way of representing software

products. The working software system is the most tangible way of
representing software products.
• Documents are the best way to ensure software products’
understandability

18
Process and Product Quality
Quality of development process directly affects the quality of
delivered products.

This is the factory approach. It doesn’t work because software is

designed rather then manufactured.

19
Quality Improvement – The Wheel of
6Sigma

Six Sigma

20
Quality Improvement – Six Sigma
Process
• Visualize – Understand how it works now and imagine how it will
work in the future

• Commit – Obtain commitment to change from the stakeholders

• Prioritize – Define priorities for incremental improvements

• Characterize – Define existing process and define the time

progression for incremental improvements

• Improve – Design and implement identified improvements

• Achieve – Realize the results of the change

21
Continuity and Independence of
SQA
• Software Quality Assurance team must be independent in order to
take an objective view of the process and report problems to senior
management directly

• If prescribed process is inappropriate for the type of software

product which is being developed, then it should be tailored

• The standards must be upheld no matter how small the task.

Prototyping doesn’t mean no standards. It means tailored standards.

• Quality is FREE, if it’s Everyone’s Responsibility!

22
Software Quality
Planning
Element II

23
Software Quality Plan
 Tailoring - SQP should select those organizational
standards that are appropriate to a particular product
 Standardization - SQP should use (call out) only
approved organizational process and product
standards
 If new standards are required a quality improvement
should be initiated
 Elements - SQP elements are usually based on the
ISO-9001 model elements
 SQP is not written for software developers. It’s
written for SQE’s as a guide for SQC and for the
customer to monitor development activities
 Things like software production, software product
plans and risk management should be defined in SDP,
IP
 Quality Factor’s shouldn’t be sacrificed to achieve
efficiency. Don’t take the job if quality process can’t
24

be upheld
Software Quality
Control

25
Methods of Software Quality
Control
SQC involves overseeing the software development process to
ensure that the
procedures and STD’s are being followed

The following activities constitute SQC:

• Quality Reviews - in-process reviews of processes and products
Reviews are the most widely used method of validating the
quality of processes and products. Reviews make quality
everyone's responsibility. Quality must be built-in. SQE is
responsible for writing Quality Engineering Records (QERs)
documenting their participation in these reviews.

• Tests - end-result verifications of products. These verifications are

conducted after the software has been developed. Test procedures
are followed during conduct of these activities. SQE is responsible for
keeping the logs and some times for writing the test report.

• Quality Audits - in-process verifications of processes. These

26

audits are conducted periodically (twice a month) to assess

Quality Reviews
• Peer reviews - reviews of processes and products by groups
of people. These reviews require pre-review preparation by all
participants. If a participant is not prepared, then the review is
not effective. This type of review requires participation of the
SQE, moderator, recorder, author(s), and one or more critical
reviewers. All issues found during these reviews are
documented on AR forms.

• Walkthroughs - reviews of products by groups of people

mostly without preparation. For example a requirements
traceability review is a walkthrough. It involves tracing a
requirement from customer requirements to the test procedures.
All issues found during these reviews are documented on CAR
forms.

• Desk inspections - reviews of products by individuals. These

reviews involve people reviewing products by themselves
27 (not in
a group) and then submitting their comments to the author(s).
The issues found during these reviews are treated in informal
Tests

• Engineering Dry-run - test conducted by engineering without

SQE. These tests include Unit Tests and engineering dry-runs of the
formal tests. These engineering dry-runs are used to verify
correctness and completeness of the test procedures. Also, these is
the final engineering verification of the end-product before sell-off to
SQE. All issues found during these tests are documented on STR
forms.

• SQE Dry-run - test conducted by SQE. These tests include PQT,

FAT and SAT dry-runs. These tests are used to verify the end-product
before the formal test with the customer. An SQE is sometimes
responsible for writing the test report. However, if a separate test
group is available, then SQE is relived of this obligation. All issues
found during these tests are documented on STR forms.

• TFR - test conducted as “RFR - run-for-record” with the SQE and the
customer. These tests include FAT and SAT. These tests
28
are
conducted to sell the end-product off to the customer. SQE is present
Quality Audits

• SQE Audits - audits conducted by SQE to verify that the

process STD’s are being followed. Examples of these audits
are IPDS compliance, Configuration Control, and Software
Engineering Management. All findings for these audits are
documented on QER forms. The results of the audits are
distributed to the next level of management (above project
level). If the issue(s) are not fixed then the findings are elevated
to upper management.

• Independent Audits - audits conducted by ISO generalists or

other independent entities to verify that the process STD’s are
being followed. These audits are usually conducted on a
division/facility level. The results of these audits are distributed
to upper management.

29
Defect Detection
Formal bug finding activities include Quality Reviews and Tests

At Baseline Capture 0
System Requirements Analysis 0 79

D Software Requirements Analysis 0 0 1

E Preliminary Design 0 6 2 10
S
T Detailed Design 1 0 0 0 42
T
E Code 0 0 0 1 2 37
A
C Unit Test 0 0 0 0 0 0 0
G
T Software Integration 1 0 0 0 4 1 0 0
E
E Software Qualification Test 0 0 0 0 0 0 0 0 0

D System Integration 1 0 0 0 4 5 0 0 0
System Test 0 0 0 0 0 0 0 0
Post System Test 0 0 0 0 0 0 0 0 0

% Defects Originated in This Phase That Were Contained By This Phase 93% 33% 91% 81% 86%

% Defects Originated in This Phase Plus Defects

93% 11% 95% 79% 74% 0% 36% 0%
That Escaped From Earlier Phases That Were Contained By This Phase

% Defects Originated I n This Phase Out Of All Defects 44% 2% 6% 27% 22% 0% 0% 0%

Chart Data Last Updated: 10/3/01

30
A Bug’s Life

Postponed Rejected

P X

New Assigned Open Resolved Merged Tested Verified

N A Engineer O R M T V
SCCB Engineer Software Lead Tester SCCB
Approves STR Accepts STR Resolves STR Verifies Fix
Plans Merge Agrees Closure

Integrator
Duplicate
Performs Merge
D

31
Software Configuration
Management
SCM – activities assuring that software products are properly identified
and their transition is tracked. In many mature organizations SCM is
not part of SQA responsibilities.

• Baseline Identification – identification of initial state of the product

• Change Identification – identification of changes made to the
baseline
• Change Control – documentation of changes via revision history,
change summary, or using automated development tools (ClearCase or
Apex)
• Status Accounting – reporting changes to others and monitoring
completeness of the project archives
• Preservation – keeper of the software products

32
Software Quality
Metrics

33
Metrics Collection

• Software measurement - the process of deriving a numeric

value
for some attribute of a software product or a software process.
Comparison of these values to each other and to STD’s allows
drawing conclusions about the quality of software products or the
process.

• The focus of the metrics collecting programs is usually on

collecting metrics on program defects and the V&V process.

• Metrics can be either Control Metrics or Predictor Metrics

• Most of the “Ilities” can not be measured directly unless there’s

historical data. Instead tangible software product attributes are
measured and the “Ility” factors are derived using predefined
relationships between measurable and synthetic attributes.
34

• The boundary conditions for all measurements should be

The Process of Product
Measurement

1. Decide what data is to be collected

2. Assess critical (core) components first
3. Measuring component characteristics might require automated
tools
4. Look for consistently (unusually only works in a factory)
high or low values
5. Analysis of anomalous components should reveal if the quality
35

of product is compromised
Software Product Metrics

There are two categories of software product metrics:

1. Dynamic metrics – this metrics is collected by measuring elements
during program’s execution. This metrics help to asses efficiency and
reliability of a software product. The parameters collected can be
easily measured (i.e. execution time, mean time between failures)

2. Static metrics – this metrics is collected by measuring parameters of

the end products of the software development. This metrics help to
asses the complexity, understandability, and maintainability of a
software product

36
Examples of Software Metric

37
Examples of OO Software Metric

38
Defect Prevention
Defect Prevention – establishment of practices that lower the reliance
on defect detection techniques to find majority of the bugs

• Lessons learned – learning from other peoples experiences and

sharing own experiences with the other projects
• Managing With Metrics – collecting the metrics, understanding
it, and making changes to the product or process based on
analysis. Metrics must be standardized to be effective.
• Risk Analysis – identifying potential risks and opportunities
early in the program and tracking them to realization.
• Build freeze – no changes are made to the code during formal
tests.
• Unit-level testing guidelines – test plans and procedures for
each UT

39
The end

40