7.

6 Quality Assurance

The proliferation of software malfunctions, cost overruns, and missed

deadlines demands that methods of software quality control be improved. In
this section, we consider some of the directions being pursued in this
endeavor.

The Scope of Quality Assurance

In the early years of computing, the problem of producing quality software
focused on removing programming errors that occurred during
implementation. Later in this section we will discuss the progress that has
been made in this direction. However, today, the scope of software quality
control extends far beyond the debugging process, with branches including
the improvement of software engineering procedures, the development of
training programs that in many cases lead to certification, and the
establishment of standards on which sound software engineering can be
based. In this regard, we have already noted the role of organizations such as
ISO, IEEE, and ACM in improving professionalism and establishing standards
for assessing quality control within software development companies. A
specific example is the ISO 9000-series of standards, which address
numerous industrial activities such as design, production, installation, and
servicing. Another example is ISO/IEC 33001, which is a set of standards
developed jointly by the ISO and the International Electrotechnical
Commission (IEC).
Many major software contractors now require that the organizations they
hire to develop software meet such standards. As a result, software
development companies are establishing software quality assurance (SQA)
groups, which are charged with overseeing and enforcing the quality control
systems adopted by the organization. Thus, in the case of the traditional
waterfall model, the SQA group would be charged with the task of approving
the software requirements specification before the design stage began or
approving the design and its related documents before implementation was
initiated.
Essential Knowledge Statements
• Collaboration can make it easier to find and correct errors when developing
programs.

Several themes underlie today’s quality control efforts. One is record

keeping. It is paramount that each step in the development process be
accurately documented for future reference. However, this goal conflicts with
human nature. At issue is the temptation to make decisions or change
decisions without updating the related documents. The result is the chance
that records will be incorrect and hence their use at future stages will be
misleading. Herein lies an important benefit of CASE tools. They make such
tasks as redrawing diagrams and updating data dictionaries much easier than
with manual methods. Consequently, updates are more likely to be made and
the final documentation is more likely to be accurate. (This example is only
one of many instances in which software engineering must cope with the
faults of human nature. Others include the inevitable personality conflicts,
jealousies, and ego clashes that arise when people work together.)
Another quality-oriented theme is the use of reviews in which various
parties involved in a software development project meet to consider a specific
topic. Reviews occur throughout the software development process, taking
the form of requirements reviews, design reviews, and implementation
reviews. They may appear as a prototype demonstration in the early stages
of requirements analysis, as a structured walkthrough among members of the
software design team, or as coordination among programmers who are
implementing related portions of the design. Such reviews, on a recurring
basis, provide communication channels through which misunderstandings
can be avoided and errors can be corrected before they grow into disasters.
The significance of reviews is exemplified by the fact that they are specifically
addressed in the IEEE Standard for Software Reviews, known as IEEE 1028.
Some reviews are pivotal in nature. An example is the review between
representatives of a project’s stakeholders and the software development
team at which the final software requirements specification is approved.
Indeed, this approval marks the end of the formal requirements analysis
phase and is the basis on which the remaining development will progress.
However, all reviews are significant, and for the sake of quality control, they
should be documented as part of the ongoing record maintenance process.

Essential Knowledge Statements

• Program documentation helps programmers develop and maintain correct
programs to efficiently solve problems.

Software Testing

Whereas software quality assurance is now recognized as a subject that

perme-ates the entire development process, testing and verification of the
programs themselves continues to be a topic of research. In Section 5.6, we
discussed techniques for verifying the correctness of algorithms in a
mathematically rigorous manner but concluded that most software today is
“verified” by means of testing. Unfortunately, such testing is inexact at best.
We cannot guarantee that a piece of software is correct via testing unless we
run enough tests to exhaust all possible scenarios. But, even in simple
programs, there may be billions of different paths that could potentially be
traversed. Thus, testing all possible paths within a complex program is an
impossible task.
On the other hand, software engineers have developed testing method-
ologies that improve the odds of revealing errors in software with a limited
number of tests. One of these is based on the observation that errors in
software tend to be clumped. That is, experience has shown that a small
number of modules within a large software system tend to be more
problematic than the rest. Thus, by identifying these modules and testing
them more thoroughly, more of the system’s errors can be discovered than if
all modules were tested in a uniform, less-thorough manner. This is an
instance of the proposition known as the Pareto principle, in reference to the
economist and sociologist Vilfredo Pareto (1848–1923), who observed that a
small part of Italy’s population controlled most of Italy’s wealth. In the field of
software engineering, the Pareto principle states that results can often be
increased most rapidly by applying efforts in a concentrated area.
Another software testing methodology, called basis path testing, is to
develop a set of test data that insures that each instruction in the software is
executed at least once. Techniques using an area of mathematics known as
graph theory have been developed for identifying such sets of test data.
Thus, although it may be impossible to insure that every path through a
software system is tested, it is possible to insure that every statement within
the system is executed at least once during the testing process.
Techniques based on the Pareto principle and basis path testing rely on
knowledge of the internal composition of the software being tested. They
therefore fall within the category called glass-box testing - meaning that the
software tester is aware of the interior structure of the software and uses this
knowledge when designing the test. In contrast is the category called black-
box testing, which refers to tests that do not rely on knowledge of the
software’s interior composition. In short, black-box testing is performed from
the user’s point of view. In black-box testing, one is not concerned with how
the software goes about its task but merely with whether the software
performs correctly in terms of accuracy and timeliness.
An example of black-box testing is the technique, called boundary value
analysis, that consists of identifying ranges of data, called equivalence
classes, over which the software should perform in a similar manner and then
testing the software on data close to the edge of those ranges. For example,
if the software is supposed to accept input values within a specified range,
then the software would be tested at the lowest and highest values in that
range, or if the software is supposed to coordinate multiple activities, then the
software would be tested on the largest possible collection of activities. The
underlying theory is that by identifying equivalence classes, the number of
test cases can be minimized because correct operation for a few examples
within an equivalence class tends to validate the software for the entire class.
Moreover, the best chance of identifying an error within a class is to use data
at the class edges.
Another methodology that falls within the black-box category is beta
testing in which a preliminary version of the software is given to a segment of
the intended audience with the goal of learning how the software performs in
real-life situations before the final version of the product is solidified and
released to the market. (Similar testing performed at the developer’s site is
called alpha testing.) The advantages of beta testing extend far beyond the
traditional discovery of errors. General customer feedback (both positive and
negative) is obtained that may assist in refining market strategies. Moreover,
early distribution of beta software assists other software developers in
designing compatible products. For example, in the case of a new operating
system for the desktop computer market, the distribution of a beta version
encourages the development of compatible utility software so that the final
operating system ultimately appears on store shelves surrounded by
companion products. Moreover, the existence of beta testing can generate a
feeling of anticipation within the marketplace - an atmosphere that increases
publicity and sales.

7.6 Questions & Exercises

1. What is the role of the SQA group within a software development
organization?
2. In what ways does human nature work against quality assurance?
3. Identify two themes that are applied throughout the development process
to enhance quality.
4. When testing software, is a successful test one that does or does not find
errors?
5. What techniques would you propose using to identify the modules within a
system that should receive more thorough testing than others?
6. What would be a good test to perform on a software package that was
designed to sort a list of no more than 100 entries?