An industrial Engineering Approach to Software

Development

D. N. Card
Computer Sciences Corporation, Silver Spring, Maryland

R. A. Berg
Synercom, Inc., Houston, Texas

Many different tools and techniques have been developed Increasingly, researchers and practitioners (especially
to increase software quality and productivity. However, managers) realize that improving software quality and
periodic acquisition of improved methods and tools, by productivity requires viewing software development as
itself, does not ensure continual improvement. To be an integrated process rather than a set of independent
effective, new technology must be integrated into an activities. (Reference 1 provides one example of this.)
underlying process. That process must be managed explic-
This paper explores one such process view of software
itly. This paper describes an industrial engineering approach
development. It makes three points: (1) technology
that treats software development as a process distinct from
its unique application to any specific project. Its essential
acquisition by itself does not necessarily lead to produc-
elements include formal process definition, software mea- tivity or quality improvement; instead a process ap-
surement, process engineering, and quality control. Al- proach is needed; (2) the software factory is not
though already successfully embedded in many manufactur- equivalent to the software process; and (3) industrial
ing processes, application of industrial engineering engineering provides appropriate techniques for making
techniques to software remains a novelty. Nevertheless, process improvement a regular part of conducting
this approach provides the software enterprise with a long- business. Moreover, this paper argues that quality
term plan for improving software quality and productivity. control must precede real productivity improvement.

1. INTRODUCTION 2. TECHNOLOGY ACQUISITION AND THE

SOFI-WARE PROCESS
Increased competition within the computer business, as
well as the spread of computer applications throughout Over the past few years, the stock of software engineer-
industry and society, has led to a heightened concern for ing tools and methods has increased dramatically.
software quality and productivity. The assessment pro- Although acquisition of technology innovations (new
cedure recently drafted by the Software Engineering tools and methods) is a basic strategy for process
Institute [l] potentially provides a special incentive for improvement, by itself it does not guarantee success. A
defense contractors to improve software-development specific innovation may or may not be beneficial in a
practice. Unfortunately, the software managers’ concern given context. Although technologies are readily availa-
usually encompasses only a single project at a time. ble, evaluative data is not.
Lessons learned are not retained because software Most practitioners lack the information they need to
enterprises lack a high-level conceptual and management make good choices among the products offered. By their
approach to improving software quality and productiv- own informal reckoning, practitioners have had only
ity. limited success in employing these innovations [2]. At
the same time, few rigorous evaluations of the benefits
of new software technology [3] have been performed by
Address correspondence to D. N. Card, Computer Sciences
researchers. Throughout recent years, overall software
Corporation, 8728 Colesville Road, Silver Spring, MD 20910. productivity has only advanced about 3% per year [4].

The Journal of Systems and Software 10, 159-168 (1989)

159
@ 1989 Elsevier Science Publishing Co.. 1”~.
0164-1212/89/$3.50
160 D. N. Card and R. A. Berg

The complexity of the software-development process self. Figure 1 illustrates this relationship. The process
contributes to the difficulty of evaluating technology [5]. engineering function evaluates and integrates new tech-
Thus, the practitioner often opts to search for some nology to improve productivity and quality.
single dramatic innovation (tool or method) that will
improve performance. Generally, each tool or method
targets only one software-development activity, such as 3. THE SOFIWARE FACTORY AND THE
design or code. Improved quality or productivity in that SOFTWARE PROCESS
one activity may be paid for in lower quality or Software development shares many (but not all) traits
productivity elsewhere. Software costs and errors are with manufacturing. The notion of a software factory is
distributed among many different activities and partici- often confused with the more useful concepts of a
pants. As Brooks [6] points out, no silver bullet (single software process and industrial engineering. Manley [7]
technology) can slay them all in a single shot. Thus, defines the software factory to include methods, tools,
achieving an order-of-magnitude improvement requires facilities, and a base of reusable software. Any place
a systematic approach that incorporates smaller im- where software is built can be regarded as a “factory.”
provements on a continuing basis. What then distinguishes the software factory from the
Developing such an approach requires distinguishing ordinary software shop? In his survey of Japanese
the underlying production process from day-to-day software factories, McNamara [8] identifies six common
production activities. Software development must be characteristics:
viewed as a general production process distinct from its
1. Strategic corporate enterprise.
application to any specific project. This process is not
2. Separate division producing software for various
just an abstract set of transformations, but also a tangible
uses.
production system consisting of people, tools, and
3. Organized like a manufacturing facility.
methods. As such, it can be measured, controlled, and
4. Precise and consistent methodology for all.
improved. Although software engineering is the applica-
5. Automated and integrated software support sys-
tion of this software-development process within a
tems.
project to produce a specific software product, software
6. More than 1000 employees.
process engineering deals with the specification and
improvement of the software-development process it- The productivity and quality achievements of Japanese
software factories [8] have lured some software enter-
Figure 1. An industrial engineering approach. prises into copying these superficial characteristics.

SOFTWARE PROCESS

SOFTWARE
PRODUCTS
WORK ORDERS
I

TECHNOLOGY

ENGINEERING
An Industrial Engineering Approach to Software 161

SOFWARE FACTORY

REWORK

WORK- SOFTWARE
ORDERS PRODUCTS

REUSE

PERFOlM4NCE MEASURES

NEW sl
TECHNOLOGY

However, adopting a software factory organization Figure 2. The software process inside the software factory.
without understanding its underlying principles leads to
technology accumulation instead of process engineering. 4. INDUSTRIAL ENGINEERING AND PROCESS
The software factory provides only the infrastructure for IMPROVEMENT
software development. The real issue is not how to This section describes one strategy for improving the
organize software development like a factory, but how software process, an industrial engineering approach
to make the underlying process operate efficiently. based on concepts already applied successfully to
Applying industrial engineering to the software proc- manufacturing processes [l l] (as recommended by
ess may lead to a factory-like organization, but “build- Agresti [ 121). Determined application of this approach
ing” a software factory does not necessarily imply a leads to gradual but steady improvements in software
systematic approach to productivity and quality im- quality and productivity, as demonstrated by actual
provement. Tajima and Matsubara [9] described the experience [ 131.
evolution of one Japanese software factory (Hitachi). Industrial engineering is a discipline that relates to the
Although it satisfies McNamara’s six criteria, the design and improvement of complex production systems
elements stressed by the insiders were process definition [ 111. Adoption of the industrial engineering concepts of
and quality assurance. Figure 2 shows that these statistical process control [14] and evolutionary opera-
activities are prerequisites to process engineering. In- tion [15] leads to a three-part process improvement
stead, many external observers have focused on automa- strategy: (1) define the process and measure its perform-
tion and “quality circles” to explain the productivity ance, (2) establish process control in terms of quality by
and quality increases. A quick look at Figure 3 shows identifying and correcting “malfunctioning” production
that major elements of the Hitachi process are not elements, and (3) increase productivity and quality by
automated. improving the production activity itself by introducing
The Hitachi factory began with wholehearted accept- innovations. Figure 2 shows how these process-oriented
ance of Crosby’s thesis that “quality is free’ [lo]; it is activities fit into the software factory.
lack of quality that costs. Quality circles are only a small
(and specially Japanese) part of that initial emphasis. At
4.1 Step 1: Process Definition
Hitachi, productivity improvement followed process
definition and quality control [9]. These are the steps of The first step toward process engineering must be the
the industrial engineering approach described in the development (or acquisition) and implementation of a
following section. formal process definition. An organization’s software
 D. N. Card and R. A. Berg

P
An Industrial Engineering Approach to Software 163

engineering methodology largely defines its software- correcting malfunctioning production elements before a
development process. However, the process definition stream of defective products can be produced increases
also must specify the roles of people, tools, and the productivity of the entire process by reducing the
facilities. To be effective, tools must support a method. effort required for rework and product testing. Rework
Thus, the environment both reflects and reinforces and testing can account for more than 50% of the total
methodology [4]. Of course, the process, methods, cost of a software project [19]. (Although software
to&, and environment change to achieve software development does not produce a stream of identical
quality and pr~uctivi~ improvement. Nevertheless, an parts, it does produce a stream of comparable software
initial devotion of the souse-development process units, builds, and projects.)
must be provided to establish a starting point for process A formal product-assurance organ~ation provides a
improvement. mechanism for achieving in-process quality control by
As an example, Figure 3 illustrates the SKIPS II ensuring that developers adhere to the prescribed proc-
software-development process used by Hitachi [9]. Note ess and identify process malfunctions so that they can be
the presence of many nonautomated activities in the plan corrected. In particular, early life-cycle reviews and
of this “software factory.” Attaching measures of cost, inspections facilitate the early detection of errors [ 181.
quality, and product (volume) to each process element Because the cost of correction increases the longer an
produces a process performance baseline. Measure- error remains in the software [19], earlier error detec-
ment provides visibility into the process and helps to tion and correction increases life-cycle productivity.
identify leverage points for improvement. Leverage
points are the cost-intensive and error-prone activities. Statistical process control. Capping the results of
Because an undefined process varies unpredic~bly in-process quality assurance and testing activities en-
from project to project, control and improvement of it ables the application of statistical process control (SPC)
are nearly impossible. Furthermore, that variation limits techniques [ 141. SPC provides a quantitative technique
the opportunities for specification, design, code, and test for verifying that a process performs as intended. Some
reuse. Prescribing a software engineering methodology performance variation always occurs. However, track-
does not lead to an assembly line mentality, but it does ing quality measures with SPC during development
promote order in the workplace. An effective methodol- enables a determination of whether or not the process is
ogy offers well-defined alternatives reflecting varying “in statistical control’ ’ (stable).
project needs. Rather than restraining the creativity of One basic SPC technique, the control chart [14], can
software engineers, prescribing an overall methodology be demonstrated easily. Adopting an incremental (i.e.,
frees them from concerns about process issues. It allows build/rel~se~~ented [ZO]) development strategy for
them to focus on the customer’s technical problem rather large projects allows testing data to be tracked on control
than process details, thus concentrating their energies on charts. Figure 4 plots quality measures from two large
achieving the best technical solution to the customer’s projects (about 0.8 and 0.9 million SLOCs, respectively)
problem. developed by Computer Sciences Corporation for
NASA. The total error rate shown (errors per thousand
lines of code delivered) includes all noncosmetic errors
4.2 Step 2: Quality Control reported during system testing, acceptance testing, and
Quality control means comparing product quality with operation. It measures the quality of the software-
predetermined quality objectives, and then taking cor- development activity. The delivered error rate includes
rective action if the quality objectives are not achieved. noncosmetic errors found during acceptance testing and
Corrections may be needed to either the process or the operations. It measures the quality of the software as
product. Quality control must be achieved before proc- perceived by the customer. Testing efficiency is the
ess improvement can begin [16]. Inserting new technol- percentage of all errors that were found during system
ogy into an unstable process will only complicate testing. It measures the quality of the software-testing
matters. The techniques of quality control include activity.
quality assurance, statistical process control, and statisti- The plots included an “upper control limit’ ’ [ 141 for
cal testing/reliability certification. the total error rate for release 7 of each project.
Exceeding this control limit indicates that the software-
Quality assurance. Conventional quality-assurance development process has deteriorated during the current
strategies rely on extensive testing of finished products release. Control limits are usually set three standard
before delivery. The alternative, in-process quality deviations from the expected error rate. Historical data
assurance, incorporates process audits [17] and inspec- or past performance on the same project can be used to
tions of inte~~iate products [ 181. Identi~ing and establish the expected error rate. A simple formula
164 D. N. Card and R. A. Berg

relates the standard deviation to proportions [ 141 such as testing hold true. Testing and error correction continue
error rate and testing efficiency. until the predicted reliability reaches a specified level
Figure 4a shows a project with a stable process. It is v31.
repeating good performance from release to release. Software developers must still perform selective
This project demonstrates that statistical process control, testing during early testing phases and for critical
such as that obtained in manufacturing processes, can be functions, but only statistical testing provides the data
achieved in software development through in-process necessary for reliability certification [2 11. Fortunately,
quality-assurance activities performed by a software this approach can be easily automated because, in its
product-assurance program. Figure 4b shows a project simplest form, it consists of random sampling over the
that clearly has exceeded its control limit. That project program’s input domain. However, it does require
has gone “out of statistical control” [14] and needs substantial work during requirements and design to
corrective action. SPC techniques also can be applied to define the input domain. Statistical testing (as end-
the results of quality-assurance activities such as design product inspection [24]) is the final element of quality
inspections (as suggested by Fagan [ 181). control.

Statistical testing. Applied during final product test-

ing, statistical testing techniques provide a basis for 4.3 Step 3: Process Engineering
determining software reliability. Unlike the selective Once process control (stability) has been achieved,
methods of structural and functional testing, statistical process improvement can begin. The steps to process
testing relies on random samples of projected user improvement are as follows: (1) analyze the process
operations as test cases [2 11. Many different mathemati- baseline, (2) innovate and evaluate, and (3) transfer and
cal models relate the observed times between failures to integrate improvements. With an appropriate measure-
subsequent software reliability [22]. Generally, these ment program and a controlled process, specific im-
models apply only when the conditions of statistical provement needs and the effects of any technology
innovation can be clearly observed. Beam et al. [25]
Figure 4. Control charts for software projects 1131(@I987 itemize potential improvement areas in the software-
Butterworth Scientific). production system. Continual innovation, evaluation,

f3.-Ml TESTING EFFICIENCY

Oz?~~ TOTAL ERROR RATE
ti DELIVERED ERROR RATE

100

80

1 2 3 4 5 6 7 1 2 3 4 5 6 7
PROJECT D RELEASE PROJECT 2 RELEASE

a. PROJECT IN STATISTICAL CONTROL b. PROJECT OUT OF STATISTICAL CONTROL

An Industrial Engineering Approach to Software 165

100

75

50

%,, EVOP PROCESS ADJUSTMENT

.?e
*+k+,,,,
*%#,,,
25 I * RUN4 -+z+-,,‘,
l RUM3
I’/r,,,,
-~I,,,-~ ,,,, fl&

0 I I I 1
0 25 50 75 100

TECHNOLOGY B: O/o
APPLICATION

and transfer results in an orderly evolution of the Figure 5. Design of EVOP experiment.
software process.
the information necessary for its own improvement. As a
Process baseline. The initial process definition es-
practical matter, the performance of a complex system
tablishes a baseline for subsequent process improve-
can be optimized only during actual operation. Results
ment. The baseline associates cost and quality measures
from laboratory experiments, even in such well-defined
with each process element (see Figure 2 for an example),
areas as chemical processes, often do not scale up in the
The following questions should be asked about each
production system [ 151.
process element [12, 117-1181:
EVOP proceeds by varying process parameters (or
1. Is this activity necessary or can it be eliminated? condition) in a systematic way, then measuring the
2. Can this activity be combined with another or effects on process output. Figure 5 shows the design of a
others? sample EVOP experiment. Four production runs are
3. Is this the proper sequence of activities or should the made with process parameters set off from the labora-
sequence be changed? tory optimum. The experimenter discovers that the
4. Can this activity be improved? conditions for maximum yield differ from the laboratory
5. Is this the proper person to be doing this activity? to the pr~uction enviro~ent. The numerical results
from the four runs define the appropriate adjustment to
Those process elements requiring the most effort or
process parameters. The “multiproject variation” [ZS]
those that are the most prone to error should be targeted
technique employed in the Software Engineering Labo-
for improvement first (through increased automation,
ratory* @EL) 1271 approximates the EVOP approach.
for example). Leverage points for process improvement
One such SEL study [SJ examined process parameters
can be identified using methods such as value chain
such as percent of code read and percent of structured
analysis, Paretocharts, ~dI~~wa~ag~s [12]. Ina
code. This type of analysis supports engineering cost/
recent value chain analysis, Boehm [26] identified the
quality trade-offs in a way that laboratory experiments
most immediately effective methods of increasing soft-
camlot.
ware productivity to be of more reuse, improved tool
Alternatively, once a process is in statistical control,
support, and reduced rework.
the effects of a single i~ovation can be readily observed
in a control chart. Figure 6 illustrates the begun
Technology evaluation. Not all proposed improve-
of a process to a new performance level due to
ments turn out to be successful. The concept of
evolutionary operation (EVOP), due to Box [ 151,
provides a model for technology evaluation in a produc-
* The Software Engineering Laboratory is a NASA/Goddard Space
tion environment. The principle of EVOP is that every Flight Center research project supported by Computer Sciences
process should, in addition to its product, also. provide Corporation and the University of Maryland.
166 D. N. Card and R. A. Berg

Figure 6. Technology innovation example. 5. PRACTICAL EXPERIENCE

Computer Sciences Corporation (CSC) has developed a
technology improvement. Sayward [29] recommends an six-point approach to improving software productivity
introduction-withdrawal-introduction cycle to ensure and quality that incorporates industrial engineering
that the effects of an innovation are not confused with concepts. These points are modeled on Deming’s 14
other factors. This activity is easier to do in a project- points for manufacturing enterprises [16]. CSC’s six
oriented environment than on an assembly line. Often, points [13] are as follows:
management reluctance to embrace a new technology
totally leads to its incremental introduction. 1. Management commitment;
Another factor to be considered is the cost of a new 2. Process formalization;
technology versus its benefit. Repeated trials of a new 3. Objective measurement;
technology lead to a more accurate assessment of its 4. In-process quality control;
benefit. Then, other statistical methods (e.g., [30]) 5. Continual process improvement; and
support the decision to adopt or reject new technology 6. Employee participation.
on a cost/benefit basis.
This approach has evolved gradually. CSC established a
formal software product-assurance program in 1975 that
Transfer and integration. Without special incen-
almost immediately produced quality improvements
tives or support mechanisms, new software technology
[32]. That program complements CSC’s internally
takes 15-20 years to get into general practice [31].
developed life-cycle methodology (DSDMt). DSDM
Obviously, the organization that can shorten this period
integrates project control and product-assurance activi-
will have a competitive edge. Management and em-
ties into conventional structured analysis, design, pro-
ployee willingness to change the way of doing business
gramming, and testing methods [20]. Periodic updates to
is a key factor in successful technology transfer.
DSDM integrate increased knowledge and improved
However, the introduction of any new technology
technology acquired by CSC. A Tools Information
should be planned to facilitate acceptance. Its role in the
Center distributes evaluative data to prospective tool
software-development process must be clearly defined “buyers” within the Corporation.
and integrated with other tools and methods. Appropri-
During me past 5 years, this program has led to
ate training must be provided. New technology must be
significant increases in measured software quality and
compatible with technology already incorporated in the
productivity at CSC [ 131, even though the industrial
process. New tools must support established methods.
Bridges between tools for different activities must be
developed. Musa [4] identifies many other consider- t DSDM (Digital System Development Methodology) is a regis-
ations for introducing new technology. tered trademark of CSC.
An Industrial Engineering Approach to Software 167

engineering approach has not yet been fully imple- Technology acquisition is only one step (a later step)
mented throughout the corporation. During the 5-year toward process improvement (although the one step
period, these process improvements were observed in most organizations take first). New technology must fit
participating projects: into an appropriate process context to be used effec-
tively. Every software enterprise must assume responsi-
1. Error rates declined by 50%
bility for improving software quality and productivity in
2. Twenty percent of cost was saved through reducing
its own shop. This requires a conscious and coordinated
rework.
effort. The industrial engineering approach just de-
3. Twenty percent of cost was saved through increased
scribed provides one method for achieving this improve-
automation.
ment.
4. Software reuse reached 40%

Mital et al. [33] present a case study of specific process ACKNOWLEDGMENT

improvement efforts for a recent CSC project. The authors would like to thank the reviewers Bruce Blum of Johns
Hopkins University and David Kitson of the Software Engineering
Institute, for their very helpful comments.
6. CONCLUSIONS
The preceding discussion demonstrated that industrial REFERENCES
engineering provides a comprehensive approach to 1. W. S. Humphrey, W. L. Sweet, et al., A Method for
managing and improving software quality and produc- Assessing the Software Engineering Capabilities of Con-
tivity . As highlighted by Brooks [6], no single technol- tractors, Software Engineering Institute, Sept. 1987.
ogy can overcome all of the obstacles faced by software 2. J. Beeler, Programmer Productivity: Never Have So

developers. Success comes from a systematic approach Many Done So Little for So Much, Computerworld 40-
to software development and process improvement, 46, Dec. 30, 1986.
3. B. A. Scheil, The Psychological Study of Programming,
rather than from any specific technology. The software-
ACM Comput. Surv. 101-120 (1981).
development process itself, not just individual projects,
4. J. D. Musa (ed.), Stimulating Software Engineering
must be explicitly managed. Technology acquisition is Progress-A Report of the Software Engineering Planning
only part of that management activity. Group, ACM Software Engineering Notes 29-48,
Process improvement can be accomplished only by (1983).
providing it as an independent centralized function. 5. D. N. Card, F. E. McGarry, and G. T. Page, Evaluating
Mostproject managers are too busy to keep up with the Software Engineering Technologies, IEEE Trans. Soft-
rapid rate of technology change in software. They often ware Eng., (1987).
do not know what is available, much less how effective it 6. F. P. Brooks, No Silver Bullet-Essence and Accidents of
is, or where to fit it into the process. Moreover, Deming Software Engineering, Proceedings, IFIP Tenth World
[ 161observes that while each of us may know everything Computer Congress, Sept. 1986, 1069-1077.
7. J. H. Manley, Computer Aided Software Engineering
about doing our job, we usually do not know how to
(CASE) Foundation for Software Factories, Proceedings,
improve it in essential ways.
IEEE Computers Conference, Sept. 1984, 84-91.
Software process engineering constitutes a technical 8. D. McNamara, Quality as a Management Strategy in
discipline separate from software engineering. Just as Japan and the United States, Presented at the NSIA Second
the engineer who designs a new car does not design the Conference on Software Quality and Productivity, March
machines that build it, software-development organiza- 1986.
tions must distinguish between the roles of the process 9. D. Tajima and T. Matsubara, Inside the Japanese Soft-
engineer and the software engineer. Agresti [12] de- ware Industry, IEEE Computer 34-43, (1984).
scribes the new role in detail. 10. P. B. Crosby, Quality Is Free. New York: McGraw-Hill,
The results of applying this approach support the 1979.
contention of Deming [16], Mills 1341, and others that 11. W. C. Turner, J. H. Mize, and K. E. Case, Introduction
to Industrial and Systems Engineering. Englewood
organizations initially must focus on improving quality
Cliffs, New Jersey: Prentice-Hall, 1978.
to achieve productivity gains. Moreover, a study [5]
12. W. W. Agresti, Applying Industrial Engineering to the
recently published by SEL showed that quality was the Software Development Process, Proceedings, IEEE
factor most strongly associated with productivity (after Computer Society Twenty-Third International Confer-
individual programmer performance). Unfortunately, ence, Sept. 1981, 264-270.
most organizations have focused on productivity im- 13. D. N. Card, T. L. Clark, and R. A. Berg, Improving
provement though automation, without the prerequisite Software Quality and Productivity, Information and
concern for quality control and improvement. Software Technology 235-241, (1987).
 D. N. Card and R. A. Ihg

14. T. Pyzdek, An SPC Primer. Quality America, Tucson, Costs, Proceeding, IFIP Tenth World Computer Con-
Arizona, 1978. gress, Sept. 1986, 703-714.
15. G. E. Box, Evolutionary Operation: A Method for 27. D. N. Card, A Software Technology Evaluation Program,
Increasing Industrial Productivity, Applied Statistics Information and Software Technology 291-300 (1987).
6(2), 81-101 (1957). 28. V. R. Basili, R. W. Selby, and D. H. Hutchens,
16. W. E. Deming, Quality, Productivity, and Competitive Experimentation in Software Engineering, IEEE Trans.
Position. MIT Press, Cambridge, Massachusetts, 1982. Software Eng. 733-743 (1986).
17. S. G. Crawford and M. H. Fallah, Software Development 29. F. G. Sayward, Design of Software Experiments, in
Process Audits-A General Procedure, Proceedings, Software Metrics: An Analysis and Evaluation,
IEEE Eighth International Conference on Software (A, Perlis et al. eds.), MIT Press, Cambridge, Massachu-
Engineering, Aug. 1985, 137-141. setts, 1981, 44-57.
18. M. E. Fagan, Advances in Software Inspections, IEEE 30. K. F. McCardle, Information Acquisition and the Adop-
Trans. Software Eng. 744-751 (1986). tion of New Technology, Management Science 1372-
19. B. W. Boehm, Software Engineering R&D Trends and 1389 (1985).
Defense Needs, in Research Directions in Software 31. S. T. Redwine and W. E. Riddle, Software Technology
Technology, MIT Press, 1979, Maturation, Proceedings, IEEE Eighth International
20. S. Steppel, T. L. Clark, et al., Digital System Develop- Conference on Software Engineering, Aug. 1985, 189-
ment Methodology, Computer Sciences Corporation, 200.
Falls Church: March 1984. 32. P. C. Belford and C. Broglio, A Quantitative Evaluation
21. P. A. Currit, M. Dyer, and H. D. Mills, Certifying the of the Effectiveness of Quality Assurance as Experienced
Reliability of Software, IEEE Trans. Software Eng. 3- on a Large-Scale Software Development Effort, Proceed-
11 (1986). ings, ACM Software Quality Assurance Workshop,
22. P. Mellor, Software Reliability Modeling: The State of the Nov. 1978.
Art, Information and Software Technology 81-98 33. R. M. Mital, M. M. Kim, and R. A. Berg, A Case Study
(1987). of Workstation Usage During the Early Phases of the
23. J. P. Cavano, Toward High Confidence Software, IEEE Software Development Life Cycle, Proceedings, ACM
Trans. Software Eng. 1449-1455 (1985). Second Conference on Practical Software Develop-
24. S. B. Vardeman, The Legitimate Role of Inspection in ment Environments, Dec. 1986, 70-76.
Modem SQC, American Statistician 325-328 (1986). 34. H. D. Mills, How to Make Exceptional Performance
25. W. R. Beam, J. D. Palmer, and A. P. Sage, Systems Dependable and Manageable in Software Engineering,
Engineering for Software Productivity, IEEE Transac- Proceedings, IEEE Fourth International Computer
tions on Systems, Man, and Cybernetics, March/April Software and Applications Conference, Oct. 1980, 19-
1987, 163-186. 23.
26. B. W. Boehm, Understanding and Controlling Software