IEEE TRANSACTIONS ON RELIABILITY, VOL. ,47, NO.

3, 1998 SEPTEMBER

Carol Smidts
University of Maryland, College Park
Martin Stutzke
Science Applications International Corp, Prague
obert W. Stoddard
Texas Instruments Digital Imaging, Plano

Key Words - Software, Software reliability prediction, & Bastani [4].Within these models, there are two main
Early phases of software development, Requirements, Failure categories :
modes, Bayes. 1. ERPM typically address the reliability of the software
early in the life-cycle at the requirements, preliminary de-
Summary & Conclusions - Models for predicting soft- sign, detailed design, or coding level. Predictive models
ware reliability in the early phases of development are of para- are used to assess the risk of developing software for a
mount importance since they provide early identification of cost given customer under a given set of requirements within
overruns, software development process issues, optimal devel- given resources (staff, budget , schedule, and development
opment strategies, etc. A few models geared towards early
environment); ze, before the project truly starts.
reliability prediction, applicable to well defined domains, have
been developed during the 1990's. However, many questions 2. E-BREM evaluate current and future software relia-
related to early prediction are still open, and more research bility from failure data gathered, beginning with the inte-
in this area is needed, particularly for developing a generic gration testing of the software. In the category of estima-
approach to early reliability prediction. tion models, there are: reliability growth models [l], input
This paper presents an approach to predicting software reli- domain models [5],and fault seeding models [6].
ability based on a systematic identification of software process While there are numerous E-BREM, there are few
failure modes and their likelihoods.' A direct consequence of the ERPM. Early prediction models, however, are of para-
approach and its supporting data collection efforts i s the iden- mount importance, because they allow early identification
tification of weak areas in the software development process. of: risk of cost overruns, SDP issues, optimal development
A Bayes framework for the quantification of software process strategies, etc. Existing ERPM have been developed dur-
failure mode probabilities can be useful since it allows use of ing the 1990's. These models include the:
historical data that are only partially relevant to the software . phase-based model developed by Gaffney & Davis [7, 81,
at hand. The key characteristics of the approach should apply . model developed by Agresti & Evanco [9] to predict
to other software-development life-cycles & phases. However, defect density based on product & process characteristics
it is unclear how difficult the implementation of the approach for Ada development efforts,
would be, and how accurate the predictions would be. Further e US AirForce Rome Laboratory work [lo, 111.
research will help mswer these questions.
However, many questions related to early prediction are
1. INTRODUCTION still unanswered, and more research in this area is needed,
in particular when it comes to:
Acronyms1 . developing an approach (not a model) to early reliabil-
ERPM early reliability prediction model ity prediction, applicable to a large variety of development
E-BREM execution-based reliability-estimation model environments,
NHPP non-homogeneous Poisson process . a more thorough understanding of the failure mecha-
SDP software development process nisms at play.
SRM software-reliability model
This paper proposes an approach to early software relia-
Many SRM have been developed. For a detailed de- bility prediction, based on the systematic identification of
the failure modes of the SDP. A method of quantification
scription of most of these models, see Musa [I]or Lyu [2];
adapted to this approach is identified. Issues related to im-
for a general classification, see Goel [3] or R.amamoorthy
plementation of such methodology are discussed. The ap-
'The singular & plural of an acronym are always spelled the same. proach is illustrated on the requirements-analysis phase for

0018-9529/98/$10 00 01998 IEEE

SMIDTS ET A L S0I;TWARE RELIABILITY MODELING: AN APPROACH TO EARLY RELIABILITY PREDICTION 269

a waterfull life-cycle process, It is our belief that the key by Royce in 1970. Its life-cycle phases include: require-
characteristics of this approach should apply to other soft- ments, design, code implementation, unit test, integration
ware life-cycles and to other software development phases. test, system test, and acceptance. A major characteristic
However, it is unclear how difficult the implementation of is that a phase can only begin when the previous phase
the approach would be, and how accurate the prediction is completed. The software waterfall modell is still widely
would be. used primarily for highly computational software.
Beforje describing the approach itself, it is necessary to 2 . Failure-mode detection is NHPP2. Although this as-
clarify briefly and contrast two concepts used in the failure- sumption might be too restrictive, NHPP is a reasonable
mode based modeling of software, first approximation to the process failure mode detection
. process failure modes, process.
. product failure modes,
because this paper uses this terminology. 2 . REQUIREMENTS-BASED
Process failure modes are failures to carry the SDP steps SOFTWARE RELIABILITY DEFINITION
correctly. Examples of such failures modes are in section 5 , The definition of software reliability in this paper rests
for the requirements analysis phase. For instance ‘the ana- upon the concept of requirements. There has been, and
lyst omitting a functional requirement’ is a process failure continues to be, a debate on the definition of software reli-
mode. Such failures might or might not lead to software ability. This debate arises from the fact that several soft-
(product) failure. ware reliability definitions can coexist a t the same time,
Product (software) failure modes are failures of the soft- depending on the parties involved. The confusion relates
ware itself, eg: failure of the reactor protection system directly to the interpretation of the term faalure by the
software to scram the reactor on demand. parties involved. For a software development company,
Ref 112 - 211 provide examples of failure mode classifica- failure to implement any requirement can lead to lawsuits
tion schemes for the process or the product. Our approach and other discomforting outcomes; hence, for such a com-
is based on the:
pany,
. systematic identification of such process failure modes, R ~= D Pr{all requirements will be met}.
. modeling of impact of process failure modes on the The user, on the other hand, might attach more impor-
tance to some categories of requirements, eg, functional
requirements are more important than portability require-
developer’s definition of reliability: ments. The user’s definition is clearly focused on the mis-
Pr{all requirements are met} sion to be achieved, and
user’s definition of reliability: RU = Pr{the software achieves its mission}.
Pr{a software achieves its mission} The question that arises immediately is whether these def-
total number of requirements initions are mutually contradictory. A relationship must
number of operational modes [22] exist between the two definitions:
operational mode z
requirement 3 DPr{Rqp n . . . n Rq;}
R ~= (1)
Rq, in OM, N
the set: Rq,,, : J = 1 , 2 , . . . , n RU = Pr{OMF}. Pr{OMPDM} (2)
logical combination (structure function) 2=1
of Rq,,3 which must be met to fulfill OM,
criticality of Rq,; 0 5 C, 5 1 Rqy = event: Rq, is Met;
failure-mode IC of Rq, OMPDM= event: this OM, is Required During the
criticality of Rq, in OM,; 0 5 C2,,5 1
Mission;
the set: C,, : j = 1 , 2 , . . . ,n
Pr{software is unreliable in OM,} OM: E event: this OM, can be performed correctly.
set of influencing factors The Pr{OMFDM}can be obtained directly from the soft-
time-dependent failure intensity function ware operational profile [22].
of Fk(Rq,) under E
number of requirement failures, by time t GI)
Pr{OMf} = PI-{@,[Rs,, (3)
mean number of failures, by time t
start of the operational phase The C, allows the importance of user needs/desires to be
specified mission time quantified.
P r ( ? ; . ~ , t * ; j , k , ~Pr{Fk(Rq,)
-) fails during 7,) N

Assumptions Ru = Pr{@, [Rq,,C , ] }‘ Pr{OMrDM} (4)

2=1
1. Software development follows a waterfall life cy-
cle. The software waterfall development was introduced ‘Appendix A . l describes the NHPP.
270 l t t t I K A N > A C 1 [UN> U N KtLIABILI I Y , VVL. 41, NU. 3 , LYYU > I 9 1 J2MtltK

R ~ isDan instance of RU that involves 1 operational mode or mistake4 in applying a process, thus creating an er-
containing all requirements: ror within the originating software development life-cycle
phase. If this error survives into the actual software code,
N = 1, 2 = I, 121 = n, c,,J = a. it is called a software fault. If the fault manifests itself
0, [RCI,,~,c z , g : 3 = 1,.. . , 121 Rqi,, n . . n Rqi,, (5) during testing or use, it is called a software failure. In this
paper, SDP failure modes axe defined to be the composite
RU is identical t o the IEEE definition of software run re- of human slips and mistakes:
liability [5, 231, as long as all requirements (including the occurring
# within each defined step of the SDP that cre-
implicit requirements) are addressed in the definition of ate software errors,
Ru.To estimate the sum in (4),a Monte Carlo algorithm related to not identifying & correcting software errors,
that samples from the user profile, could be used. This es- faults, and failures.
timate would be equivalent to running the software under SDP failure modes can be detected in multiple ways:
various operational modes, each such mode being repre- software & process inspection (including formal design
sented by OM,. A logical construct, 4, with values of 1 reviews and code walk-throughs),
or 0 would also be required to represent the successful or . dynamic testing and formal reasoning.
unsuccessful run of the program. 4 is identical to the 0, In the early phases of SDP (requirements definition &
of each OM,. The reliability estimate, equivalent to the analysis, preliminary design, detailed design, and coding),
IEEE run reliability estimate, would be: inspection is often the only way in which SDP failure
RIEEE= (number of successes)/(number of runs) - modes can be detected (since code has not been developed
which is identical to (4). to the point where dynamic testing is possible). In the
later phases, inspections continue; however, dynamic test-
If Pr(0,) is time dependent, one obtains the conventional ing becomes the dominant means of detecting SDP failure
definition of software reliability in [l]. modes.
The same requirements can appear in one or more dif-
ferent OM,. Additionally, there are explicit & implicit 4. OVERVIEW OF THE APPROACH
requirements. Implicit requirements must be foreseen by
The approach in this paper is inherently rooted in risk5
the development group and then modeled explicitly.
assessment philosophy. Any risk assessment approach at-
3. SOFTWARE FAULT GENERATION tempts to answer the 3 fundamental questions, adapted
here for SDP:
Software faults, which can become software failures, are 1. Why and how does software fail, ie, what are the
generated during software creation. Consequently, the different software process & technology failure modes and
roots of software faults lie in the SDP, More precisely, how are they generated?
faults are generated either within an SDP phase, or at 2. How do these SDP failures propagate?
the interface between two SDP phases. To determine how 3. What is the likelihood of a given software failure
software faults are introduced, the realm of SDP failure
mode?
modes that might occur during each phase of the SDP (eg,
the requirements definition & analysis phase, the coding Propagate means that a transformation process occurs
phase) must be identified. The interfaces between consec- by which the SDP failure modes have become faults in the
utive phases (eg, the interface between the requirements software which, in turn, generate failures of the operational
definition & analysis phase, and the preliminary design software, ie, product failures or product failure modes.
phase) must also be considered so that failures that might For instance, ‘Analyst Omits a Functional Requirement’
occur at these interfaces can be identified. is a SDP failure mode. If we assume that such SDP fail-
From assumption #1 [24]; since the waterfall is a rigid ure occurs in the development of a reactor-protection sys-
frame of development, phases of the life-cycle are rather tem, software with functional requirement ‘Software shall
well defined. Assumption # S is certainly restrictive, be- Scram the Reactor on Demand’, the process failure injects
cause all software is not developed under assumption #l; a fault in the reactor protection system software which
however, assumption #1 does not prevent the application we label “the requirement ‘Software shall Scram the Re-
of the prediction model to other software life-cycle models. actor on Demand’ has been omitted by the analyst”. If
the scram €unction is needed during real operation of the
Software is still mostly a human activity; consequently,
reactor protection system software, the fault becomes a
failures in the process, which translate into faults in the
product failure.
software, are the result of human error. This considera-
tion is critical to the generation of SDP failure models. A software reliability prediction method that addresses
these questions has 4 steps:
One interesting application of this approach is to consider
a software failure’s life-cycle. It begins with a human slip3 4A planning failure, vzz, a deficiency in the judgmental and/or
inferential processes involved in the selection of an objective or in
3An execution failure, regardless of whether or not the plan which the specification of the means to achieve it [25].
guided it was adequate to guide its objective. 5The risk to the software product introduced by SDP failures.
SMIDTS ET AL: SOFTWARE RELIABILITY MODELING: AN APPROACH TO EARLY RELIABILITY PREDICTION 271

1. Identification of SDP failure modes. reasons why some people favor Bayes statistics over classi-
2. Development of a requirements-based fault tree that cal statistics in the quantification of SDP failure data; see
relates software failure to applicable SDP failure modes. [26 - 301 for a detailed presentation.
3. Estimation of SDP failure mode probabilities.
4. Determination of software reliability through fault 5 . SDP FAILURE-MODE IDENTIFICATION
tree solution. To determine how SDP failure modes are created within
It is recognized that SDP is dynamic; thus, a credible a particular software development phase, the activities and
software reliability prediction must be capable of refine- participants of a specific life-cycle phase are identified. For
ment a s the software development progresses (require- instance, consider the requirements definition & analysis
ments can be modified, added, or deleted; design ap- phase whose activities include the elicitation of:
proaches can be altered; etc.): 1. Functional requirements (the different functionalities
1. Predictions must include contributions from all life- of the software).
cycle phases, regardless of when the prediction is made. 2. Performance requirements (such as speed).
Errors have been (or will be) made in all stages of the 3. Reliability & and safety requirements (how reliable
life-cycle. and how safe the software should be).
2. Predictions can be updated throughout the life-cycle. 4. Portability requirements (eg, the software should be
Early in the lifecycle, knowledge of the attributes & char- portable from machine A to machine B).
acteristics of the SDP is limited: coding personnel and 5. Memory requirements.
coding languages are undetermined, the design approach 6. Accuracy requirements, etc.
might not be defined, etc. As one progresses through the A list of the types of requirements is in Sage & Palmer
life-cycle phases, information on the specific SDP at hand [31]. The participants in this phase are the software user
becomes available, which can be used to add detail (spe- and the software analyst. The software analyst is generally
cific SlDP failure modes) to the fault tree model. a systems engineer working for the software development
3. Initial prediction of SDP failure mode likelihoods can company. The number of people involved in this phase can
be based on historical information about relevant software vary; for simplicity here, it is assumed that only one user
developments. As the software development progresses, and one analyst are present.
new evidence becomes available (eg, design reviews are
conducted) that can be used to refine SDP failure mode
likelihoods. 1
SOFTWARE REQUIREMENTS 1
The refinement of SDP failure mode likelihoods (obser-
vation #3 noted just above) can sometimes be achieved .'
.'.'
I -

through the use of Bayes' methods. SDP failure data can,

in particular, be characterized as follows: V

1. SDP failure data can be rare; in some instances,

the SDP will be of high quality. Data are rare also be-
cause each software is inherently unique and data cannot Figure 1: Sources of Failure in the Requirements Definition
be saffely aggregated without preliminary treatment. & Analysis Phase
2. SDP failure data can be both objective and subjec-
tive. For objective data, one will find data such as the 5.1 Generic Failure Modes
numkler of requirements omitted, the number of require-
mentis misunderstood, etc. For subjective data, one will Once the activities and participants are identified, it is
find data such as expert-opinion about the degree of rele- possible to identify some generic SDP failure modes (see
vance of certain data sets to the problem (software) cur- figure 1) that apply to all types of requirements.
rently examined. A. User's Generic Failure Modes
3. Different items of data can have different weights . Omission of a known requirement: the user forgets or,
representing the degree of relevance or information of the for other reasons such as cost, omits a specification.
software failure mode. For instance, recent data might . The user is not aware of a specific requirement.
have more weight than older data, because it reflects the . A specified requirement is incorrect: the user misunder-
current SDP better; SDP failures can evolve over time due stands the project's own needs.
to changes in the software engineering technology, devel- . The user specifies a requirement that conflicts with an-
opm'ent environment, training & learning of the human other requirement.
element of SDP, etc. B. Analyst's Generic Failure Modes
Bayes statistics can provide a framework for addressing . The analyst omits (eg, forgets) a specific user require-
thes(e3 characteristics of SDP failure data. There are other ment.
~~~

6Appendix A.2 describes Bayes assumptions & statistics in more

. The analyst misunderstands (cognitive error) a specific
detail. user requirement.
 IKAN>ALIIVN> UN KkLIAtJILII Y, VVL. 41, NU. 3 , 1998 bkY1IzMtlk.K

ner. These failure modes depend on the process and spe-

cific parts of the software product (architecture). Focusing
only on the functional requirements, the example software
will function satisfactorily only if both Rqf, & Rqfi are
implemented correctly (this condition implies that the crit-
icality of both Rqf, and Rqf2 is 1).Hence, the Pf due to
process failures occurring in the requirements definition
and analysis phase (assuming that the failure modes are
s-independent of each other) is:

+ +
pf = P r ( k i } Pr(~%,4} Pr{~%,5}+ Pr.(-%,d
+ +
Pr{E2,1) Pr{E2,4} Pr{E2,5) t W.&,G) +
t Pr{EO,Z} ' Pr(E0,3} - c(6)
The appropriate Ek,%are defined in the fault tree in figure
2; the C denotes the cross- products generated by expand-
ing the union of events (typically, these can have negligible
probability).

SOFTWAREFAILS TO
MEET
REQUIREMENTS

SOFTWARE FNLS TO SOFWAKE FAlls TO

MEET Rq,

USER SPECIFIES
ANALYST OMITS ANALYST OMITS
INCORRECTLY

7
ANALYST ANALYST
MISUNDERSTANDS ANALYST MISHEARS ANALYSTMISHEARS
MISUNOERSTANDS

------l

CONFLICT B E M E N CONFLICT BETWEEN

Kq, AND RR

A Y

life-cycles, and that this set can be tailored for a specific Figure 2: Requirements-Based Fault Tree Showing SDP
application. Failure Modes During the Requirements Definition & Analysis
Phase
6. REQ UIR EMENTS-BASED
FAULT-TREE DEVELOPMEN
'!F
The next step is to find the combination f SDP failure
modes that will either fail a software requi ement or de-
grade its operational capabilities in an unac ,eptable man-
The actual fault tree will be more complex because:
. SDP failure modes generated by all phases of the SDP
contribute to failure of Rqf, & Rqfi,
all types of requirements must be included.
SMIDTS ET A L SOFTWARE RELIABILITY MODELING: AN APPROACH TO EARLY RELIABILITY PREDICI‘ION 213

Table 1: Preliminary List of Failure Modes of the Software Requirements, Definition & Analysis Phase

Functional Requirements I Performance RRquirements I Reliability Requirements 1 Portability Requirements

By User
. Omission of known . Omission of known . Omission of known . Omission of known
requirement requirement requirement requirement
. Unaware of requirement . Unaware of requirement . Unaware of requirement . Unaware of requirement
. Rqp, is incorrect . R.q, is incorrect . R.q, is incorrect . Rqpo is incorrect
. Rqf, is incorrect
. Rqf, conflicts with . Rqp conflicts with . R.q, conflicts with Rqp, conflicts with
other requirements other requirements other requirements other requirements
. Rqf2 conflicts with other
recluiiements
By Analyst [ mis‘stands + misunderstands 1
. Analyst omits Rqf, . Analyst omits Rq, 3 Analyst omits Rq, . Analyst omits Rqpo
. Analyst omits Rqf2
. Analyst mis’stands Rqf, . Analyst mis’stands Rq, 3 Analyst mis’stands Rq, . Analyst mis’stands Rq,,
Analyst mis’stands Rqf,
~

. Analyst transposes Rqf, . Analyst transposes Rq, + Analyst transposes Rq, . Analyst transposes Rq,,
. Analyst transposes Rqf,
. Analyst does not notice Analyst does not notice -Analyst does not notice . Analyst does not notice
conflict between Rqf, and conflict between Rq, and conflict between Rq, and conflict between Rqpo and
other requirements other requirements other requirements other requirements
. Analyst does not notice
conflict between Rqfz and
other requirements

. User’s Rqf, is misheard

The software can be used to achieve several operations. 7. REQUIREMENTS FAILURE QUANTIFICATION
Some operations, like the one described in this section,
Notatzon
might, require functioning of both R,qfl & Rqf, whereas
t time from the start of the SDP,
other operations might only require functioning of R,qfl.
measured in equivalent person-hours
This information should be contained in the operational
1 index for failure mode
profile drawn for the software, which can then be used to
derive Ru: n3 number of failure modes in R Q
cr,(j, 1, E ) scale parameters, q = 1,2:
approximate locations of the peaks
& ( j , 1 , E ) shape parameters, q = I , 2:
determine the amounts of peakedness
a,(3, I , E ) blending parameters, q = 1,2:
oi represents the logical combinations of failure modes determine the relative heights of the peaks
in the fault tree leading to non-achievement of OM,; this
definition differs from (4) only in Chat failure modes have
vx%E’ €1 {%hL €1, P& I, €1, a,(.% E , €1, Q = L21:
set of parameters which define X ( t ; j , 6, E ) ;
been modeled explicitly. details are in the text
RTJcan be expressed directly as a function of the prob- D ( j ,I, E ) evidence related to failure mode I of Rq,
abilities of various failure modes: 7r ( + ( j , I, E ) ( D (I,~ , distribution of @ ( j I,, E ) , given
E))

N
Ni’I, e)
implies: updated averaged value:
Ru = 1 - ~ P r { O M ~ D .Mpi }( C ~ .JPr{~?k[R,q;,~]))(8) as more data become available
i=l
7.1 R.equirements Failure Data
p i is, obtained through Boolean reduction techniques.
Figure 3 provides the basic structure for a requirements- Figure 4 displays the typical behavior observed for re-
based fault tree that links SDP failure modes in all phases quirement changes, additions, and deletions as a function
of the life-cycle to the various operating modes. of the life-cycle effort as observed by the authors from data
274 IEEE TRANSACTIONS ON RELIABILITY, VOL. 47, NO. 3, 1998 SEPTEMBER

ANALYSIS PHASE
TS TESTING PHASE
PD PRELIMINARY DESIGN PHASE

\
SOFTWARE FAILS TO SOFTWARE EXECUTED
MEET REQS DURING IN OPERATINGMODE
0% 0%

SOFTWARE FAILS TO MEET ALL

MEET ALL REQS REQS RELEVANTTOOM,

PHASE CAUSE PHASE CAUSE PHASE CAUSE PHASE CAUSE

Figure 3: Requirements-Based Fault Tree Linking the SDP Failure Modes Occurring in All Life-cycle Phases with the
Operational Profile

collected by Texas Instruments7 [33].Figure 4 shows that 7.2 Time-Dependent Requirements Failure Behavior
most of the requirement changes, additions, and deletions To estimate the requirements-based fault-tree event
occur in the requirements and preliminary-design phases, probabilities, a time-dependent quantification framework
with a decreasing trend until the start of integration and is established. The most general case requires considera-
system/acceptance testing. The histogram shows that er- tion of failure mode 1 of Rq, occurring during a specified
rors are reported in blocks corresponding to distinct blocks life-cycle phase when the software is developed under E (eg,
of effort (eg reviews, inspections, walk-throughs, and dy- “Rql i s omitted during the requirements definition and
namic testing). These trends can change when faced with analysis phase.”). Throughout the SDP, requirement fail-
a user who continuously feeds new requirements into the ures occur and are subsequently repaired. Hopefully, the
process. This section uses the assumption that new re- process is one of positive reliability growth, and the soft-
quirements are introduced in the requirements definition ware becomes more reliable as the requirements become
61 analysis phase only and are constrained to this phase. less likely to fail. This behavior can be modeled using the
Figure 4 shows that the requirements failure mode pdf NHPP (see appendix AI):
is bimodal. The values of the 2 modes are s-correlated. If
more effort were spent in the requirements analysis & def- Pr{M(t; j , 1, E ) = m>= s p [ P ( t ; j , I, €1, ml (9)
inition phase, then the requirements would be more likely
to be unambiguous & complete. Hence the mode during
the integration testing and system testing would be less
than it would be otherwise.

7This data-set depends upon the particular influencing factors (eg,

From the requirements failure data discussed in section
skill level) and the particulars of the Texas Instruments waterfall
life-cycle (eg, number of reviews). Hence the data-set is used for 7.1, one can postulate the time-dependent failure intensity
illustration only. distribution in figure 5.
SMIDTS ET AL: SOFTWARE RELIABILITY MODELING: AN APPROACH TO EARLY RELIABILITY PREDICTION 215

1- REQ- DE-ON AND 1- REQUIREMENTS DEFINITION A M )

ANALYSIS ANALYSIS
2- PRELIMINARY DESIGN 2- PRELIMINARY DESIGN
5 DETAILGD DESIGN 3- DETAILED DESIGN
4- CODING
5 UNIT TEST
C INTEGRATION TEST
7- SYSTEMTEST 6-INTEGRATIONTEST
b OPERATION 7- SYSTEM TEST
8- OPERATION

1 2 3 4 5 1 3 7 8
1 2 3 4 5 6 7 8
LIFECYCLE EFFORT

LIFECYCLE EFFORT
Figure 4: Requirements Failure Data History
Figure 5: Requirements Failure Intensity as a Function of
This bimodal distribution can be represented as a mix- Life-Cycle Effort
ture of 2 Weibull-like distributions:
2 develop a conversion from equivalent dynamic testing time
A ( t ; j , W = .&&,l,6) * cw [ t ; Q q ( j , ~ , ~ ) , ~ q ( j(11)
, ~ , ~ )to
I execution time. From Musa [l]:
q=l
TDY OR * TR f PR ' P(TR) (15)
TDY dynamic testing time
Other distributions (than the Weibull) could have been TR execution time
used to fit the data in figure 5, eg, Gaussian or truncated OR,p~ are constants which can be determined historically.
Gaussian. Then:
7.3 Reliability as a F'unction of Execution Time
Various updates of the failure intensity function are real- Because
ized after the end of each life-cycle phase (as more evidence
becomes available):
A*+ = x + [ t * ; j , z , E J D ( j , 1 , E ) ]
= 1Nt*;j , 1 , €l+(jlz, €)I * .[+(j, I , E ) I D ( j , l , .)I d+ (12)

Upper & lower bounds to A* can be obtained easily since

n(+,lD) is known. Failure probabilities, as calculated using (14), can subse-
Notation quently be used to determine either RSDor Ru:
T,!J~
points in a multi-dimensional space where the
fraction p of the probability is covered. RSD = nr1- pf(RqJ )I (19)
3

1- RU = p r { O M y M 1 . PJ G , l * Pf (RqJ,l)l (20)
391

Using (9) - (20), groupings can be performed by fail-

ure modes (independently of requirements) or per require-
ments (independently of failure modes).
The failure-mode data obtained during the phases of the
life-cycle can be extremely valuable as an estimation tool
without need for a Bayes framework. By plotting the frac-
tion of each failure-mode observed a t the end of each life-
cycle phase, failure modes can be ranked and weaknesses
I

276 IEEE TRANSACTIONS ON RELIABILITY, VOL. 47, NO. 3, 1998 SEPTEMBER

in the process identified & corrected. This information 2. Investigation of Alternative Software Life-cycles. To
also helps in the design of the process and the product. date, efforts have been focused on the waterfall life-cycle;
however, many other life-cycle models have been devised
7.4 Use of Historic Software-Process Failure-Mode Data (such as the spiral, evolutionary model, etc.). The early
To obtain initial distributions for ?r ($(j, I , ~ ) l D ( Ij, ,E ) ) , reliability prediction framework developed in this paper
one can use the valuable information contained in older can treat all software life-cycle models when appropriate
software development projects. As new software is being forms of the prior distribution and likelihood functions are
developed, the degree of relevance of accumulated software developed using relevant data.
failure data from other projects should be determined with 3. Development of a Software Failure Mode and Root
respect to the software development effort under assess- Cause Taxonomy. To develop a software requirements
ment. To evaluate the degree of relevance, the analyst logic model (such as a fault tree), it is necessary to consider
can: relevant software failure modes. A taxonomy of failure
1. Use subjective data only, obtained by eliciting & ag- modes would help an analyst during logic model develop-
gregating expert opinion [35]. For example, the analyst ment, ensure completeness, and establish a common basis
might elicit from the expert: for the collection of software reliability data. One promis-
. a degree of relevance, ing approach is to use concepts from human factors en-
. the expert’s degree-of-belief distribution, gineering (such as software developer mental models, and
. the attributes that will affect the software reliability (as a organizational factors); use of a human factors approach
guideline, one can use the RADC reliability metric), their should also provide insight into the root causes of software
respective impact, and a comparative value between old failures.
and new software.
4. Development of a Tool. A tool is necessary to pro-
2. Use objective data only, provided by human-factors vide standard failure mode lists for possible software de-
studies on software development [32]. velopment life-cycles, standard distributions, and efficient
3. Use both #1 & #2. This is the best approach because algorithms to implement the Bayes updating.
it uses all sources of available information.
5. Validation of the Approach. Validation of the ap-
8. FURTHER DEVELOPMENTS proach requires comparison for a cross-section of software
programs of the predicted reliability (mean value, upper &
This paper has discussed developing a technique for soft- lower bounds) with software field-reliability. This could be
ware reliability prediction during the early phases of soft- a challenging effort. The validation approach has 2 steps:
ware development; its salient points include:
a. Development of small size software under labo-
1. reconciling known definitions of software reliability; ratory or quasi-laboratory conditions. An issue that re-
it addresses both the developers and users perspectives, lates to the development of small size software is the likely
and is consistent with the IEEE definitions; paucity of the process failure modes data. At this point in
2 . considering the human errors occurring during soft- time, a possible lumping strategy of failure modes can be
ware development that give rise to software failures; an ini- examined.
tial taxonomy for software failures originating from human
b. Development of larger size software in industrial
errors occurring in the requirements definition & analysis
environment.
phase has been developed;
3. using a fault tree approach to quantify software reli-
ACKNOWLEDGMENT
ability based on the taxonomy of software process failures;
4. allowing for refinement of the fault tree model dur- We are grateful for the support provided by Texas In-
ing various life-cycle phases as new data and information struments and Science Applications International Corpo-
become available; ration during the conduct of this research and the devel-
5. being able to use Bayes statistics to quantify the opment of this paper.
fault tree events, thus allowing any combination of project-
specific evidence, historic evidence, or expert opinion to be APPENDIX
used. A.l NHPP
To develop these concepts into a practical, validated ap- NHPP allows statistical modeling of event-arrival pro-
proach, the following items need further resolution: cesses, such as the number of arrivals in a queue per
1. Development of Bayes Prior Distributions and Likeli- unit-time, or the number of process failures detected per
hood Functions. Data exist within most software develop- amount of effort spent. NHPP assumes that:
ment organizations to obtain these distributions and func- 1. The number of events of interest is 0 at time t = 0.
tions. The Bayes framework can have 2 advantages in this +
2. The number of events occurring in [t,t d t ] is s-in-
regard: a) discrete or continuous data can be used, and dependent of the number of events that occurred in [0,t ) .
b) incomplete data, or data of partial relevance, can be +
3. The number of events occurring in [t,t d t ] is of the
incorporated using subjective judgement. order of dt.
SMIDTS ET AL: SOFTWARE RELIABILITY MODELING: AN APPROACH TO EARLY RELIABILITY PREDICTION 277

A.2 Bayes [121 N.G. Leveson, L‘Softwaresafety in embedded computer

There are philosophical and statistical-inference differ- systems”, Communications ACM, vol 34, 1991 Feb, pp
34 - 36.
ences between classical & Bayes - probability theory. In
classical theory, probability is used t o model the relative N.G. Leveson, S.S. Cha, J.C.Knight, T.J. Shimeall, “The
frequency of an event; in Bayes, probability is used to use of self checks and voting in software error detection:
model the user’s degree-of-belief that the event will oc- An empirical study”, IEEE Trans. Software Engineering,
cur. Thus, in a classical interval estimate, the s-confidence vol SE-16, 1990 Apr, pp 432 - 443.
level relates t o the frequency with which the estimator [14] N.G. Leveson, J.L. Stolzy, “Safety analysis using Petri
produces intervals that cover the true mean; while in the nets”, IEEE Trans. Software Engineering, vol SE-13, 1987
Bayes paradigm, the b-credibility level relates t o the user’s Mar, pp 386 - 397.
degree-of-belief that the true mean lies in the interval. To [15] N.G. Leveson, P.R. Harvey, “Analyzing software safety”,
use Bayes wisely, one must always do a pre-posterior anal- IEEE Trans. Software Engineering, vol SE-9, 1983 Sep,
ysis of one’s prior distribution. pp 569 - 579.
In a pre-posterior analysis, various experimental data [16] R. Chillarege, W.-L. Kao, R.G. Condit, “Defect type and
(from very bad to very good) are hypothesized, and the its impact on the growth curve”, PTOC. 13th Int’l Conf.
posterior is calculated. This posterior must be what the Software Engineering, 1991.
analyst would believe after seeing those data; else the prior
1171 R. Chillarege, 1.S. Bhandari, J.K. Chaar, et al, “Orthog-
distribution must be redone. onal defect classification - A concept for in-process mea-
The Bayes method is not covered in this paper because surements”, IEEE Trans. Software Engineering, vol SE-
it is reasonably standard and straight-forward and because 18, 1992 NOV,pp 943 - 956.
there are very many books and articles on the topic, eg, J.B. Dugan, M.R. Lyu, “Dependability modeling for fault
[181
281. tolerant software and systems”, Software Fault Tolerance
(M.R. Lyu Ed), 1995; John Wiley & Sons.
REFER.ENCES R. Scott, J.W. Gault, D.F. McAllister, “Fault tolerant
software reliability modeling”, IEEE Trans. Software En-
J.D. Musa, A. Iannino, K. Okumoto, Software Reliability,
gineering, vol SE-13, 1987 May, pp 582 - 592.
1987; McGraw-Hill.
J.C. Laprie, J. Arlat, C. Beournes, K. Kanoun, “Archi-
M.R. Lyu, Handbook of Software Reliability Engineering,
1996; McGraw Hill. tectural issues in software fault tolerance”, Software Fault
Tolerance (M.R. Lyu Ed), 1995; John Wiley & Sons.
A.L. Goel, “Software reliability models: Assumptions,
Software Process Improvement Guidebook, Software Engi-
limitations, and applicability”, IEEE Trans. Software En-
neering Program, NASA- GB-00-95, 1996 Jan.
gineering, vol SE-11, num 12, 1985, pp 1411 - 1423.
J.D. Musa, “Operational profiles in software reliability en-
C.V. Ramamoorthy, F.B. Bastani, “Software reliability:
gineering”, IEEE Software, vol 10, 1993, pp 14 - 32.
Status and perspectives” IEEE Trans. Software Engi-
neering, vol SE-8, 1982, pp 354 - 371. IEEE Guide f o r the Use of IEEE Standard Dictionary of
Measures to Produce Reliable Software, 1989, IEEE Std
E. Nelson, “Estimating software reliability from test
data”, Microelectronics & Reliability, vol 17, 1978, pp 67 982.2-1 988.
- 73. W.W. Royce, “Managing the development of large soft-
ware systems: Concepts and techniques”, PTOC. Wescon,
H.D. Mills, O n the Statistical VaLidation of Computer Pro-
grams, Report 72-6015, 1972; IBM Federal Sys. Div;
1970 Aug. Also available in Proc. ICSES, 1987; Computer
Society Press.
Gaithersburg.
J. Reason, Human ETTOT,1990; Cambridge Univ. Press.
J.E. Gaffney, C.F. Davis, “An approach to estimating soft-
ware errors and availability”, SPC-TR-88-007, ver 1.0, B. Littlewood J.L. Verrall, “A Bayesian reliability growth
1988 Mar, PTOC. Eleventh Minnowbrook Workshop on model for computer software”, J. Royal Statistical SOC-
Software Reliability, 1988 Jul. series C, vol 22, num 3, 1973, pp 332 - 346.
J.E. Gaffney, J. Pietrolewicz, “An automated model for H.F. Martz, R.A. Waller, Bayesian Reliability Analysis,
software early prediction (SWEEP)”, Proc. 13th Minnow- 1992; John Wiley & Sons.
brook Workshop on Software Reliability, 1990 Jul. G.E.P. Box, G.C. Tiao, Bayesian Inference in Statistical
W.W. Agresti, W.M. Evanco, “Projecting software defects Analysis, 1992; John Wiley & Sons.
from analyzing software designs”, IEEE Trans. Software S. Campodonico, N. Singpurwalla, “A Bayesian analysis
Engineering, vol 18, 1992 Nov, pp 988 - 997. of the logarithmic-Poisson execution time model based on
Methodology for Software Prediction, RADC-TR-87-171, expert opinion and failure data”, IEEE B a n s . Software
1987; USAF Systems Command. Engineering, vol 20, num 9, 1994.
Methodology for Software Reliability Prediction and As- R. Stoddard, P. Bottcher, “A Bayesian approach for mod-
sessment, TechRep RL-TR-92-52, vol 1 - 2, 1992; Rome eling quality in software products and processes”, 5th Int ’1
Laboratory. Conf. Software Quality, 1995; ASQC.
218 IEEE TRANSAmIONS ON RELIABILITY, VOL. 47, NO. 3, 1998 SEFTEMBER

[31] A.P. Sage. J.D. Palmer, Software Systems Engineering, M.A. Stutzke; Science Applications Int’l Corp; Husova 7; 110
1990; John Wiley & Sons. 00 Prague 1, Czech Republic.
[32] Human Factors zn Software Development (B. Curtis, E d ) , Internet (e-maz1): mstutzke@terminalxz
1989; IEEE Computer Soc. Press. Martin Stutzke received his BS (1976) and MS (1984) in
Nuclear Engineering from University of Tennessee and is cur-
[33] R.W. Stoddard, “Evolution of requirements during the rently in a PhD program in the Reliability Engineering Pro-
software development life-cycle: TI data”, Extract from gram at the University of Illaryland. He works for SAIC as a
TI Internal Report, 1995. senior risk & reliability engineer.
[34] A. Mosleh, Personal Communication, 1996.
[35] R.M. Cooke, Experts in Uncertaznty: opznzon and SubJec- R*W. Stoddard; Texas Instruments Digita1 Imaging;
tive Probabzlzty zn Sczence, 1991; Oxford Univ. Press. 869305 (M/S 8477); Plano, Texas 75086 USA.
Internet (e-mail): [email protected]
Robert W. Stoddard is a software assurance manager
AUTHORS with Texas Instruments. He holds a BS (1981) in Business from
Dr. C. Smidts; Reliability Engineering Prog; 2100 Marie Mount the University Of Maine at Orano, and a MS (1990) in Systems
Hal], Univ, of Maryland; College Park, Maryland 20742-7531 from the University Of Southern California, Angeles.
USA.
Internet (e-mad): [email protected]
Carol Smidts received her MS & PhD from the Universite
Libre de Bruxelles, Belgium. She is an Assistant Professor
in the Dep’t of Materials and Nuclear Engineering, Reliability
Engineering Program at the University of Maryland, College
Park and the Director of the Software Reliability Engineering Manuscript TR96-179 received: 1996 December 26;
Curriculum. Her research interests include software reliability revised: 1998 July 31
modeling, software testing, human reliability modeling, Markov Responsible editor: M.A. Vouk
processes, and risk assessment. Publisher Item Identifier S 0018-9529(98)09526-8