Chapter

12
Evaluating Products,
Processes, and
Resources
Shari L. Pfleeger
Joann M. Atlee
4th Edition

4th Edition

Contents
12.1
12.2
12.3
12.4
12.5
12.6
12.7
12.8
12.9

Approaches to Evaluation
Selecting an Evaluation Techniques
Assessment vs. Prediction
Evaluating Products
Evaluating Process
Evaluating Resources
Information Systems Example
Real-Time Example
What This Chapter Means for You

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.2

Chapter 12 Objectives
Feature analysis, case studies, surveys, and
experiments
Measurement and validation
Capability maturity, ISO 9000, and other
process models
People maturity
Evaluating development artifacts
Return of investment

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.3

12.1 Approaches to Evaluation

Measure key aspects of product, processes,
and resources
Determine whether we have met goals for
productivity, performance, quality, and
other desire attributes

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.4

12.1 Approaches to Evaluation

Categories of Evaluation

Feature analysis: rate and rank attributes

Survey: document relationships
Case studies
Formal experiment

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.5

12.1 Approaches to Evaluation

Feature Analysis Example: Buying a Design
Tool
List five key attributes that the tool should
have
Identify three possible tools and rate the
criterion
Examine the scores, creating a total score
based on the importance of each criterion
Based on the score, we select the highest
score (t-OO-1)

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.6

12.1 Approaches to Evaluation

Buying a Design Tool (continued)
Design tool ratings
Features

Tool 1:
T-OO-1

Tool 2:
Object
Tool

Tool 3:
Easy
Design

Importance

Good user interface

Object-oriented design

Consistency checking

Use cases

Runs on UNIX

85

77

73

Score

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.7

12.1 Approaches to Evaluation

Surveys
Record data
to determine how project participants reacted to
a particular method, tool, or technique
to determine trends or relationships

Capture information related to products or

projects
Document the size of components, number
of faults, effort expended

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.8

12.1 Approaches to Evaluation

Case Studies
Identify key factors that may affect an
activitys outcome and then document
them
Involve sequence of steps: conception
hypothesis setting, design, preparation,
execution, analysis, dissemination, and
decision making
Compare one situation with another

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.9

12.1 Approaches to Evaluation

Case Study Types
Sister projects: each is typical and has
similar values for the independent
variables
Baseline: compare single project to
organizational norm
Random selection: partition single project
into parts

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.10

12.1 Approaches to Evaluation

Formal Experiment
Controls variables
Uses methods to reduce bias and
eliminate confounding factors
Often replicated
Instances are representative: sample over
the variables (whereas case study samples
from the variables)

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.11

12.1 Approaches to Evaluation

Evaluation Steps
Setting the hypothesis: deciding what we
wish to investigate, expressed as a
hypothesis we want to test
Maintaining control over variables: identify
variables that can affect the hypothesis,
and decide how much control we have
over the variables
Making investigation meaningful: the
result of formal experiment is more
generalizable, while a case study or survey
only applies to certain organization
Pfleeger and Atlee, Software Engineering: Theory and
Practice

Chapter 12.12

12.2 Selecting An Evaluation Technique

Formal experiments: research in the small
Case studies: research in typical
Surveys: research in the large

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.13

12.2 Selecting An Evaluation Technique

Key Selection Factors
Level of control over the variables
Degree to which the task can be isolated
from the rest of the development process
Degree to which we can replicate the basic
situation

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.14

12.2 Selecting An Evaluation Technique

What to Believe
When results conflict, how do we know
which study to believe?
Using series of questions, represented by the
game board

How do you know if the result is valid?

Evaluation pitfalls table

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.15

12.2 Selecting An Evaluation Technique

Investigation and Evaluation Board Game

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.16

12.2 Selecting An Evaluation Technique

Common Pitfalls in Investigation
Pitfall

Description

1. Confounding

Another factor is causing the effect

2. Cause of effect?

The factor could be a result, not a cause, of the treatment

3. Chance

There is always a small possibility that your result happened by

chance

4. Homogeneity

You can find no link because all subjects had the same level of
the factor

5. Misclassification

You can find no link because you can not accurately classify each
subjects level of the factor

6. Bias

Selection procedures or administration of the study inadvertently

bias the result

7. Too short

The short-term effects are different from the long-term ones

8. Wrong amount

The factor would have had an effect, but not in the amount used
in the study

9. Wrong situation

The factor has the desired effect, but not in the situation studied

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.17

12.3 Assessment vs. Prediction

Assessment system examines an existing
entity by characterizing it numerically
Prediction system predicts characteristic of
a future entity; involves a model with
associated prediction procedures
deterministic prediction (we always get the
same output for an input)
stochastic prediction (output varies
probabilistically)

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.18

12.3 Assessment vs. Prediction

Validating Prediction System
Comparing the models performance with
known data in the given environment
Stating a hypothesis about the prediction,
and then looking at data to see whether
the hypothesis is supported or refuted

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.19

12.3 Assessment vs. Prediction

Sidebar 12.1 Comparing Software Reliability Prediction
Modeling
techniques

Predictive
validity

Proportion
of false
negatives
(%)

Proportion
of false
positive
(%)

Proportion of
false
classification
(%)

Completeness
(%)

Overall
Inspection

Wasted
Inspection

Discriminant
Analysis

P= 0.621

28

26

52

42

46

56

Principal
component
analysis plus
discriminant
analysis

P=0.408

15

41

56

68

74

55

Logistic
regression

P=0.491

28

28

56

42

49

58

Principal
component
analysis plus
logistic
regression

P=0.184

13

46

59

74

82

56

Logical
classification
model

P=0.643

26

21

46

47

44

47

Layered
neural
network

P=0.421

28

28

56

42

49

58

Holographic
network

P=0.634

26

28

54

47

51

55

Pfleeger
and Atlee,
Software25Engineering:
Heads
or tails
P=1.000
50 Theory and
50

50

Practice

Chapter
12.2050
50

12.3 Assessment vs. Prediction

Validating Measures
Assuring that the measure captures the
attribute properties it is supposed to
capture
Demonstrating that the representation
condition holds for the measure and its
corresponding attributes

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.21

12.3 Assessment vs. Prediction

Sidebar 12.2 Lines of Code and Cyclomatic Number

The number of lines of code is a valid

measure of program size, however, it is
not a valid measure of complexity
On the other hand, there are many studies
that exhibit a significant correlation
between lines of code and cyclomatic
number

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.22

12.3 Assessment vs. Prediction

A Stringent Requirement for Validation
A measure (e.g., LOC) can be
an attribute measure (e.g., program size)
an input to a prediction system (e.g., predictor
of number of faults)

Do not reject a measure if it is not part of a

prediction system
If a measure is valid for assessment only, it is
called valid in the narrow
If a measure is valid for assessment and useful
for prediction, it is called valid in the wide
sense
Pfleeger and Atlee, Software Engineering: Theory and
Practice

Chapter 12.23

12.4 Evaluating Products

Examining a product to determine if it has
desirable attributes
Asking whether a document, file, or system
has certain properties, such as
completeness, consistency, reliability, or
maintainability
Product quality models
Establishing baselines and targets
Software reusability

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.24

12.4 Evaluating Products

Product Quality Models
Boehms model
ISO 9126
Dromeys Model

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.25

12.4 Evaluating Products

Boehms Quality Model

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.26

12.4 Evaluating Products

Boehms Quality Model (continued)
Reflects an understanding of quality where
the software
does what the user wants it do
uses computer resources correctly and
efficiently
is easy for the user to learn and use
is well-designed, well-coded, and easily tested
and maintained

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.27

12.4 Evaluating Products

ISO 9126 Quality Model
A hierarchical model with six major
attributes contributing to quality
Each right-hand characteristic is related to only
to exactly one left-hand attribute

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.28

12.4 Evaluating Products

ISO 9126 Quality Model (continued)

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.29

12.4 Evaluating Products

ISO 9126 Quality Characteristics
Quality Characteristic

Definition

Functionality

A set of attributes that bear on the existence of a set of functions

and their specified properties. The functions are those that satisfy
stated or implied needs

Reliability

A set of attributes that bear on the capability of software to

maintain its performance level under stated conditions for a
stated period of time

Usability

A set of attributes that bear on the effort needed for use and on
the individual assessment of such use by a stated or implied set
of users

Efficiency

A set of attributes that bear on the relationship between the

software performance and the amount of resources used under
stated conditions

Maintainability

A set of attributes that bear on the effort needed to make

specified modifications (which may include corrections,
improvements, or adaptations of software to environmental
changes and changes in the requirements and functional
specifications)

Portability

A set of attributes that bear on the ability of software to be

transferred from one environment to another (including the
organizational, hardware or software environment)
Pfleeger and Atlee, Software Engineering: Theory and
Chapter 12.30
Practice

12.4 Evaluating Products

Dromey Quality Model
Product quality depends on the tangible
properties of components and component
composition

Correctness properties
Internal properties
Contextual properties
Descriptive properties

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.31

12.4 Evaluating Products

Dromey Quality Model Attributes
The six attributes of ISO 9126
Attributes of reusability
machine independence
separability
configurability

Process maturity attributes

client orientation
well-definedness
assurance
effectiveness

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.32

12.4 Evaluating Products

Dromey Quality Model Framework
Linking product properties to quality attributes

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.33

12.4 Evaluating Products

Dromey Quality Model Example

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.34

12.4 Evaluating Products

Dromey Quality Model Example (continued)
The model is based on the following five
steps
identify a set of high-level quality attributes
identify product components
identify and classify the most significant,
tangible, quality-carrying properties for each
component
propose a set of axioms for linking product
properties to quality attributes
evaluate the model, identify its weaknesses, and
refine or recreate it
Pfleeger and Atlee, Software Engineering: Theory and
Practice

Chapter 12.35

12.4 Evaluating Products

Establishing Baseline and Targets
A baseline describes the usual or typical
result in an organization or category
Baselines are useful for managing
expectations
A target is a variation of a baseline
minimal acceptable behavior

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.36

12.4 Evaluating Products

Quantitative Targets For Managing US Defense
Projects
Item

Target

Malpractice Level

Fault removal efficiency

>95%

<70%

Original fault density

<4 per function point

>7 per function point

Slip or cost overrun in

Excess of risk reverse

0%

>=10%

Total requirements creep

(function points or
equivalent)

<1% per month

average

>= 50%

Total program
documentation

<3% pages per

function point

>6 pages per function

point

Staff turnover

1 to 3% per year

>5% per year

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.37

12.4 Evaluating Products

Software Reusabilty
Software reuse: the repeated use of any
part of a software system

documentation
code
design
requirements
test cases
test data

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.38

12.4 Evaluating Products

Type of Reuse
Producer reuse: creating components for
someone else to use
Consumer reuse: using components
developed for some other product
Black-box reuse: using component without
modification
Clear- or white-box reuse: modifying
component before reusing it

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.39

12.4 Evaluating Products

Reuse Approaches
Compositional reuse: uses components as
building blocks; development done from
bottom up
Generative reuse: components designed
specifically for a domain; design is topdown
Domain analysis: identifies areas of
commonality that make a domain ripe for
reuse

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.40

12.4 Evaluating Products

Aspects of Reuse
Substance
Ideas and
concepts
Artifacts and
components

Scope
Vertical
Horizontal

Mode
Planned and
Systematic

Technique
Compositiona
l
Generative

Ad hoc,
opportunistic

Procedures,

Intention
Black-box,
as is
Clear-box
modified

Product
Source Code
Design
Requirements
Objects
Data

skills, and

Processes

experience

Documentatio
n

Patterns

Tests

Architecture

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.41

12.4 Evaluating Products

Reuse Technology
Component classification: collection of
reusable components are organized and
catalogued according to a classification
scheme
hierarchical
faceted classification

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.42

12.4 Evaluating Products

Example of A Hierarchical Scheme
New topic can be added easily at the lowest
level

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.43

12.4 Evaluating Products

Faceted Classification Scheme
A facet is a kind of descriptor that helps to
identify the component
Example of the facets of reusable code

a application area
a function
an object
a programming language
an operating system

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.44

12.4 Evaluating Products

Component Retrieval
A retrieval system or repository: an automated
library that can search for and retrieve a
component according to the users description
A repository should address a problem of
conceptual closeness (values that are similar to
but not exactly the same as the desired
component)
Retrieval system can
record information about user requests
retain descriptive information about the component

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.45

12.4 Evaluating Products

Sidebar 12.3 Measuring Reusability
The measures must
address a goal
reflect perspective of the person asking the
question

Even if we had a good list of measurements,

still it is difficult to determine the
characteristic of the most reused component
Look at past history
Engineering judgment
Automated repository
Pfleeger and Atlee, Software Engineering: Theory and
Practice

Chapter 12.46

12.4 Evaluating Products

Experience with Reuse
Raytheon
A new system contained an average of 60%
reused code increasing productivity by 50%

GTE Data Services

Established incentives and rewards for program
authors whenever their components were reused
14% reuse on its project, valued at a savings of
$1.5 million

Nippon Novel
Paid 5 cents per line of code to a developer who
reused a component
Pfleeger and Atlee, Software Engineering: Theory and
Practice

Chapter 12.47

12.4 Evaluating Products

Sidebar 12.4 Software Reuse at Japans Mainframe
Makers

NEC: reuse library was established to classify,

catalog, and document
Hitachi: integrated software environment,
called Eagle, to allow software engineers to
reuse standard program patterns and
functional procedures
Fujitsu: created Information Support Center
(ISC), that is a regular library staffed with
system analysts, software engineers, reuse
experts, and switching system domain experts
Pfleeger and Atlee, Software Engineering: Theory and
Practice

Chapter 12.48

12.4 Evaluating Products

Benefits of Reuse
Reuse increases productivity and quality
Reusing component may increase
performance and reliability
A long term benefit is improved system
interoperability

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.49

12.4 Evaluating Products

Example of Reuse Success
Quality, productivity, and time to market at
HP
Project
Characteristics

HP Project 1

HP Project 2

Size

1100 noncommented
source statements

700 noncommented
source statements

Quality

51% fault reduction

24% fault reduction

Productivity

57% increase

40% increase

Time to market

Data not available

42% reduction

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.50

12.4 Evaluating Products

Example of Cost of Reuse
Cost to produce and reuse at HP
Air traffic control
system

Menu- and forms

Graphics
Firmware (%)

(%)

Management system
(%)

Relative cost to
create
Reusable code

200

120 to 480

111

Relative cost to
reuse

10 to 20

10 to 63

19

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.51

12.4 Evaluating Products

Sidebar 12.5 Critical Reuse Success Factors at NTT

Success factors at NTT in implementing

reuse
senior management commitment
selecting appropriate target domains
systematic development of reusable modules
based on domain analysis
investing several years of continuous effort in
reuse

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.52

12.4 Evaluating Products

Reuse Lessons
Reuse goals should be measurable
Management should resolve reuse goals
early
Different perspectives may generate
different questions about reuse
Every organization must decide at what
level to answer reuse questions
Integrate the reuse process into the
development process
Let your business goals suggest what to
measure
Pfleeger and Atlee, Software Engineering: Theory and
Practice

Chapter 12.53

12.4 Evaluating Products

Conflicting Interpretation of Goals
A division managers reuse goal may conflict with
a project managers goal, so no reuse ever gets
done

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.54

12.4 Evaluating Products

Questions for Successful Reuse
Do you have the right model of reuse
What are the criteria for success
How can current cost models be adjusted to
look at collections of projects, not just
single projects?
How do regular notions of accounting fit
with reuse?
Who is responsible for component quality?
Who is responsible for process quality and
maintenance
Pfleeger and Atlee, Software Engineering: Theory and
Practice

Chapter 12.55

12.5 Evaluating Process

Postmortem Analysis
A postimplementation assessment of all
aspects of the project, including products,
process, and resources, intended to
identify areas of improvement for future
projects
Takes places shortly after a projects is
completed, or can take place at any time
from just before delivery to 12 months
afterward

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.56

12.5 Evaluating Process

When Postimplemlentaion Evaluation Is
Done
Time period

Percentage of Respondent
(of 92% organizations)

Just before delivery

At delivery

27.8
4.2

One month after delivery

22.2

Two months after delivery

6.9

Three months after delivery

18.1

Four months after delivery

1.4

Five months after delivery

1.4

Six months after delivery

13.9

Twelve months after delivery

Pfleeger and Atlee, Software Engineering: Theory and
Practice

4.2
Chapter 12.57

12.5 Evaluating Process

Sidebar 12.6 How Many Organizations Perform
Postmortem Analysis

Kumar (1990) surveyed 462 medium-sized

organizations
92 organizations that responded, more than
one-fifth did not postmortem analysis
those that did, postmortem were conducted on
fewer than half of the projects in the
organization

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.58

12.5 Evaluating Process

Postmortem Analysis Process
Design and promulgate a project survey to
collect relevant data
Collect objective project information
Conduct a debriefing meeting
Conduct a project history day
Publish the results by focusing on lessons
learned

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.59

12.5 Evaluating Process

Postmortem Analysis Process: Survey
A starting point to collect data that cuts
across the interests of project team
members
Three guiding principles
Do not ask for more than you need
Do not ask leading questions
Preserve anonymity

Sample questions shown in Sidebar 12.7

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.60

12.5 Evaluating Process

Sidebar 12.7 Sample Survey Questions From
Wildfire Survey
Were interdivisional lines of responsibility clearly
defined throughout the project?
Did project-related meetings make effective use
of your time?
Were you empowered to participate in discussion
regarding issues that affected your work?
Did schedule changes and related decision
involve the right people
Was project definition done by the appropriate
individuals?
Was the build process effective for the component
area your work on?
What is your primary function on this project?
Pfleeger and Atlee, Software Engineering: Theory and
Practice

Chapter 12.61

12.5 Evaluating Process

Postmortem Analysis Process: Objective
Information
Obtain objective information to
complement the survey opinions
Collier, Demarco and Fearey suggest three
kinds of measurements: cost, schedule,
and quality
Cost measurements might include
person-months of effort
total lines of code
number of lines of code changed or added
number of interfaces
Pfleeger and Atlee, Software Engineering: Theory and
Practice

Chapter 12.62

12.5 Evaluating Process

Postmortem Analysis Process: Debriefing
Meeting
Allows team members to report what did
and did not go well on the project
Project leader can probe more deeply to
identify the root cause of positive and
negative effects
Some team members may raise issues not
covered in the survey questions
Debriefing meetings should be loosely
structured
Pfleeger and Atlee, Software Engineering: Theory and
Practice

Chapter 12.63

12.5 Evaluating Process

Postmortem Analysis Process: Project History
Day
Objective: identify the root causes of the
key problems
Involves a limited number of participants
who know something about key problems
Review schedule predictability charts
Show where problems occurred
Spark discussion about possible causes of each
problem

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.64

12.5 Evaluating Process

Postmortem Analysis Process: SchedulePredictability Charts
For each key project milestone, the chart
shows when the predictions was made
compared with the completion date

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.65

12.5 Evaluating Process

Postmortem Analysis Process: Publish
Results
Objective: Share results with the project
team
Participants in the project history day write
a letter to managers, peers, developers
The letter has four parts
Introduction to the project
A summary of postmortems positive findings
A summary of three worst factors that kept the
team from meeting its goals
Suggestions for improvement activities
Pfleeger and Atlee, Software Engineering: Theory and
Practice

Chapter 12.66

12.5 Evaluating Process

Process Maturity Models
Capability Maturity Model (CMM)
Software Process Improvement and
Capability dEtermination (SPICE)
ISO 9000

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.67

12.5 Evaluating Process

Capability Maturity Model
Developed by Software Engineering
Institute
There are five levels of maturity
Each level is associated with a set of key
process area

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.68

12.5 Evaluating Process

CMM Levels of Maturity

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.69

12.5 Evaluating Process

CMM Maturity Levels

Level
Level
Level
Level
Level

1:
2:
3:
4:
5:

Initial
Repeatable
Defined
Managed
Optimizing

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.70

12.5 Evaluating Process

CMM Level 1
Initial: describes a software development
process that is ad hoc or even chaotic
It is difficult even to write down or depict
the overall process
No key process areas at this level

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.71

12.5 Evaluating Process

Required Questions for Level 1 of The Process
Maturity Model
Question number

Question

1.1.3

Does the software quality assurance function have a management reporting

channel separate from the software development project management?

1.1.6

Is there a software configuration control function for each project that involves
software development?

2.1.3

Is a formal process used in the management review of each software development

prior to making contractual commitments?

2.1.14

Is a formal procedure used to make estimates of software size?

2.1.15

Is a formal procedure used to produce software development schedules?

2.1.16

Are formal procedures applied to estimating software development cost?

2.2.2

Are profiles of software size maintained for each software configuration item over
time?

2.2.4

Are statistic on software code and test errors gathered?

2.4.1

Does senior management have a mechanism for the regular review of the status of
software development projects/

2.4.7

Do software development first-line managers sign off on their schedule and cost
estimates?

2.4.9

Is a mechanism used for controlling changes to the software requirements?

2.4.17

Is a mechanism used for controlling changes to the code?

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.72

12.5 Evaluating Process

Key Process Areas in The CMM
CMM Level

Key Process Areas

Initial

None

Repeatable

Requirement Management
Software project planning
Software project tracking and oversight
software subcontract management
Software quality assurance
Software Configuration management

Defined

Organization process focus

Organization process definition
Training program
Integrated software management
Software product engineering
Intergroup coordination
Peer reviews

Managed

Quantitative process management

Software quality management

Optimizing

Fault prevention
Technology change management
Process change management

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.73

12.5 Evaluating Process

CMM Level 2
Repeatable: identifying the inputs and
outputs of the process, the constraints,
and the resources used to produce final
product

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.74

12.5 Evaluating Process

CMM Level 3
Defined: management and engineering
activities are documented, standardized
and integrated

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.75

12.5 Evaluating Process

CMM Level 4
Managed: process directs its effort at
product quality

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.76

12.5 Evaluating Process

CMM Level 5
Optimizing: quantitative feedback is incorporated
in the process to produce continuous process
improvement

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.77

12.5 Evaluating Process

CMM Key Practices

Commitment to perform
Ability to perform
Activities performed
Measurement and analysis
Verifying implementation

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.78

12.5 Evaluating Process

SPICE
SPICE is intended to harmonize and extend
the existing approaches (e.g., CMM,
BOOTSTRAP)
SPICE is recommended for process
improvement and capability determination
Two types of practices
Base practices: essential activities of a specific
process
Generic practices: institutionalization
(implement a process in a general way)
Pfleeger and Atlee, Software Engineering: Theory and
Practice

Chapter 12.79

12.5 Evaluating Process

SPICE Architecture for Process Assessment

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.80

12.5 Evaluating Process

SPICE Functional View Activities/Processes
Customer-supplied: processes that affect
the customer directly
Engineering: processes that specify,
implement, or maintain the system
Project: processes that establish the
project, and coordinate and manage
resources
Support: processes that enable other
processes
Organizational: processes that establish
business goals
Pfleeger and Atlee, Software Engineering: Theory and
Practice

Chapter 12.81

12.5 Evaluating Process

SPICE Six Levels of Capability
0: Not performed failure to perform
1: Performed informally: not planned and tracked
2: Planned and tracked: verified according to the
specified procedures
3: Well-defined: well-defined processusing
approved processes
4: Quantitatively controlled: detailed performance
measures
5: Continuously improved: quantitative targets for
effectiveness and efficiency based on business
goals
Pfleeger and Atlee, Software Engineering: Theory and
Practice

Chapter 12.82

12.5 Evaluating Process

ISO 9000
Produced by The International Standards
Organization (ISO)
Standard 9001 is most applicable to the
way we develop and maintain software
Used to regulate internal quality and to
ensure the quality suppliers

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.83

12.5 Evaluating Process

ISO 9001 Clauses
Clause number

Subject matter

4.1

Management responsibility

4.2

Quality system

4.3

Contract review

4.4

Design control

4.5

Document and data control

4.6

Purchasing

4.7

Control of customer-supplied product

4.8

Product identification and traceability

4.9

Process control

4.10

Inspection and testing

4.11

Control of inspection, measuring, and test equipment

4,12

Inspection and test status

4,.13

Control of nonconforming product

4.14

Corrective and preventive action

4.15

Handling, storage, packaging, preservation, and delivery

4.16

Control of quality records

4,17

Internal quality audits

4.18

Training

Pfleeger and Atlee, Software Engineering: Theory and

Servicing
Practice 4.19

Chapter 12.84

12.6 Evaluating Resources

People Maturity Model
Return on investment

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.85

12.6 Evaluating Resources

People Maturity Model
Proposed by Curtis, Hefley and Miller for
improving the knowledge and skills of the
workforce
It has five levels

Initial
Repeatable
Defined
Managed
Optimizing

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.86

12.6 Evaluating Resources

People Capability Maturity Model Levels
Level

Focus

Key Practices

5: Optimizing

Continuous knowledge and

Skill improvements

Continuous workforce innovation

Coaching
Personal competency development

4: Managed

Effectiveness measure and

managed, high-performance
teams developed

Organizational performance alignment

Organizational competency
management
Team-based practice
Team building
Mentoring

3: Defined

Competency-based workforce
practice

Participatory culture
Competency-based practices
Career development
Competency development
Workforce planning
Knowledge and skill analysis

2: Repeatable

Management takes
responsibility for managing its
people

Compensation
Training
Performance management
Staffing
Communication
Work environment

1: Initial
Pfleeger and Atlee, Software Engineering: Theory and
Practice

Chapter 12.87

12.6 Evaluating Resources

Return on investment
Use net present value
value today of predicted future cash flows

Example:
Cash flows
Initial investment
Year 1
Year 2
Year 3
Year 4
Sum of all cash flows
NPV at 15%

COTS
-9000
5000
6000
7000
-4000
5000
2200

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Reuse
-4000
-2000
2000
4500
6000
6500
2162

Chapter 12.88

12.6 Evaluating Resources

Return on Investment at Chase Manhattan
RMS has increased customer calls by 33%
and improved profitability by 27%
By protecting its old investment and
encouraging communication among
employees, Chase Manhattan accomplished
avoid huge investment in new hardware
provide more data more quickly to its service
representative
achieved an admirable return on investment
created cohesive teams that understand more
about Chase Manhattans business
Pfleeger and Atlee, Software Engineering: Theory and
Practice

Chapter 12.89

12.7 Information Systems Example

Piccadilly System
A postmortem analysis must review the
business as well as technology
Is this system good for business

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.90

12.7 Real-Time Example

Ariane-5
A fine example of a postmortem analysis
Focused on the obvious need to determine what
caused the fault that required exploding the
rocket
Avoided blamed and complaint

Pfleeger and Atlee, Software Engineering: Theory and

Practice

Chapter 12.91

12.8 What This Chapter Means For You

There are several approaches to evaluation,
including feature analysis, surveys, case studies,
and formal experiments
Measurement is essential for any evaluation
It is important to understand the difference
between assessment and prediction
Product evaluation is usually based on a model of
the attributes of interest
Process evaluation can be done in many ways
Return-on-investment strategies helps us
understands whether business is benefiting from
investment in people, tools, and technology
Pfleeger and Atlee, Software Engineering: Theory and
Practice

Chapter 12.92