Bradford Clark

Raymond Madachy (Eds.)

Software Cost Estimation

Metrics Manual for
Defense Systems

Activities

Effort =
A · SizeB

.
Duration
 i

Editors

Bradford Clark Raymond Madachy

Software Metrics Inc. Naval Postgraduate School
Haymarket, VA, USA Monterey, CA, USA

⃝
c 2015 Software Metrics Inc.

All rights reserved. The Software Cost Estimation Metrics Manual for Defense Systems
may be used for non-commercial purposes. Users may access, download, copy, translate,
text mine or data mine, and include it in a collective work, as long as they credit the
authors and provided they do not alter or modify it. The book can be cited as:

B. Clark and R. Madachy (Eds.), Software Cost Estimation Metrics Manual

for Defense Systems, Software Metrics Inc., Haymarket, VA, 2015.

First edition: 2015

ISBN 978-0-9904787-0-6

Software Metrics Inc.

Haymarket, VA, USA
software-metrics.com
ii
Acknowledgements

Many people and institutions supported this manual and made it possible.
We thank the numerous reviewers for valuable feedback and suggestions on
earlier wiki versions. Reviews were provided both online formally and dur-
ing workshops. Forty-eight workshop participants helped guide our research
and improve this material hosted by the University of Southern California
Center for Systems and Software Engineering (USC CSSE) for several years,
the Massachusetts Institute of Technology Engineering Systems Division, and
the Carnegie Mellon University Software Engineering Institute. The DoD-
sponsored Cyber Security and Information Systems Information Analysis Cen-
ter and Quanterion Solutions also helped support this work and hosted web
seminars.
The external reviewers for this edition were:

• Jo Ann Lane, University of Southern California

• Daniel Ligett, Softstar Systems
• Daniel Nussbaum, Naval Postgraduate School
• David Seaver, National Security Agency
• Richard Selby, Northrop Grumman
• Daniel Strickland, Missile Defense Agency
• David Zubrow, Software Engineering Institute

This empirical research depended on many data submitters for their

project data and time, but they must remain anonymous. Tool vendors pro-
vided us with people, tools, and manuals. The cost estimation tool companies
supporting this work were:
• Galorath Systems
• PRICE Systems
• Quantitative Software Management
• Softstar Systems
From the estimation tool companies we were assisted by Dan Galorath,
Tim Hohmann, Bob Hunt, and Karen McRitchie at Galorath Systems; Arlene
Minkiewicz, James Otte and David Seaver from PRICE Systems; Larry Put-
nam from Quantitative Software Management; and Dan Ligett from Softstar
Systems.
We also thank Jairus Hihn and Sherry Stukes from NASA Jet Propulsion
Laboratory, Tom McGibbon from Quanterion Solutions, and Don Reifer at

iii
Reifer Consultants for their support and collaboration effort. Dave Olwell
from Naval Postgraduate School and Lori Vaughn from Northrop Grumman
also helped us with the manual.
Wilson Rosa initiated the research for this manual while at the Air Force
Cost Analysis Agency (AFCAA) to help improve software cost estimation
practices for analysts and decision makers across the DoD community collab-
oratively. He worked tirelessly bridging disparate stakeholders. The research
was also supported by the Systems Engineering Research Center (SERC) un-
der Contract H98230-08-D-0171, and the US Army Contracting Command,
Joint Munitions & Lethality Center, Joint Armaments Center, Picatinny Ar-
senal, NJ, under RFQ 663074. These research results have been approved for
public distribution.
Contributors

Barry Boehm
University of Southern California
Los Angeles, California

Bradford Clark
Software Metrics Inc.
Haymarket, Virginia

Joseph Dean
Air Force Cost Analysis Agency
Washington, DC

Cheryl Jones
US Army Armament Research Development and Engineering Center
Picatinny Arsenal, New Jersey

Raymond Madachy
Naval Postgraduate School
Monterey, California

John McGarry
US Army Armament Research Development and Engineering Center
Picatinny Arsenal, New Jersey

Wilson Rosa
Naval Center for Cost Analysis
Washington, DC

v
vi
Preface

This manual transitions our research analyzing and improving software devel-
opment cost metrics reported with the United States Department of Defense
(DoD) Software Resource Data Report (SRDR). The goal is to create cost
estimating relationship (CER) models and productivity benchmarks based on
consistent metrics definitions, and provide guidance in their usage. It is pri-
marily intended for DoD software cost analysts at different levels from entry
level to seasoned experts. Cost analysts and program management in related
and similar application domains will also find it useful.
It includes a description of the metrics data that the SRDR instruments;
how to inspect and normalize it for consistency; and characterizes the CERs
and productivity benchmarks derived from the current data. Readers can gain
useful lessons learned from analyzing the metrics, some of which were used to
improve the data reporting requirements.
The manual covers in detail the core metrics definitions relevant to re-
porting and analyzing DoD cost data. These standardized software metrics
enable a consistent description of the data used in creating CERs. It illus-
trates the steps involved in normalizing SRDR data or software metrics in
similar domains. There is a comprehensive discussion on modern estimating
challenges that provides guidance for applying CERS on actual and upcom-
ing DoD programs. The full thread of the program software cost estimation
process is overviewed including data reporting and CER usage.
This evolving research will continue with additional program data con-
tinuously collected in the Defense Automated Cost Information Management
System (DACIMS) that we analyze from, and there will be further improve-
ments in the reporting. The SRDR data requirements are periodically revisited
in the DoD and updated to collect valuable and higher quality data. The in-
tent is to keep this manual relevant with future editions incorporating new
results.
This book has a companion web site at http://softwarecost.org for sup-
plemental resources, content updates and errata. Readers can also submit
comments and improvement suggestions.

vii
viii
List of Figures

1.1 Example Defense Operating Environments . . . . . . . . . . . 2

1.2 Estimation and Metrics Processes . . . . . . . . . . . . . . . . 3
1.3 Normalized Effort vs. Equivalent Size in SRDRs . . . . . . . 4

5.1 Building CERs and Benchmarks . . . . . . . . . . . . . . . . 45

5.2 No Relationship . . . . . . . . . . . . . . . . . . . . . . . . . . 57
5.3 Strong Positive Relationship . . . . . . . . . . . . . . . . . . . 58
5.4 Strong Negative Relationship . . . . . . . . . . . . . . . . . . 58
5.5 Homoscedastic Relationship . . . . . . . . . . . . . . . . . . . 59
5.6 Heteroscedastic Relationship . . . . . . . . . . . . . . . . . . 60
5.7 RTE Size Data Distributions . . . . . . . . . . . . . . . . . . 61
5.8 Residuals Scatter Plot . . . . . . . . . . . . . . . . . . . . . . 67
5.9 RTE CER Prediction Interval and Error Distribution . . . . . 68
5.10 AV Dataset and CER Summary . . . . . . . . . . . . . . . . . 71
5.11 GS Dataset and CER Summary . . . . . . . . . . . . . . . . . 72
5.12 GV Dataset and CER Summary . . . . . . . . . . . . . . . . 73
5.13 MV Dataset and CER Summary . . . . . . . . . . . . . . . . 74
5.14 OV Dataset and CER Summary . . . . . . . . . . . . . . . . 75
5.15 CC Dataset and CER Summary . . . . . . . . . . . . . . . . 77
5.16 CAS Dataset and CER Summary . . . . . . . . . . . . . . . . 78
5.17 COM Dataset and CER Summary . . . . . . . . . . . . . . . 79
5.18 MP Dataset and CER Summary . . . . . . . . . . . . . . . . 80
5.19 RTE Dataset and CER Summary . . . . . . . . . . . . . . . . 81
5.20 S&S Dataset and CER Summary . . . . . . . . . . . . . . . . 82
5.21 SCP Dataset and CER Summary . . . . . . . . . . . . . . . . 83
5.22 SYS Dataset and CER Summary . . . . . . . . . . . . . . . . 84
5.23 TMDE Dataset and CER Summary . . . . . . . . . . . . . . 85
5.24 VC Dataset and CER Summary . . . . . . . . . . . . . . . . 86
5.25 VP Dataset and CER Summary . . . . . . . . . . . . . . . . 87
5.26 CCGS Dataset and CER Summary . . . . . . . . . . . . . . . 89
5.27 CASGS Dataset and CER Summary . . . . . . . . . . . . . . 90
5.28 COMGS Dataset and CER Summary . . . . . . . . . . . . . . 91
5.29 COMMV Dataset and CER Summary . . . . . . . . . . . . . 92
5.30 MPGS Dataset and CER Summary . . . . . . . . . . . . . . . 93
5.31 RTEAV Dataset and CER Summary . . . . . . . . . . . . . . 94

ix
x

5.32 RTEGS Dataset and CER Summary . . . . . . . . . . . . . . 95

5.33 SCPAV Dataset and CER Summary . . . . . . . . . . . . . . 96
5.34 SCPGS Dataset and CER Summary . . . . . . . . . . . . . . 97
5.35 SYSGS Dataset and CER Summary . . . . . . . . . . . . . . 98
5.36 VCAV Dataset and CER Summary . . . . . . . . . . . . . . . 99
5.37 VCGV Dataset and CER Summary . . . . . . . . . . . . . . . 100
5.38 VPAV Dataset and CER Summary . . . . . . . . . . . . . . . 101
5.39 Operating Environment Productivities Boxplot . . . . . . . . 105
5.40 Application Type Productivities Boxplot . . . . . . . . . . . . 106
5.41 Application Type and Operating Environment Productivities
Boxplot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

6.1 Effects of IDPD on Multiple Builds . . . . . . . . . . . . . . . 114

6.2 Summary of Different Processes . . . . . . . . . . . . . . . . . 120

7.1 GAO’s Cost Estimation Process . . . . . . . . . . . . . . . . . 127

7.2 High-Level Aircraft WBS Decomposed to Software Elements 133
7.3 Cost Cumulative Probability Distribution . . . . . . . . . . . 138
7.4 Common Probability Distributions . . . . . . . . . . . . . . . 139
7.5 Size Input Normal Probability Distribution . . . . . . . . . . 142
7.6 RTE Effort Probability Distribution Function . . . . . . . . . 142
7.7 RTE Effort Cumulative Distribution Function . . . . . . . . . 143

C.1 Simple CER Nomogram Sensitivity Analysis . . . . . . . . . . 177

C.2 COCOMO Nomogram Estimate . . . . . . . . . . . . . . . . . 178
C.3 Aerial Vehicle - CER Nomogram . . . . . . . . . . . . . . . . 179
C.4 Ground Site - CER Nomogram . . . . . . . . . . . . . . . . . 180
C.5 Ground Vehicle - CER Nomogram . . . . . . . . . . . . . . . 181
C.6 Maritime Vessel - CER Nomogram . . . . . . . . . . . . . . . 182
C.7 Ordnance Vehicle - CER Nomogram . . . . . . . . . . . . . . 183
C.8 Ground Site - Command and Control (C&C) CER Nomogram 185
C.9 Ground Site - Custom AIS Software (CAS) CER Nomogram 186
C.10 Ground Site - Communications (COM) CER Nomogram . . . 187
C.11 Ground Site - Mission Planning (MP) CER Nomogram . . . 188
C.12 Ground Site - Real Time Embedded (RTE) CER Nomogram 189
C.13 Ground Site - Sensor Control and Signal Processing (SCP) CER
Nomogram . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
C.14 Ground Site - System Software (SYS) CER Nomogram . . . . 191
C.15 Ground Vehicle - Vehicle Control (VC) CER Nomogram . . . 193
C.16 Aerial Vehicle - Real Time Embedded (RTE) CER Nomogram 195
C.17 Aerial Vehicle - Sensor Control and Signal Processing (SCP)
CER Nomogram . . . . . . . . . . . . . . . . . . . . . . . . . 196
C.18 Aerial Vehicle - Vehicle Control (VC) CER Nomogram . . . . 197
C.19 Aerial Vehicle - Vehicle Payload (VP) CER Nomogram . . . . 198
C.20 Maritime Vessel - Communications (COM) CER Nomogram . 200
 xi

C.21 Command and Control (C&C) CER Nomogram . . . . . . . . 202

C.22 Communications (COM) CER Nomogram . . . . . . . . . . . 203
C.23 Custom AIS Software (CAS) CER Nomogram . . . . . . . . . 204
C.24 Sensor Control and Signal Processing (SCP) CER Nomogram 205
C.25 Scientific and Simulation (S&S) CER Nomogram . . . . . . . 206
C.26 Mission Processing (MP) CER Nomogram . . . . . . . . . . . 207
C.27 Real Time Embedded (RTE) CER Nomogram . . . . . . . . . 208
C.28 System Software (SYS) CER Nomogram . . . . . . . . . . . . 209
C.29 Test, Measurement, and Diagnostic Equipment (TMDE) CER
Nomogram . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
C.30 Vehicle Control (VC) CER Nomogram . . . . . . . . . . . . . 211
C.31 Vehicle Payload (VP) CER Nomogram . . . . . . . . . . . . . 212
C.32 COCOMO Nominal Nomogram . . . . . . . . . . . . . . . . . 214
C.33 COCOMO with EAF Nomogram . . . . . . . . . . . . . . . . 215

D.34 Unified CodeCount Example Summary Output . . . . . . . . 217

xii
List of Tables

2.1 SRDR Reporting Events . . . . . . . . . . . . . . . . . . . . . 10

3.1 Software Size Types . . . . . . . . . . . . . . . . . . . . . . . 20

3.2 SLOC Count Definitions . . . . . . . . . . . . . . . . . . . . . 22
3.3 ESLOC Summary . . . . . . . . . . . . . . . . . . . . . . . . 25
3.4 User Function Types . . . . . . . . . . . . . . . . . . . . . . . 27
3.5 Function Point Counting Weights for Internal Logical Files and
External Interface Files . . . . . . . . . . . . . . . . . . . . . 28
3.6 Function Point Counting Weights for External Outputs and
External Inquiries . . . . . . . . . . . . . . . . . . . . . . . . 28
3.7 Function Point Counting Weights for External Inputs . . . . 28
3.8 Unadjusted Function Point Complexity Weights . . . . . . . . 29
3.9 Effort Activities . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.10 Effort Phases . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

4.1 Data Quality Rating Scale . . . . . . . . . . . . . . . . . . . 37

4.2 SRDR Activities . . . . . . . . . . . . . . . . . . . . . . . . . 41
4.3 Proxy DM, CM, and IM Values . . . . . . . . . . . . . . . . . 42

5.1 Operating Environments Definition . . . . . . . . . . . . . . . 47

5.2 Application Types Definitions . . . . . . . . . . . . . . . . . . 48
5.3 CER Evaluation Statistics . . . . . . . . . . . . . . . . . . . . 64
5.4 RTE CER Regression Statistics . . . . . . . . . . . . . . . . . 67
5.5 RTE CER Coefficient Statistics . . . . . . . . . . . . . . . . . 67
5.6 Model Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5.7 CERs by Operating Environment . . . . . . . . . . . . . . . . 70
5.8 CERs by AppType Across All Environments . . . . . . . . . 76
5.9 CERs by AppType and OpEnv . . . . . . . . . . . . . . . . . 88
5.10 Effort Distributions . . . . . . . . . . . . . . . . . . . . . . . . 102
5.11 Productivity Statistic Definitions . . . . . . . . . . . . . . . . 104
5.12 Productivity Benchmarks by Operating Environment . . . . . 105
5.13 Productivity Benchmarks by Application Type . . . . . . . . 106
5.14 Productivity Benchmarks by Operating Environment and Ap-
plication Type . . . . . . . . . . . . . . . . . . . . . . . . . . 108

6.1 Recommended RVOL Rating Levels . . . . . . . . . . . . . . 112

xiii
xiv

6.2 IDPD Effort Drivers . . . . . . . . . . . . . . . . . . . . . . . 113

6.3 Situation-Dependent Processes and Estimation Approaches . 121
6.4 Process Selection Criteria . . . . . . . . . . . . . . . . . . . . 122

7.1 Radar Software WBS Decomposition . . . . . . . . . . . . . . 133

A.1 Cost Model Size Inputs . . . . . . . . . . . . . . . . . . . . . 151

A.2 Cost Model Factors . . . . . . . . . . . . . . . . . . . . . . . . 159
A.3 Model Lifecycle Phases . . . . . . . . . . . . . . . . . . . . . . 162
A.4 Model Cost Activities . . . . . . . . . . . . . . . . . . . . . . 163
A.5 Model Cost Categories . . . . . . . . . . . . . . . . . . . . . . 164
Contents

1 Introduction 1
1.1 Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Estimation and Metrics Processes . . . . . . . . . . . . . . . 2
1.2.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.2 Usage Scenarios . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Cost Estimating Relationship Approach . . . . . . . . . . . . 5
1.4 Manual Contents . . . . . . . . . . . . . . . . . . . . . . . . . 6

2 Software Resources Data Report 9

2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 Collecting Organization . . . . . . . . . . . . . . . . . . . . . 9
2.3 Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.4 Reporting Frequency . . . . . . . . . . . . . . . . . . . . . . 11
2.4.1 Ramifications . . . . . . . . . . . . . . . . . . . . . . . 11
2.5 SRDR Content . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.5.1 SRDR Data Elements . . . . . . . . . . . . . . . . . . 12
2.5.1.1 Administrative Information . . . . . . . . . . 12
2.5.1.2 Product and Development Description . . . . 13
2.5.1.3 Product Size Reporting . . . . . . . . . . . . 14
2.5.1.4 Resource and Schedule Reporting . . . . . . 15
2.5.1.5 Product Quality Reporting . . . . . . . . . . 16
2.5.1.6 Data Dictionary . . . . . . . . . . . . . . . . 16
2.6 Further SRDR Resources . . . . . . . . . . . . . . . . . . . . 17

3 Metrics Definitions 19
3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.2 Product Size Measures . . . . . . . . . . . . . . . . . . . . . 19
3.2.1 Source Lines of Code (SLOC) . . . . . . . . . . . . . . 19
3.2.1.1 SLOC Type Definitions . . . . . . . . . . . . 19
3.2.1.2 SLOC Counting Rules . . . . . . . . . . . . . 21
3.2.2 Equivalent Size . . . . . . . . . . . . . . . . . . . . . . 23
3.2.2.1 Purpose in Estimating . . . . . . . . . . . . . 24
3.2.2.2 Adapted SLOC Adjustment Factors . . . . . 25
3.2.2.3 Total Equivalent Size . . . . . . . . . . . . . 26
3.2.2.4 Volatility . . . . . . . . . . . . . . . . . . . . 26
3.3 Functional Size Measures . . . . . . . . . . . . . . . . . . . . 26

xv
xvi

3.3.1 Function Points . . . . . . . . . . . . . . . . . . . . . . 26

3.4 Development Effort . . . . . . . . . . . . . . . . . . . . . . . 29
3.4.1 Activities and Lifecycle Phases . . . . . . . . . . . . . 29
3.4.2 Labor Categories . . . . . . . . . . . . . . . . . . . . . 30
3.4.3 Labor Hours . . . . . . . . . . . . . . . . . . . . . . . 31
3.5 Development Duration . . . . . . . . . . . . . . . . . . . . . 31

4 Data Assessment 33
4.1 Gather Collected Data . . . . . . . . . . . . . . . . . . . . . 34
4.2 Inspect each Data Record . . . . . . . . . . . . . . . . . . . . 34
4.3 Determine Data Quality Levels . . . . . . . . . . . . . . . . . 36
4.4 Correct Missing or Questionable Data . . . . . . . . . . . . . 38
4.5 Normalize Size and Effort Data . . . . . . . . . . . . . . . . 38
4.5.1 Converting to Logical SLOC . . . . . . . . . . . . . . 38
4.5.2 Standardizing Effort . . . . . . . . . . . . . . . . . . . 39
4.6 Convert Raw SLOC into Equivalent SLOC . . . . . . . . . . 40
4.7 Future Work and Limitations . . . . . . . . . . . . . . . . . . 42

5 Cost Estimating Relationships 45

5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.2 Data Segmentation . . . . . . . . . . . . . . . . . . . . . . . 46
5.2.1 Operating Environments (OpEnv) . . . . . . . . . . . 46
5.2.2 Application Types (AppType) . . . . . . . . . . . . . 48
5.2.3 MIL-STD-881C WBS Mapping to Application Types 54
5.3 Estimating Relationships . . . . . . . . . . . . . . . . . . . . 56
5.3.1 Scatter Plots . . . . . . . . . . . . . . . . . . . . . . . 57
5.3.2 Data Transformation . . . . . . . . . . . . . . . . . . . 60
5.3.3 CER Models . . . . . . . . . . . . . . . . . . . . . . . 61
5.3.4 CER Calibration . . . . . . . . . . . . . . . . . . . . . 63
5.3.5 CER Evaluation . . . . . . . . . . . . . . . . . . . . . 63
5.3.6 CER Example . . . . . . . . . . . . . . . . . . . . . . 66
5.4 SRDR-Derived CERs . . . . . . . . . . . . . . . . . . . . . . 69
5.4.1 Model Coverage . . . . . . . . . . . . . . . . . . . . . 69
5.4.2 Operating Environment CERs . . . . . . . . . . . . . 70
5.4.2.1 Aerial Vehicle (AV) . . . . . . . . . . . . . . 71
5.4.2.2 Ground Site (GS) . . . . . . . . . . . . . . . 72
5.4.2.3 Ground Vehicle (GV) . . . . . . . . . . . . . 73
5.4.2.4 Maritime Vessel (MV) . . . . . . . . . . . . . 74
5.4.2.5 Ordnance Vehicle (OV) . . . . . . . . . . . . 75
5.4.3 Application Types CERs . . . . . . . . . . . . . . . . 76
5.4.3.1 Command and Control (C&C) . . . . . . . . 77
5.4.3.2 Custom AIS Software (CAS) . . . . . . . . . 78
5.4.3.3 Communication (COM) . . . . . . . . . . . . 79
5.4.3.4 Mission Planning (MP) . . . . . . . . . . . . 80
5.4.3.5 Real Time Embedded (RTE) . . . . . . . . . 81
 xvii

5.4.3.6 Scientific and Simulation (S&S) . . . . . . . 82

5.4.3.7 Sensor Control and Signal Processing (SCP) 83
5.4.3.8 System Software(SYS) . . . . . . . . . . . . . 84
5.4.3.9 Test, Measurement and Diagnostic Equipment
(TMDE) . . . . . . . . . . . . . . . . . . . . 85
5.4.3.10 Vehicle Control (VC) . . . . . . . . . . . . . 86
5.4.3.11 Vehicle Payload (VP) . . . . . . . . . . . . . 87
5.4.4 Application Type and Operating Environment CERs . 88
5.4.4.1 Command & Control - Ground Site . . . . . 89
5.4.4.2 Custom AIS Software - Ground Site . . . . . 90
5.4.4.3 Communications - Ground Site . . . . . . . . 91
5.4.4.4 Communications - Maritime Vessel . . . . . . 92
5.4.4.5 Mission Planning - Ground Site . . . . . . . 93
5.4.4.6 Real Time Embedded - Aerial Vehicle . . . . 94
5.4.4.7 Real Time Embedded - Ground Site . . . . . 95
5.4.4.8 Sensor Control and Signal Processing -
Ground Vehicle . . . . . . . . . . . . . . . . . 96
5.4.4.9 Sensor Control and Signal Processing -
Ground Site . . . . . . . . . . . . . . . . . . 97
5.4.4.10 System Software - Ground Site . . . . . . . . 98
5.4.4.11 Vehicle Control - Aerial Vehicle . . . . . . . 99
5.4.4.12 Vehicle Control - Ground Vehicle . . . . . . . 100
5.4.4.13 Vehicle Payload - Aerial Vehicle . . . . . . . 101
5.4.5 CER Effort Distribution . . . . . . . . . . . . . . . . . 102
5.5 SRDR-Derived Benchmarks . . . . . . . . . . . . . . . . . . . 102
5.5.1 Benchmark Model and Evaluation . . . . . . . . . . . 102
5.5.2 Operating Environment Benchmarks . . . . . . . . . . 103
5.5.3 Application Type Benchmarks . . . . . . . . . . . . . 105
5.5.4 AppType & OpEnv Benchmarks . . . . . . . . . . . . 107
5.6 Limitations and Future Work . . . . . . . . . . . . . . . . . . 108

6 Modern Estimating Challenges 111

6.1 Changing Objectives, Constraints and Priorities . . . . . . . 111
6.1.1 Rapid Change, Emergent Requirements, and Evolution-
ary Development . . . . . . . . . . . . . . . . . . . . . 111
6.1.2 Net-centric Systems of Systems (NCSoS) . . . . . . . 114
6.1.3 Model-Driven and Non-Developmental Item (NDI)-
Intensive Development . . . . . . . . . . . . . . . . . . 115
6.1.4 Ultrahigh Software Systems Assurance . . . . . . . . . 116
6.1.5 Legacy Maintenance and Brownfield Development . . 116
6.1.6 Agile and Kanban Development . . . . . . . . . . . . . 117
6.1.7 Putting It All Together at the Large-Project or
Enterprise-Level . . . . . . . . . . . . . . . . . . . . . 118
6.2 Estimation Approaches for Different Processes . . . . . . . . 119
xviii

7 Estimation Process 125

7.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
7.1.1 Four Estimation Stages . . . . . . . . . . . . . . . . . 126
7.1.1.1 Initiation and Research . . . . . . . . . . . . 128
7.1.1.2 Assessment . . . . . . . . . . . . . . . . . . . 128
7.1.1.3 Analysis . . . . . . . . . . . . . . . . . . . . 129
7.1.1.4 Documentation and Presentation . . . . . . . 129
7.2 Estimation Purpose . . . . . . . . . . . . . . . . . . . . . . . 130
7.3 Program Definition . . . . . . . . . . . . . . . . . . . . . . . 130
7.4 Estimation Scope . . . . . . . . . . . . . . . . . . . . . . . . 131
7.4.1 Ground Rules and Assumptions . . . . . . . . . . . . . 132
7.4.2 Estimation Structure . . . . . . . . . . . . . . . . . . . 132
7.5 Data Collection and Normalization . . . . . . . . . . . . . . 134
7.6 Estimate Creation . . . . . . . . . . . . . . . . . . . . . . . . 135
7.6.1 Historical Reconstruction . . . . . . . . . . . . . . . . 135
7.6.2 Estimate Construction . . . . . . . . . . . . . . . . . . 135
7.6.3 Estimate Cost Analysis . . . . . . . . . . . . . . . . . 136
7.6.4 Alternate Estimate . . . . . . . . . . . . . . . . . . . . 136
7.7 Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . 137
7.8 Risk and Uncertainty Analysis . . . . . . . . . . . . . . . . . 137
7.8.1 Common Probability Distributions . . . . . . . . . . . 138
7.8.2 Monte Carlo Analysis with a CER . . . . . . . . . . . 141
7.9 Estimate Documentation and Packaging . . . . . . . . . . . . 143

Appendix A - Cost Model Descriptions 145

A.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
A.2 Cost Models . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
A.2.1 COCOMO II . . . . . . . . . . . . . . . . . . . . . . . 146
A.2.2 True Planning . . . . . . . . . . . . . . . . . . . . . . 147
A.2.3 SEER-SEM . . . . . . . . . . . . . . . . . . . . . . . . 147
A.2.4 SLIM . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
A.3 Cost Model Input Factors . . . . . . . . . . . . . . . . . . . . 149
A.3.1 Software Size . . . . . . . . . . . . . . . . . . . . . . . 149
A.3.1.1 Overview and Sizing Units . . . . . . . . . . 149
A.3.1.2 New Software . . . . . . . . . . . . . . . . . . 152
A.3.1.3 Adapted Software (Modified and Reused) . . 152
A.3.1.4 Generated Software . . . . . . . . . . . . . . 156
A.3.1.5 Automatically Translated Software . . . . . . 156
A.3.1.6 Commercial Off-The-Shelf Software (COTS) 157
A.3.2 Software Cost Drivers . . . . . . . . . . . . . . . . . . 158
A.4 Cost Model Lifecycles and Work Breakdown Structures . . . 161
 xix

Appendix B - MIL-STD-881C WBS Mapping to Application

Types 165
B.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
B.2 Aerial Vehicle Manned (AVM) . . . . . . . . . . . . . . . . . 166
B.3 Aerial Vehicle Unmanned(AVU) & Ground Site Fixed (GSF) 167
B.4 Ordnance Vehicle Unmanned (OVU)-Ordnance . . . . . . . . 168
B.5 Ordnance Vehicle Unmanned (OVU)-Missile . . . . . . . . . 169
B.6 Ordnance Vehicle Unmanned (OVU)-Launch Vehicles . . . . 171
B.7 Maritime Vessel Manned (MVM) . . . . . . . . . . . . . . . 171
B.8 Maritime Vessel Unmanned (MVU) and Maritime Vessel
Manned (MVM) . . . . . . . . . . . . . . . . . . . . . . . . . 172
B.9 Space Vehicle Manned / Unmanned (SVM/U) & Ground Site
Fixed (GSF) . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
B.10 Ground Site Fixed (GSF) . . . . . . . . . . . . . . . . . . . . 174
B.11 Ground Vehicle Manned & Unmanned(GVM/U) . . . . . . . 175
B.12 Applies to ALL Environments . . . . . . . . . . . . . . . . . 176

Appendix C - CER Nomograms 177

C.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
C.2 Operating Environment CERs . . . . . . . . . . . . . . . . . 178
C.3 CERs by Application Type within Operating Environments . 184
C.3.1 Ground Site Operating Environment . . . . . . . . . . 184
C.3.2 Ground Vehicle Operating Environment . . . . . . . . 192
C.3.3 Aerial Vehicle Operating Environment . . . . . . . . . 194
C.3.4 Maritime Vessel Operating Environment . . . . . . . . 199
C.4 CERs by AppType Across All Environments . . . . . . . . . 201
C.5 Cost Model Nomograms . . . . . . . . . . . . . . . . . . . . . 213
C.5.1 COCOMO . . . . . . . . . . . . . . . . . . . . . . . . 213

Appendix D - Unified CodeCount 217

D.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217

Appendix E - Acronyms 219

Bibliography 223

Index 229
xx
Chapter 1
Introduction

There is no good way to perform a software cost-benefit analysis,

breakeven analysis, or make-or-buy analysis without some reason-
ably accurate method of estimating software costs and their sensi-
tivity to various product, process, personnel, project, and environ-
mental factors.
-Barry Boehm

1.1 Purpose
This metrics manual helps analysts and decision makers develop accurate,
easy and quick early software cost estimates for different types of systems and
operating environments oriented for the United States Department of Defense
(DoD) and government agencies in related domains. The intent is to improve
quality and consistency of early software estimating methods across govern-
ment cost agencies and program offices through guidance, standardization,
and knowledge sharing.
These defense environments may be sea, air, space, or ground-based like the
typical examples in Figure 1.1. We have analyzed empirical software cost data
for these types of programs and are transitioning the results back in this open
access manual. Furthermore we describe our processes for data normalization,
analysis, derivation of Cost Estimating Relationships (CERs) and productivity
benchmarks.
These processes depend on data submittals from executing programs that
adhere to standardized metrics definitions. This submission is the Software
Resources Data Report (SRDR). The responsibility to submit consistent and
complete SRDRs is incumbent on programs for the continuous metrics feed-
back loop to improve software cost estimation in the DoD. This manual il-
lustrates how standardized metrics reporting leads to better predictive cost
models.
These reported metrics are used to derive costs for future programs. This
manual follows the SRDR interpretation of cost as the amount of effort in
Person-Months required to develop the software. This can be used to generate
cost estimates based on time-varying labor rates in cost per person-month.

1
2 Software Cost Estimation Metrics Manual for Defense Systems

FIGURE 1.1: Example Defense Operating Environments

1.2 Estimation and Metrics Processes

The context of this manual within overall related estimation and met-
rics processes in the DoD is shown in Figure 1.2. The Defense Acquisition
Management System (DAMS) at the top consists of five phases and major
milestones A, B and C that DoD programs transition through [38]. Programs
must provide ongoing cost estimates as they proceed through the phases.
SRDR submittals are also required throughout as major software builds are
completed [44]. These metrics data reports are shown as outputs from differ-
ent phases. The submitted SRDRs from contractors are stored in the DoD
Defense Automated Cost Information Management System (DACIMS).
The SRDRs mostly originate in the Engineering and Manufacturing Devel-
opment phase where the majority of software development takes place. Large
prototyping or early development contracts exceeding $20M in Technology De-
velopment also contribute SRDRs. A smaller number originate in Production
and Deployment for large system upgrades or Pre-Planned Product improve-
ments more than $20M. This may also rarely happen in the Operations and
Support phase, and SRDRs are submitted accordingly as the upgrades be-
come programs themselves. Any software development effort exceeding the
$20M threshold must report SRDRs.
 Introduction 3

Defense Acquisition Management System

A B C
Materiel Technology Engineering and Production and Operations and
Solution Development Manufacturing Deployment Support
Analysis Development

SRDRs Simple CER Multi-Parameter

Estimates CER Estimates

Program Estimation Process

Initiation and Documentation

Assessment Analysis
Research and Presentation

DACIMS
Simple CERs Multi-Parameter CERs

Software Cost Estimation

Metrics Process
SRDRs
Cost Model
Metrics Definitions Descriptions

Address
Collect Data Prepare Data Create CERs Challenges

Evolve CERs

FIGURE 1.2: Estimation and Metrics Processes

The Program Estimation Process in Figure 1.2 supports the ongoing pro-
gram estimates depicted by vertical arrows up to the DAMS. These esti-
mates become increasingly detailed with better knowledge over time. Rela-
tively simple CERs are used in early phases because little is known about
the future design and implementation of the software. As a program pro-
ceeds through development, production and deployment, more information
is available so multi-parameter CERs (e.g. detailed cost models) are used to
provide estimates-to-complete and track progress. These CERs incorporating
additional factors correspond to increased knowledge of the program.
The Program Estimation Process has an Assessment activity which in-
cludes deciding the best CER to apply. Available to be used are the simple
CERs derived from SRDRs in this manual, or multi-parameter CERs corre-
sponding to the detailed cost models also described herein.
The Software Cost Estimation Metrics Process described in this manual
operates with both other major processes. It uses the empirical SRDR data
collected from programs transitioning through the DAMS as input data. The
collected SRDRs which are stored in the DACIMS are made available to gov-
ernment cost analysts. This data is then prepared by normalizing it to stan-
4 Software Cost Estimation Metrics Manual for Defense Systems

dardized metric definitions. Simple CERs are created from the data and used
in the Program Estimation Process. The same metrics definitions also underpin
the multi-parameter CERs which support the Program Estimation Process.
Important metrics feedback loops are also illustrated in Figure 1.2. An
overall feedback loop includes the submission of SRDRS to the DACIMS repos-
itory being analyzed for future program estimates. Those programs eventually
execute and provide ongoing data for continuous CER calibration. Within the
software cost estimation metrics process there is a continuous feedback loop
for evolving CERs. This happens due to ongoing analysis of current data com-
bined with new data submittals. The program estimation process is inherently
iterative and the results are fed back to earlier phases in the estimation pro-
cess. The iterations may repeat for successive estimates throughout a project’s
lifecycle.

1.2.1 Dataset
The analyses in this manual are based on 317 SRDRs reported from recent
DoD projects during 2004-2013. That raw data was normalized as described
later in this manual. A top-level view of the normalized dataset across the ser-
vices is shown in Fig. 1.3. Cost is expressed as Person-Months plotted against
software size in Thousand Equivalent Source Lines of Code (KESLOC) pre-
sented in logarithmic scale. This normalized data was then segmented by
application type (described in Chapter 5) to develop the CERs, whereby cost
is predicted as a function of size.

10000

1000
Person-Months

100

. Navy.
10 . Army .
. Air Force
.

1. .
1 10 100 1000
KESLOC

FIGURE 1.3: Normalized Effort vs. Equivalent Size in SRDRs

 Introduction 5

1.2.2 Usage Scenarios

The primary usage of this manual is for government analysts estimating
software development costs or assessing costs in complex systems. They may
be creating an independent cost estimate, or validating and cross-checking
software cost estimates provided by contractors. Existing CERs can provide
sanity checks without full detailed information on the future programs. In
some cases, new CERs need to be developed when existing CERs are in-
adequate. Productivity benchmarks are also useful for comparison purposes.
This manual provides expected software productivities across environments
and applications.
CERs and productivity benchmarks are also important to contractors who
provide SRDR data. In most cases, the people responsible for reporting the
SRDR are the same preparing the cost proposals for source selection. Pub-
licized CERs and benchmarks will help them crosscheck their cost proposal
estimates against government benchmarks.
Defensible estimates are mostly needed at the early conceptual phase of
a software-intensive system’s definition and development. A cost estimation
model with 20-30 input parameters is not very helpful without a defensible
approach for specifying the input values for key parameters as the software’s
complexity, database size, platform volatility, required schedule compression,
tool coverage, or proposed project personnel experience.
This manual provides simple software effort models for early estimates for
these reasons. It examines the direct effect of product size and application
type on cost. The analysis framework and variables used in this study builds
on causal relationships established in past studies.

1.3 Cost Estimating Relationship Approach

This manual describes a bootstrap approach to creating software cost es-
timation relationships based on continuous data analysis. The method follows
these basic steps:

1. Collect software cost metrics data

• The data used in this manual is collected from the DoD’s DACIMS
repository of submitted SRDRs.
2. Prepare the data for analysis (e.g., normalizing different measures of the
same metric)
• This step heavily relies on a set of standard metric definitions
• Data is then normalized to the standard set of metric definitions
3. Create CER models and benchmarks from the data
6 Software Cost Estimation Metrics Manual for Defense Systems

• Segmenting the data into logical groups

• Derive estimation formulae with parameters relating size and ef-
fort.
• Derive productivity benchmarks or ratios
4. Address challenges in using CERs with modern software development
practices.

5. Evolve the CERs based on comparing estimates with actual costs on

future completed systems.

1.4 Manual Contents

The manual generally follows the five steps above as reflected in subse-
quent chapters:

SRDR
The SRDR is discussed first since the collected data on these reports un-
derpins the analysis in this manual. It describes what the data consists of and
when it is reported. It is important to understand the SRDR because this
defines what the CERs cover. This chapter covers:

• Programs that report SRDRs

• Frequency of data collection
• Scope of data collection
• Data elements collected

Metrics Definitions
The next step is preparing the data for analysis and ensuring consistency.
The metrics must be defined in a standard way before doing anything with the
data. This chapter covers baseline definitions used to assess the homogeneity
of the data collected. When the analysis step begins, the data needs to be as
comparable as possible for apples to apples comparison.
Metric definitions also define the inputs and outputs of the CERs. An
awareness of the metric definitions is very important to prevent misuse of the
CERs, and it also supports data collection activities.

Data Assessment
With an understanding of the data sources and metric definitions, this
chapter covers the assessment of the data for use in analysis. Missing data is
removed. The completeness of the data is assessed. Each data record is com-
pared to the metric definitions and, where necessary, data is transformed to
 Introduction 7

be compliant with the metric definition.

CERs and Benchmarks

The cost estimating relationships are derived from the data collected and
normalized. This chapter shows the use of relatively simple statistical methods
to create CERs from the data:

• Data is segmented into two categories: Operating Environment and Ap-

plication Type
• Effort and size data are analyzed for CERs within each group
• Data is analyzed for productivity ratios within the groups

Modern Estimating Challenges

This chapter presents an overview of the challenges in using the CERs
in modern development practices and guidance for dealing with them. These
challenges include:

• Rapid Change, Emergent Requirements, and Evolutionary Development

• Net-Centric System of Systems
• Model-Driven and Non-Development Item-Intensive Development
• Ultrahigh Software Systems Assurance
• Agile and Kanban Development
Estimation Process
This chapter discusses a software cost estimation process following Gov-
ernment Accountability Office (GAO) guidelines. The guidelines are discussed
as they apply to software development cost estimates. Previous chapters in
this manual support different steps in the estimation process, which are ref-
erenced where applicable.

Appendices
A detailed appendix is included on additional parametric cost model de-
scriptions. These cost models are applicable for detailed estimates when more
is known later in the lifecycle. Also included are appendices on MIL-STD-881C
Work Breakdown Structure (WBS) mapping, code counting, nomograms for
each CER, and acronyms found in this manual.
8 Software Cost Estimation Metrics Manual for Defense Systems
Chapter 2
Software Resources Data Report

2.1 Overview
The Software Resources Data Report (SRDR) is used to obtain both the
estimated and actual characteristics of new software developments or upgrades
[44]. Both the Government program office and, after contract award, the soft-
ware contractor submit this report. For contractors, this report constitutes a
contract data deliverable that formalizes the reporting of software metric and
resource data.
SRDR submittals are required for all contractors developing or producing
any software development element with a projected software effort greater
than $20M (then year dollars) on major contracts and subcontracts within
ACAT I and ACAT IA programs, regardless of contract type. The data collec-
tion and reporting applies to developments and upgrades whether performed
under a commercial contract or internally by a government Central Design Ac-
tivity (CDA) under the terms of a Memorandum of Understanding (MOU).
Table 2.1 summarizes the reporting events.

2.2 Collecting Organization

The Defense Cost and Resource Center (DCARC), which is part of OSD
Cost Assessment and Program Evaluation (CAPE), exists to collect Major
Defense Acquisition Program (MDAP) cost and software resource data and
make those data available to authorized Government analysts. Their website is
the authoritative source of information associated with the Cost and Software
Data Reporting (CSDR) system, including but not limited to: policy and
guidance, training materials, and data.

9
10 Software Cost Estimation Metrics Manual for Defense Systems

TABLE 2.1: SRDR Reporting Events

Report
Event Due Who Provides Scope of Report
Pre-Contract Initial Government Estimates of the entire
(180 days prior Program Office completed project. Mea-
to award) sures should reflect cumu-
lative grand totals.

Contract award Initial Contractor Estimates of the entire

project at the level of de-
tail agreed upon. Mea-
sures should reflect cumu-
lative grand totals.

Start of each Initial Contractor Estimates for completion

build for the build only.

Estimate Initial Contractor Corrections to the sub-

corrections mitted estimates.

End of each Final Contractor Actuals for the build only.

build

Contract Final Contractor Actuals for the entire

completion project. Measures should
reflect cumulative grand
totals.

Actuals Final Contractor Corrections to the sub-

corrections mitted actuals.

2.3 Repository
The DCARC’s Defense Automated Cost Information Management System
(DACIMS) is the database for access to current and historical cost and soft-
ware resource data needed to develop independent, substantiated estimates.
DACIMS is a secure website that allows DoD government cost estimators and
analysts to browse through almost 30,000 CCDRs, SRDR and associated doc-
 Software Resources Data Report 11

uments via the Internet. It is the largest repository of DoD cost information.

2.4 Reporting Frequency

The SRDR Final Developer Report contains measurement data as de-
scribed in the contractor’s SRDR Data Dictionary. The data reflects the scope
relevant to the reporting event. Both estimates (DD Form 2630-1,2) and actual
results (DD Form 2630-3) of software (SW) development efforts are reported
for new or upgrade projects. SRDR submissions for contract complete event
shall reflect the entire software development project. When the development
project is divided into multiple product builds, each representing production
level software delivered to the government, the submission should reflect each
product build. SRDR submissions for completion of a product build shall re-
flect size, effort, and schedule of that product build.

2.4.1 Ramifications
It is very important to understand the submission criteria because there
are critical ramifications. SRDR records are a mixture of complete contracts
and individual builds within a contract. These must be distinguished in the
reporting. Furthermore, each has initial and final reports along with correc-
tions. Mixing contract data and build data or mixing initial and final results
or not using the latest corrected version will produce inconclusive and possibly
incorrect results.

2.5 SRDR Content

The SRDR has five sections plus an accompanying Data Dictionary [44].
These sections are:
1. Administrative Information
2. Product and Development Description
3. Product Size Reporting
4. Resource and Schedule Reporting
5. Product Quality Reporting
The Data dictionary contains details on the software that do not match an
SRDR data field, explanations for lack of data or data coverage, and definitions
of the metrics reported.
12 Software Cost Estimation Metrics Manual for Defense Systems

2.5.1 SRDR Data Elements

Below is a list of data elements reported in the SRDR as of the 2011
specification. The highlighted data elements are considered important for
the analysis conducted for this manual. These data elements are from an
Excel spreadsheet implementation of the SRDR Final Report 2011 form [44].

2.5.1.1 Administrative Information (SRDR Section 3.1)

• Security Classification
• Major Program
– Program Name
– Phase / Milestone
– Reporting Organization Type (Prime, Subcontractor, Government)
• Name / Address
– Reporting Organization
– Division
• Approved Plan Number
• Customer (Direct-Reporting Subcontractor Use Only)
• Contract Type
• WBS Element Code
• WBS Reporting Element

• Type Action
– Contract No
– Latest Modification
– Solicitation No
– Common Reference Name
– Task Order / Delivery Order / Lot No
• Period of Performance
– Start Date (YYYYMMDD)
– End Date (YYYYMMDD)
• Appropriation (RDT&E, Procurement, O&M)
• Submission Number
• Resubmission Number
• Report As Of (YYYYMMDD)
• Date Prepared (YYYYMMDD)
• Point of Contact
– Name (Last, First, Middle Initial)
– Department
– Telephone Number (include Area Code)
 Software Resources Data Report 13

– Email
– Development Organization
• Software Process Maturity
– Lead Evaluator
– Certification Date
– Evaluator Affiliation

• Precedents (List up to five similar systems by the same organization or

team.)
• SRDR Data Dictionary Filename
• Comments (on Report Context and Development Organization)

2.5.1.2 Product and Development Description (SRDR Section 3.2)

• Functional Description. A brief description of its function.
• Software Development Characterization
• Application Type
– Primary and Secondary Programming Language.
– Percent of Overall Product Size. Approximate percentage (up to
100%) of the product size that is of this application type.
– Actual Development Process. Enter the name of the development
process followed for the development of the system.
– Software Development Method(s). Identify the software develop-
ment method or methods used to design and develop the software
product .
– Upgrade or New Development. Indicate whether the primary
development was new software or an upgrade.
– Software Reuse. Identify by name and briefly describe software
products reused from prior development efforts (e.g. source code,
software designs, requirements documentation, etc.).

• COTS / GOTS Applications Used

– Name. List the names of the applications or products that consti-
tute part of the final delivered product, whether they are COTS,
GOTS, or open-source products.
– Integration Effort (Optional). If requested by the CWIPT, the SRD
report shall contain the actual effort required to integrate each
COTS / GOTS application identified in Section 3.2.4.1.
• Staffing
– Peak Staff. The actual peak team size, measured in full-time equiv-
alent (FTE) staff.
– Peak Staff Date. Enter the date when the actual peak staffing oc-
curred.
14 Software Cost Estimation Metrics Manual for Defense Systems

– Hours per Staff-Month. Enter the number of direct labor hours per
staff-month.
• Personnel Experience in Domain. Stratify the project staff domain
experience by experience level and specify the percentage of project staff
at each experience level identified. Sample Format 3 identifies five levels:
– Very Highly Experienced (12 or more years)
– Highly Experienced (6 to 12 years)
– Nominally Experienced (3 to 6 years)
– Low Experience (1 to 3 years)
– Inexperienced / Entry Level (less than a year)

2.5.1.3 Product Size Reporting (SRDR Section 3.3)

• Number of Software Requirements. Provide the actual number of
software requirements.
– Total Requirements. Enter the actual number of total requirements
satisfied by the developed software product at the completion of the
increment or project.
– New Requirements. Of the total actual number of requirements
reported, identify how many are new requirements.
• Number of External Interface Requirements. Provide the number
of external interface requirements, as specified below, not under project
control that the developed system satisfies.
– Total External Interface Requirements. Enter the actual number
of total external interface requirements satisfied by the developed
software product at the completion of the increment or project.
– New External Interface Requirements. Of the total number of ex-
ternal interface requirements reported, identify how many are new
external interface requirements.
• Requirements Volatility. Indicate the amount of requirements
volatility encountered during development as a percentage of require-
ments that changed since the Software Requirements Review.

• Software Size.
– Delivered Size. Capture the delivered size of the product devel-
oped, not including any code that was needed to assist development
but was not delivered (such as temporary stubs, test scaffoldings,
or debug statements). Additionally, the code shall be partitioned
(exhaustive with no overlaps) into appropriate development cate-
gories. A common set of software development categories is new,
reused with modification, reused without modification, carry-over
code, deleted code, and auto-generated code.
 Software Resources Data Report 15

– Reused Code With Modification. When code is included that

was reused with modification, provide an assessment of the amount
of redesign, recode, and retest required to implement the modified
or reused code.
– Reuse Code Without Modification. Code reused without mod-
ification is code that has no design or code modifications. However,
there may be an amount of retest required. Percentage of retest
should be reported with the retest factors described above.
– Carryover Code. Report shall distinguish between code devel-
oped in previous increments that is carried forward into the cur-
rent increment and code added as part of the effort on the current
increment.
– Deleted Code. Include the amount of delivered code that was
created and subsequently deleted from the final delivered code.
– Auto-generated Code. If the developed software contains auto-
generated source code, report an auto-generated code sizing parti-
tion as part of the set of development categories.
– Subcontractor-Developed Code.
– Counting Convention. Identify the counting convention used to
count software size.
– Size Reporting by Programming Language (Optional).
– Standardized Code Counting (Optional). If requested, the con-
tractor shall use a publicly available and documented code counting
tool, such as the University of Southern California Code Count tool,
to obtain a set of standardized code counts that reflect logical size.
These results shall be used to report software sizing.

2.5.1.4 Resource and Schedule Reporting (SRDR Section 3.4)

The Final Developer Report shall contain actual schedules and actual total
effort for each software development activity.

• Effort. The units of measure for software development effort shall be

reported in staff-hours. Effort shall be partitioned into discrete software
development activities.
• WBS Mapping.
• Subcontractor Development Effort. The effort data in the SRDR report
shall be separated into a minimum of two discrete categories and re-
ported separately: Prime Contractor Only and All Other Subcontrac-
tors.
• Schedule. For each software development activity reported, provide the
actual start and end dates for that activity.
16 Software Cost Estimation Metrics Manual for Defense Systems

The activities referred to in the SRDR are based on the ISO/IEC-12207

standard [25].The possible development activities reported include:
• Software Requirements Analysis
• Software Architecture and Detailed Design
• Software Coding and Unit Testing
• Software Integration
• Software Qualification Testing
• System/Software Qualification Testing
• Software Quality Assurance
• Software Configuration Management
• Software Program Management
The current SRDR data has the top five items in addition to an activity
called Development Test and Evaluation. All of the other activities are lumped
under a category called Other.

2.5.1.5 Product Quality Reporting (SRDR Section 3.5 - Optional)

Quality should be quantified operationally (through failure rate and de-
fect discovery rate). However, other methods may be used if appropriately
explained in the associated SRDR Data Dictionary.
• Number of Defects Discovered. Report an estimated number of de-
fects discovered during integration and qualification testing. If available,
list the expected defect discovery counts by priority, e.g. 1, 2, 3, 4, 5.
Provide a description of the priority levels if used.
• Number of Defects Removed. Report an estimated number of de-
fects removed during integration and qualification testing. If available,
list the defect removal counts by priority

2.5.1.6 Data Dictionary

The SRDR Data Dictionary contains, at a minimum, the following infor-
mation in addition to the specific requirements identified in SRDR Sections
3.1 through 3.5:
• Experience Levels. Provide the contractor’s specific definition (i.e.,
the number of years of experience) for personnel experience levels re-
ported in the SRDR report.
• Software Size Definitions. Provide the contractor’s specific internal
rules used to count software code size.
• Software Size Categories. For each software size category identified
(i.e., New, Modified, Unmodified, etc.), provide the contractor’s specific
rules and / or tools used for classifying code into each category.
• Peak Staffing. Provide a definition that describes what activities were
included in peak staffing.
 Software Resources Data Report 17

• Requirements Count (Internal). Provide the contractor’s specific

rules and / or tools used to count requirements.
• Requirements Count (External). Provide the contractor’s specific
rules and / or tools used to count external interface requirements.
• Requirements Volatility. Provide the contractor’s internal definitions
used for classifying requirements volatility.
• Software Development Activities. Provide the contractor’s inter-
nal definitions of labor categories and activities included in the SRDR
report’s software activity.
• Product Quality Reporting. Provide the contractor’s internal defi-
nitions for product quality metrics being reported and specific rules and
/ or tools used to count the metrics.

2.6 Further SRDR Resources

An Excel spreadsheet implementation of the SRDR Final Report (2011
form) is on this book’s web site. Supplemental analysis of SRDR data is avail-
able through the DoD sponsored Cyber Security and Information Systems
Information Analysis Center (CSIAC) [50]. See the DCARC website [43] for
the DoD’s CSDR overview and policy.
18 Software Cost Estimation Metrics Manual for Defense Systems
Chapter 3
Metrics Definitions

3.1 Overview
This chapter defines software product size measures to support data consis-
tency and interpretation, used as input to cost estimation models and produc-
tivity analyses. These measures are used throughout the manual including for
data normalization, to derive CERs, productivity benchmarks and interpreted
for selected cost models.

3.2 Product Size Measures

An accurate size estimate is the most important input to parametric cost
models. However, determining size can be challenging. Projects may be com-
posed of new code, code adapted from other sources with or without modifi-
cations, automatically generated or translated code, commercial software, and
other types.

3.2.1 Source Lines of Code (SLOC)

The common measure of software size used in this manual and the cost
models is Source Lines of Code (SLOC). SLOC are logical source statements
consisting of data declarations and executables.

3.2.1.1 SLOC Type Definitions

The core software size type definitions used throughout this manual are
summarized in Table 3.1. These definitions apply to size estimation, data
collection, and analysis. To prevent confusion in reporting measures of size
and in storing results in databases, the type of SLOC count should always be
recorded.
The size types are applied at the source code file level for the appropriate
system-of-interest. If a component, or module, has just a few lines of code

19
20 Software Cost Estimation Metrics Manual for Defense Systems

TABLE 3.1: Software Size Types

Size Type Description

New New software created for the first time.
Adapted Pre-existing software that is used as-is (Reused) or changed
(Modified).
Reused Pre-existing software that is not changed with the adaption
parameter settings:
• Design Modification % (DM) = 0%
• Code Modification % (CM) = 0%.
Modified Pre-existing software that is modified for use by making
design, code and / or test changes:
• Code Modification % (CM) > 0%.
Equivalent A relative measure of the work done to produce software
compared to the code-counted size of the delivered software.
It adjusts the size of adapted software relative to developing
it all new.
Generated Software created with automated source code generators.
The code to include for equivalent size consists of auto-
mated tool generated statements.
Converted Software that is converted between languages using auto-
mated translators.
Commercial Pre-built commercially available software components. The
Off-The- source code is not available to application developers. It is
Shelf not included for equivalent size.
(COTS)
Other unmodified software not included in equiva-
lent size are Government Furnished Software (GFS),
libraries, operating systems and utilities.

changed then the entire component is classified as Modified even though most
of the lines remain unchanged. The total product size for the component will
include all lines.
COTS normally isn’t reported in the delivered size. However, COTS glue
code is often necessary. It should be included and used for estimation.
Open source software is handled, as with other categories of software,
depending on the context of its usage. If it is not touched at all by the devel-
opment team it can be treated as a form of COTS or reused code. However,
when open source is modified it must be quantified with the adaptation pa-
rameters for modified code and be added to the equivalent size. The costs of
integrating open source with other software components should be added into
overall project costs.
 Metrics Definitions 21

3.2.1.2 SLOC Counting Rules

Logical Lines
The common measure of software size used in this manual and the cost
models is Source Lines of Code (SLOC). SLOC are logical source statements
consisting of data declarations and executables. Table 3.2 shows the SLOC
definition inclusion rules for what to count. Based on the Software Engineer-
ing Institute’s Software Size Measurement: A Framework for Counting Source
Statements [47], each checkmark in the Includes column identifies a particular
statement type or attribute included in the definition, and vice-versa for the
Excludes. See the following details on the software size terms.

Non-Commented Source Statements (NCSS)

The Non-Commented Source Statement count type only counts lines con-
taining a programming language source statement. No blank lines or comment-
only lines are counted.

Physical Lines
The Physical SLOC count type is a count type where programming lan-
guage terminators or delimiters are counted. This count type excludes blank
lines in a source code file and includes everything else.

Total Lines
The Total SLOC count type includes a count of everything, including blank
lines.

Executables
Executable statements cause runtime actions. They may be simple state-
ments such as assignments, gotos, procedure calls, macro calls, returns, breaks,
exits, ends, stops, continues, nulls, no-ops, or empty statements. Or they may
be structured or compound statements, such as conditional statements, repet-
itive statements, and ”with” statements. Languages like Ada, C, C++, and
Pascal have block statements [begin ... end and {...}] that are classified as
executable when used where other executable statements would be permitted.
C and C++ define expressions as executable statements when they terminate
with a semicolon, and C++ has a declaration statement that is executable.

Declarations
Declarations are non-executable program elements that affect an assem-
bler’s or compiler’s interpretation of other program elements. They are used
to name, define, and initialize; to specify internal and external interfaces; to
assign ranges for bounds checking; and to identify and bound modules and
sections of code. Examples include declarations of names, numbers, constants,
objects, types, subtypes, programs, subprograms, tasks, exceptions, packages,
generics, macros, and deferred constants.
22 Software Cost Estimation Metrics Manual for Defense Systems

TABLE 3.2: SLOC Count Definitions

Logical NCSS Physical Total

Statement Type In Out In Out In Out In Out
Executable X X X X

Nonexecutable
Declarations X X X X
Compiler directives X X X X
Comments X X X X
Blank lines X X X X

Flow Control
Logical expressions X X X X
If - Then - Else X X X X
Go-To, Break, Continue X X X X
Exception Handling X X X X
Flow Control Punctuation X X X X

Block Statements
Subprogram Label & X X X X
Delimiters
Procedure / Function X X X X
Label & Delimiters
Begin-End Blocks or X X X X
Punctuation
Block Punctuation X X X X

How Produced
Programmed New X X X X
Reused X X X X
Modified X X X X
Generated
Generator X X X X
statements
3GL generated
statements
In Maintenance X X X X
In Development X X X X
Converted X X X X

(continued )
 Metrics Definitions 23

TABLE 3.2: SLOC Count Definitions (continued )

Logical NCSS Physical Total

Statement Type In Out In Out In Out In Out
Origin
New X X X X
Adapted
A previous version, X X X X
build, or release
Unmodified COTS, X X X X
GFS, library, OS or
utility

Declarations also include renaming declarations, use clauses, and decla-

rations that instantiate generics. Mandatory begin .. end and {...} symbols
that delimit bodies of programs and subprograms are integral parts of pro-
gram and subprogram declarations. Language superstructure elements that
establish boundaries for different sections of source code are also declarations.
Examples include terms such as PROCEDURE DIVISION, DATA DIVISION,
DECLARATIVES, END DECLARATIVES, INTERFACE, IMPLEMENTA-
TION, SYS-PROC, and SYS-DD. Declarations, in general, are never required
by language specifications to initiate runtime actions, although some languages
permit compilers to implement them that way.

Compiler Directives
Compiler directives instruct compilers, preprocessors, or translators (but
not runtime systems) to perform special actions. Some, such as Adas
PRAGMA and COBOLs COPY, REPLACE, and USE, are integral parts
of the source language. In other languages like C and C++, special symbols
like # are used along with standardized keywords to direct preprocessor or
compiler actions. Still other languages rely on non-standardized methods sup-
plied by compiler vendors. In these languages, directives are often designated
by special symbols such as #, $, and {$}.

3.2.2 Equivalent Size

A key element in using software size for effort estimation is the concept
of equivalent size. Equivalent size is a quantification of the effort required
to use previously existing code along with new code. For cost estimating rela-
tionships, the size of previously existing code does not require the same effort
as the effort to develop new code of the same size. Equivalent size normalizes
the effort relative to new software.
The guidelines in this section will help the estimator in determining the
total equivalent size. All of the models described in Appendix A have tools for
24 Software Cost Estimation Metrics Manual for Defense Systems

doing this. However, for non-traditional size categories (e.g., a model may not
provide inputs for auto-generated code), this manual will help the estimator
calculate equivalent size outside of the tool and incorporate the size as part
of the total equivalent size.

3.2.2.1 Purpose in Estimating

In addition to newly developed software, adapted software that is modified
and reused from another source and used in the product under development
also contributes to the product’s equivalent size. A method is used to make
new and adapted code equivalent so they can be rolled up into an aggregate
size estimate.
The size of reused and modified code is adjusted to be its equivalent
in new code for use in estimation models. The adjusted code size is called
Equivalent Source Lines of Code (ESLOC). The adjustment is based on the
additional effort it takes to modify the code for inclusion in the product taking
into account the amount of design, code and testing that was changed and as
described next.
There are also different ways to produce software that complicate deriving
ESLOC including generated and converted software. All of the categories are
aggregated for equivalent size. A primary source for the equivalent sizing
principles in this section is Chapter 9 of [53].
For traditional Third Generation Language (3GL) software such as C
or Java, count the logical 3GL statements. For Model-Driven Development
(MDD), Very High Level Languages (VHLL), or macro-based development,
count the generated statements. A summary of what to include or exclude in
ESLOC for estimation purposes is in Table 3.3.
 Metrics Definitions 25

TABLE 3.3: ESLOC Summary

Source Includes Excludes

New X
Reused X
Modified X
Generated
Generator statements X
3GL generated statements X
Converted X
COTS X
Volatility X

3.2.2.2 Adapted SLOC Adjustment Factors

The AAF factor is applied to the size of the adapted software to get its
equivalent size. The normal Adaptation Adjustment Factor (AAF) is com-
puted as:

AAF = (0.4 · DM ) + (0.3 · CM ) + (0.3 · IM ) (3.1)

where
% Design Modified (DM)
The percentage of the adapted software’s design which is modified in
order to adapt it to the new objectives and environment. This can be
a measure of design elements changed such as UML descriptions.
% Design Modified (DM)
The percentage of the adapted software’s design which is modified in
order to adapt it to the new objectives and environment. This can be
a measure of design elements changed such as UML descriptions.
% Code Modified (CM)
The percentage of the adapted software’s code which is modified in
order to adapt it to the new objectives and environment. Code count-
ing tools can be used to measure CM. See the chapter on the Unified
Code Count tool in Appendix 9.2 for its capabilities, sample output
and access to it.
% Integration Required (IM)
The percentage of effort required to integrate the adapted software
into an overall product and to test the resulting product as compared
to the normal amount of integration and test effort for software of
comparable size.
Reused software has DM = CM = 0. IM is not applied to the total size of
the reused software, but to the size of the other software directly interacting
26 Software Cost Estimation Metrics Manual for Defense Systems

with it. It is frequently estimated using a percentage. Modified software has

CM > 0.

3.2.2.3 Total Equivalent Size

Using the AAF to adjust Adapted Code size, the total equivalent size is:

T otal Equivalent Size = N ew Size + (AAF · Adapted Size) (3.2)

AAF assumes a linear effort relationship, but there can also be nonlinear ef-
fects. Data indicates that the AAF factor tends to underestimate modification
effort [51] [8] [53]. Two other factors used to account for these effects are Soft-
ware Understanding and Programmer Unfamiliarity. These two factors and
their usage are discussed in Appendix A.

3.2.2.4 Volatility
Volatility is requirements evolution and change, but not code thrown out.
To account for the added effort, volatility is expressed as an additional per-
centage to size to obtain the total equivalent size for estimation.

T otal Equivalent Size = N ew Size + (AAF · Adapted Size) · (1 + V olatility)

(3.3)

3.3 Functional Size Measures

3.3.1 Function Points
The function point cost estimation approach is based on the amount of
functionality in a software project and a set of project factors [22]. Function
points measure software size by quantifying the information processing func-
tionality associated with major external data or control input, output, or file
types. Five user function types should be identified as defined in Table 3.4.
Each instance of these function types is then classified by complexity level.
The complexity levels determine a set of weights, which are applied to their
corresponding function counts to determine the Unadjusted Function Points
(UFP) quantity. The steps for counting UFPs are listed below.

1. Determine function counts by type. The unadjusted function counts

should be counted by a lead technical person based on information in the
software requirements and design documents. The number of each of the
five user function types should be counted (Internal Logical File (ILF),
External Interface File (EIF), External Input (EI), External Output
 Metrics Definitions 27

TABLE 3.4: User Function Types

Function Point Description

External Input (EI) Count each unique user data or user con-
trol input type that enters the external
boundary of the software system being
measured.

External Output (EO) Count each unique user data or con-

trol output type that leaves the exter-
nal boundary of the software system being
measured.

Internal Logical File (ILF) Count each major logical group of user
data or control information in the soft-
ware system as a logical internal file type.
Include each logical file (e.g., each logical
group of data) that is generated, used, or
maintained by the software system.

External Interface Files (EIF) Files passed or shared between software

systems should be counted as external in-
terface file types within each system.

External Inquiry (EQ) Count each unique input-output combina-

tion, where input causes and generates an
immediate output, as an external inquiry
type.

(EO), and External Inquiry (EQ)). See [22] for more detailed interpre-
tations of the counting rules for those quantities.
2. Determine complexity-level function counts. Classify each function count
into Low, Average and High complexity levels depending on the number
of data element types contained and the number of file types referenced.
The weights in Tables 3.5, 3.6 and 3.7 are used.
3. Apply complexity weights. Weight the number of function types at each
complexity level using the the weights reflecting relative effort required
to implement the function in Table 3.8.
4. Compute Unadjusted Function Points. Add all the weighted functions
counts to get one number, the Unadjusted Function Points.
The usual Function Point procedure involves assessing the degree of influ-
ence of fourteen application characteristics on the software project determined
28 Software Cost Estimation Metrics Manual for Defense Systems

TABLE 3.5: Function Point Counting Weights for Internal Logical Files and
External Interface Files
Record Data Elements
Elements 1 - 19 20 - 50 51+
1 Low Low Avg.
2-5 Low Avg. High
6+ Avg. High High

TABLE 3.6: Function Point Counting Weights for External Outputs and
External Inquiries

Data Elements
File Types 1 - 5 6 - 19 20+
0 or 1 Low Low Avg.
2-3 Low Avg. High
4+ Avg. High High

TABLE 3.7: Function Point Counting Weights for External Inputs

Data Elements
File Types 1 - 4 5 - 15 16+
0 or 1 Low Low Avg.
2-3 Low Avg. High
3+ Avg. High High

according to a rating scale of 0.0 to 0.05 for each characteristic. The 14 ratings
are added together, and added to a base level of 0.65 to produce a general
characteristic adjustment factor that ranges from 0.65 to 1.35. This is used as
a multiplier with UFP to obtain the Adjusted Function Point (AFP) count.
For consistency, not all of the cost models use this adjustment because they
already include cost factors for these.
The cost estimation models based on lines of code require conversion of
function points into to lines. This is also called backfiring. Conversion, or
backfiring ratios for different programming languages can be determined with
available historical data. It is recommended to determine your own ratios for
your local environment. Some sources for programming language conversion
factors include [19] and [22].
 Metrics Definitions 29

TABLE 3.8: Unadjusted Function Point Complexity Weights

Complexity Weight
Function Type Low Average High
Internal Logical Files 7 10 15
External Interfaces Files 5 7 10
External Inputs 3 4 6
External Outputs 4 5 7
External Inquiries 3 4 6

3.4 Development Effort

3.4.1 Activities and Lifecycle Phases
Software development involves much more activity than just coding. It
includes the work involved in developing requirements, designs and tests. It
involves documentation and reviews, configuration management, and quality
assurance. It can be done using different life cycles and different ways of
organizing the work (matrix, product lines, etc.). Using the SRDR as the
basis, the following work activities/phases are included or excluded for effort.

TABLE 3.9: Effort Activities

Activity Includes Excludes
System Conceptualization X
Systems Requirements Development X
Software Requirements Analysis X
Software Architecture and Detailed Design X
Software Coding and Unit Test X
Software and System Integration X
Hardware / Software Integration and Test X
System Test and Evaluation X
Operational Test and Evaluation X
Production X

Software requirements analysis includes any prototyping activities. The

excluded activities are normally supported by software personnel but are con-
sidered outside the scope of their responsibility for effort measurement. Sys-
tems Requirements Development includes equations engineering (for derived
requirements) and allocation to hardware and software.
All these activities include the effort involved in documenting, reviewing
30 Software Cost Estimation Metrics Manual for Defense Systems

TABLE 3.10: Effort Phases

Phase Includes Excludes
Inception X
Elaboration X
Construction X
Transition X

and managing the work-in-process. These include any prototyping and the
conduct of demonstrations during the development.
Transition to operations and operations and support activities are not
addressed by these analyses for the following reasons:

• They are normally accomplished by different organizations or teams.

• They are separately funded using different categories of money within
the DoD.
• The cost data collected by projects therefore does not include them
within their scope.

From a life cycle point-of-view, the activities comprising the software life
cycle are represented for new, adapted, reused, generated and COTS (Com-
mercial Off-The-Shelf) developments. Reconciling the effort associated with
the activities in the Work Breakdown Structure (WBS) across life cycle is
necessary for valid comparisons to be made between results from cost models.

3.4.2 Labor Categories

The labor categories included or excluded from effort measurement are
another source of cost data variation. The categories are a decomposition of
effort. Most large software projects have staff fulfilling the functions of:

• Project Managers
• Application Analysts
• Implementation Designers
• Programmers
• Testers
• Quality Assurance personnel
• Configuration Management personnel
• Librarians
• Database Administrators
• Documentation Specialists
• Training personnel
• Other support staff
 Metrics Definitions 31

Adding to the complexity of measuring what is included in effort data is

that staff could be fulltime or part time and charge their hours as direct or
indirect labor. The issue of capturing overtime is also a confounding factor in
data capture.

3.4.3 Labor Hours

Labor hours, also called Staff Hours or Person-Hours, is the recommended
form of measuring software development effort. This measure can be trans-
formed into Labor Weeks, Labor Months and Labor Years. For modeling pur-
poses, when weeks, months or years is required, choose a standard and use it
consistently, e.g. 152 labor hours in a labor month.
If data is reported in units other than hours, additional information is re-
quired to ensure the data is normalized. Each reporting Organization may use
different amounts of hours in defining a labor week, month or year. For what-
ever unit being reported, be sure to also record the Organization’s definition
for hours in a week, month or year. See the Software Engineering Institute’s
Framework for Counting Staff-Hours and Reporting Schedule Information [21]
for a more detailed discussion.

3.5 Development Duration

Schedule data are the start and end date for different development phases,
such as those shown in Table 3.10. Another important aspect of schedule data
is entry or start and exit or completion criteria each phase. The criteria could
vary between projects depending on its definition. As an example of exit or
completion criteria, are the dates reported when:

• Internal reviews are complete

• Formal review with the customer is complete
• Sign-off by the customer
• All high-priority actions items are closed
• All action items are closed
• Products of the activity / phase are placed under configuration manage-
ment
• Inspection of the products are signed-off by QA
• Management sign-off

An in-depth discussion is provided in the Software Engineering Institute’s

Framework for Counting Staff-Hours and Reporting Schedule Information [47].
32 Software Cost Estimation Metrics Manual for Defense Systems
Chapter 4
Data Assessment

This chapter discusses transforming software engineering cost data into useful
information for use in creating Cost Estimating Relationships (CERs) and
productivity benchmarks for use in cost estimation and management over-
sight. There can be many challenges encountered when preparing cost data
for analysis. The list below shows common problems with cost datasets:

• Inadequate information on modified code (only size provided)

• Inadequate information on size change or growth
• Size measured inconsistently
• Inadequate information on average staffing or peak staffing
• Inadequate or ineffective information on personnel experience
• Inaccurate effort data in multi-build components
• Missing effort data for all software activities
• Replicated duration (start and end dates) across components
• Inadequate information on schedule compression
• Missing schedule data
• No quality (defect) data

The remedy for some of these challenges is to find a way to normalize the
data to the definitions discussed in 3. Other techniques are required to fill in
missing data, either by consulting other sources or using statistical means or
medians to fill in missing values. A process workflow was developed to assess
the data and make it usable.
The data assessment and processing workflow has six steps. This workflow
was used in the analysis of the SRDR data by the contributors to this manual.
Each of these steps is described in detail in subsequent sections.

1. Gather Collected Data

2. Inspect each Data Record
3. Determine Data Quality Levels.
4. Correct Missing or Questionable Data
5. Normalize Size and Effort Data
6. Convert Raw SLOC to Equivalent SLOC

33
34 Software Cost Estimation Metrics Manual for Defense Systems

4.1 Gather Collected Data

The SRDR has evolved over time. There is required data in a SRDR that is
not relevant for cost estimation analysis. Likewise, there is missing contextual
data that is needed for analysis. The SRDR data necessary for cost estimation
analysis requires detail information on:

• Amount of workload (expressed as source lines of code)

• Development and support effort effort
• Project or build duration

Additional contextual data needed for analysis are:

• What does the application do, i.e., a functional description?

• Does the record contain estimated data?
• Does the record represent aggregated data or does it represent a single
Computer Software Configuration Item (CSCI)?
• Where in the acquisition lifecycle the software was developed, e.g., tech-
nology demonstration, engineering and manufacturing, operations and
sustainment?
• Are there extenuating circumstances during software development that
would impact size, effort or schedule?

4.2 Inspect each Data Record

The SRDR data that is relevant for data analysis is inspected for com-
pleteness, integrity, and reasonable-ness. The first activity is to examine the
project context information.

Project Context

• Is this record a roll-up of multiple CSCIs or a single item?

• Is this record an aggregation of multiple builds or a single build?
• How would this software component be characterized?
– What does this component do?
– Were there any extenuating circumstances concerning development,
e.g. management change, large requirements change, stop / restart
work?
– Is the Data Dictionary available for this record?
– Is there any additional information that can be consulted about
the data during analysis, such as:
 Data Assessment 35

∗ Acquisition Strategy
∗ Acquisition Support Plan (ASP)
∗ Contract Plan
∗ Cost Analysis Requirements Document (CARD)
∗ Capability Description Document (CDD)
∗ Software Requirements Specification (SRS)
∗ Work Breakdown Structure (WBS)
∗ Earned Value Management System data (EVMS)
∗ Software Development Plan (SDP)
∗ System Engineering Plan (SEP)

Next, the size, effort, schedule and productivity data for each record are
examined.

Size Data

• Does the size data look sound?

– Does the size data match the record description, e.g. one million
SLOC size may indicate program level data with multiple CSCIs?
– Is the size part of multiple builds or releases, e.g. is the size data
an aggregated number?
– Was much or all software auto-generated? This will result in an
extraordinary productivity unless converted to an equivalent size
as discussed in 4.6.
– Was code rewritten after being auto-generated? This means the
auto-code was modified. It should be treated as modified code in
this case.
• Was a portion of a legacy system included in the sizing data?
– How much software was adapted (modified)?
– How much software was reused (no changes)?
• Is there repeating size data? If there is repeating size data, this record
may be an update of another record.

Effort Data
The effort data are for the activities as discussed in Section 2.5.1.4.

• What labor was included in the reported hours?

– Engineering labor
– Management labor
– Support labor: CM, QA, Process Improvement, Safety, Security,
Dev. Environment support
• What labor was reported in the ”Other” activity?
– Was requirement analysis effort reported for all builds?
– Were there continuous integration activities across all builds?
36 Software Cost Estimation Metrics Manual for Defense Systems

Schedule Data
• Was there schedule compression mentioned on the project?
• Were there multiple parallel builds (same start and end date)?
Productivity Screening
Productivity is the ratio of size to effort as detailed in Section 5.5.1.
• Is a check of the productivity reasonably close to software with sim-
ilar functionality? Productivities for similar software functionality are
grouped by Application Type (discussed in Section 5.5.3).
• Is this record an outlier in a scatter plot with other similar data? This
check can only be done if you have access to the SRDR for the appro-
priate Application Type.

4.3 Determine Data Quality Levels

From the inspection process, assign the SRDR record a data quality rating.
The ratings range from 1.0, Best, to 0.0, Worst. The criteria in Table 4.1 can
be used to determine rating values based on:
1. Size - counts exist for the different code types as discussed in Table 3.1
(providing size data is present).
2. Size Count Type - the rules for counting the code are explicit as discussed
in Section 3.2.1.2.
3. ESLOC Factors - the ESLOC factors for Modified, Reused and Auto-
Generated code exist as discussed in Section 3.2.2.2.
4. CSCI-level Data - the data is clearly for one CSCI or it is roll-up data.
5. Effort - effort hours exist for all activities as discussed in Section 2.5.1.4.
6. Schedule - activity duration data exists for same activities as effort.
7. Productivity - productivity is within range of similar software CSCIs.
As each record is rated by the criteria in Table 4.1, an overall quality level
is assigned by:
Quality Level = (Size + Size Count T ype + ESLOC f actor +
CSCI level + Ef f ort + Schedule + P roductivity)/7.
The quality level is a quick indicator of the degree of issues found in the
record. As the record is corrected through supplemental information, the rat-
ing is revised. As the rating value decreases, the record becomes less useful for
CER and productivity analysis. If all size or effort data is missing, the record
cannot be used.
Because the range of the quality level scale is between 0.0 and 1.0, it could
be used as a weight during analysis.
 Data Assessment 37

TABLE 4.1: Data Quality Rating Scale

Attribute Value Condition

Size 1 Size data present
0 No size data
Size Count Type 1 Size is Logical SLOC
0.7 Size is Non-Commented Source Statements
0.5 Size is Physical Lines (Comment and Source
Statements)
0.4 Size is Total Lines (all lines in file: blank,
comment, source)
0 No size data
ESLOC Factors 1 Modification factors are provided for Auto-
(See Section 3.2.2.2) Gen, Modified & Reuse code counts from
outside sources (sometimes found in the Data
Dictionary or after contacting the Data Sub-
mitter)
0.75 New SLOC and no size data for Auto-Gen,
Modified or Reuse (this could happen but it
is rare)
0 No modification factors are available for ei-
ther Modified, Auto-Gen, or Reused SLOC
counts (including the guidance provided in
Table 4.3)
CSCI-level Data 1 5,000 <Equivalent SLOC <250,000
0 Equivalent SLOC <5,000 or Equivalent
SLOC >250,000 (if greater than 250,000 may
be an indicator of program-level data)
Effort 1 Effort reported for all phases (as discussed in
Section 2.5.1.4)
0.5 Effort is reported as a total (all hours aggre-
gated)
0 Effort is missing for a phase
Schedule 1 Duration reported for all phases (as dis-
cussed in Section 2.5.1.4)
0.75 Duration is reported as a total (all durations
aggregated)
0.5 Duration is missing for one phase
0 Duration data is missing
Productivity 1 Record is within 1 standard deviation from
the mean (see Section 5.5.3 for examples)
0.5 Record is within between 1 and 3 standard
deviations from the mean
0 Record is a clear outlier
38 Software Cost Estimation Metrics Manual for Defense Systems

4.4 Correct Missing or Questionable Data

The quality level identifies SRDR records that need additional work. There
are several approaches available to resolving missing or questionable data.
These are listed in a recommended order:
1. Consult the accompanying SRDR Data Dictionary discussed in Chapter
2.
2. Consult any supplemental information on the project that is available,
e.g., ASP, CARD, CDD, EVMS, SRS, WBS, SDP, SEP, etc.
3. Scheduling follow-up meetings with SRDR data contributor.
SRDR data quality issues that were fixed in the past by the SRDR con-
tributor may include:
• Revised missing size, effort and duration data
• Obtained ESLOC factors
• Confirmed Application Type and Operating Environment
• Confirmed CSCI-level of reporting
• Asked about problems with - high / low, long / short - size, effort and
duration data
As a result of inspecting the data and attempting to correct the issues
found, no ”bad” data or outliers are excluded from the analysis on arbitrary
grounds. However, data with issues that cannot be resolved are excluded from
analysis.

4.5 Normalize Size and Effort Data

Normalizing data is making a type of data the same. For example, if SLOC
was measured by different criteria, all SLOC counts are converted into a com-
mon count method. If effort data covers different lifecycle phases, all effort
data is converted to cover the same phases. Normalization reduces noise in
the data. Otherwise, it will pose a significant threat to statistical validity.

4.5.1 Converting to Logical SLOC

With the SRDR data, the SLOC were counted using different methods
(see the full explanation in Section 3.2.1.2).
• Total Count: a line in a file, e.g. carriage returns including blanks and
comment lines
 Data Assessment 39

• Non-Commented Source Statements (NCSS) Count: a line in a file that

is not a blank or comment line
• Logical Count: as defined in Chapter 3.

For analysis, the definition of a source line of code needs to be as consistent

as possible to eliminate noise in the data. A logical source line of code has
been selected as the baseline SLOC definition in Table 3.2.
If a source line of code count was defined as either Total or NCSS , these
counts were converted to a Logical SLOC count. The conversion factors are
derived from counting public domain software applications and additional
contributions from USC-CSSE Affiliates using the Unified Code Count tool
[41].
The statistics from the counting experiment produced conversion factors
for converting code counted using NCSS rules and Total rules are show below.
By inspecting Table 3.2, it can be observed that Logical counting rules count
source code statements that effect software application execution. The NCSS
and Total count rules show that increasing more statements are counted, hence
the factors below are less than one because NCSS and Total counts are reduced
to an equivalent Logical count.

Logical SLOC count = 0.66 · N CSS count (4.1)

Logical SLOC count = 0.34 · T otal count (4.2)

Most of the SRDR SLOC count data was produced using Logical counting
rules. However, the Data Dictionary sometimes revealed that counts labeled
as Logical were in fact NCSS or Total counts. It is very worthwhile to verify
the count type by consulting the Data Dictionary.

4.5.2 Standardizing Effort

Software CERs have a breadth and a depth. The breadth corresponds to
the number of lifecycle activities covered and the depth is the type of labor
counted in or across each activity. A CERs breadth and depth are defined by
the data used to derive them. It is important to use data that is homogeneous
in the number of activities covered and the labor categories included.
Normalizing effort consists of selecting records for analysis that cover the
same activities and labor categories. This is made challenging because some
SRDR records contain development data where effort for an activity was ex-
pended and recorded in another record. The requirements activity is an ex-
ample. The first build of a software CSCI may have all of the requirements
activity for that build and subsequent builds, i.e., requirements were analyzed
for all builds in the first build. The remedy is to amortize the hours for the
requirements activity across all builds using software size as a proportioning
factor.
40 Software Cost Estimation Metrics Manual for Defense Systems

The activity data in the SRDR follows the ISO 12207 [25] processes for
software development. These are listed below. The ones covered by SRDR
data are in bold. This is the breadth of the CERs reported in this manual.
• System requirements analysis
• System architectural design
• Software requirements analysis
• Software architectural design
• Software detailed design
• Software coding and testing
• Software integration
• Software qualification testing
• System integration
• System qualification testing
• Software installation
• Software acceptance support
The SRDR effort hours also cover a number of labor categories. These
categories are discussed in the companion Data Dictionary. The different labor
categories in the SRDR data are shown in Table 4.2. Not all of the records
had all of the categories. However, the Software Engineering and Assessment
categories were reported for in each record.
When using a CER or any cost model, it is important to know the breadth
and depth of activities covered by the estimate.

4.6 Convert Raw SLOC into Equivalent SLOC

Equivalent size is a method used to make new and adapted code equivalent
so they can be rolled up into an aggregate size estimate (discussed in Section
3.2.2.2). Equivalent Source Lines of Code (ESLOC) is a synthetic measure
and is not a actual measure of software size per [7]:
• It is not a measure of software effort
• It is not a measure of delivered software capability
• It is a quantity derived from software component sizes and reuse factors
that helps estimate effort
• Once a product or increment is developed, its ESLOC loses its identity
– Its size expands into full SLOC
– Can apply reuse factors to this to determine an ESLOC quantity
for the next increment (but this would also have no relation to the
product’s size)
For new, modified, reused and auto-generated code sized, ESLOC is de-
rived by the formula below:
 Data Assessment 41

TABLE 4.2: SRDR Activities

Category SRDR Labor Categories

Management Engineering Management
Business Management

Software Engineering Software Requirements Analysis

Architecture and Detailed Design
Coding and Unit Testing
Test and Integration

Assessment Qualification Testing

Development Test Evaluation Support

Support Software Configuration Management

Software Quality Assurance
Configuration Audit
Development Environment Support
Tools Support
Documentation
Data Preparation
Process Management
Metrics
Training
IT Support / Data Center

ESLOC = N ew SLOC + (AAFM · M odif ied SLOC) + (AAFR ·

Reused SLOC) + (AAFAG · AutoGenerated SLOC)

Where AAFi = (0.4 · DM ) + (0.3 · CM ) + (0.3 · IM ).

The data in an SRDR does not require the submission of values for DM,
CM and IM. Sometimes the values were provided in the Data Dictionary.
Sometimes the data submitter was contacted for the values. However, there
were data remaining with no DM, CM or IM values. Guidelines for filling-in
missing data were derived from the SRDR data that was available.
As shown in the equation above, there are four code types: New, Modified,
Reused, and Auto-Generated. The DM, CM and IM factors are not required
for each type.
• New code does not require any adaption factors. Nothing has been mod-
ified.
• Modified code requires all three factors for DM, CM and IM, represent-
ing modifications to the code’s design, code and integration testing.
42 Software Cost Estimation Metrics Manual for Defense Systems

• Reused code does not require the DM or CM adaption factors but is

reasonable to assume that it will require testing captured with the IM
factor. If Reused code does require modification, then it becomes Mod-
ified code and the adaptation factors for Modified code apply.
• Auto-Generated code does not require the DM or CM adaption factors.
However, it does require testing, IM. If Auto-Generated code does re-
quire modification, then it becomes Modified code and the adaptation
factors for Modified code apply.

Table 4.3 shows median values for DM, CM and IM derived from the subset
of SRDR data that had values for each factor. The median values were used
because the data was not normally distributed. The table shows the code type,
the number of data points, the range of DM, CM and IM values, the median
(Mdn.) value, and the resulting AAF value.

TABLE 4.3: Proxy DM, CM, and IM Values

Code DM CM IM
Type # Range Mdn. Range Mdn. Range Mdn. AAF
New N/A N/A N/A 1.0
Modified 101 0-100% 31% 1-100% 44% 3-100% 72% 0.47
Reused 145 0% 0% 0-100% 10% 0.03
Auto-Gen 6 0% 0% 0-100% 50% 0.15

The median values in Table 4.3 should be used with caution. For instance,
the IM factor for the amount of modification to existing test and integration
procedures may be 100% or more if the software application has high reli-
ability or security requirements or the integration is complex. For modified
software, the DM and CM values for the amount of design and code mod-
ified may be higher than the median values due to additional information
assurance requirements, time and storage constraints, and product complex-
ity. Conversely the reused and auto-generated IM value may be below median
if the software has already been certified in an earlier release or if the software
is a prototype.

4.7 Future Work and Limitations

The Data Quality Rating Scale in Table 4.1 presents values meant to reflect
the quality of the data for CER analysis. Full and complete data is assigned the
highest score, one (1), and missing data the lowest score, zero (0). Assuming
these two values establish an acceptable scale, the intermediate values are
 Data Assessment 43

subjectively assigned. Subjective assignment means the intermediate value

and its distance from other intermediate values cannot be verified. The Quality
Level formula also assumes that each quality category has an equal weight to
the overall rating. This is an assumption and may not be valid either. This
section should be used with caution.
A better approach to describing the quality of data is would be to discuss
the lessons learned in reviewing SRDR data grouped into categories such as
size, effort, and schedule. It is also not necessary to assign numerical values to
the quality of data. Something as simple as using colors (red, yellow, green)
would be sufficient. The next edition of this manual will replace the Data
Quality Rating Scale with the lessons learned approach.
The guidelines for converting raw SLOC counts for the different code types
into equivalent SLOC have gone through many revisions. Providing guidelines
is difficult when there is no insight into how much existing software was mod-
ified, how well the software is structured for modification, and how much
experience existed in understanding how the software implemented it func-
tionality.

• Auto-generated code does require some amount of effort to specify before

generation. Yet, %DM is set to zero.
• A guideline for the Deleted code type is not provided. The SRDR is now
requesting deleted code counts. This will provide future data that may
reveal its contribution to equivalent SLOC.

As more SRDR data is collected these guidelines will expand and improve.
44 Software Cost Estimation Metrics Manual for Defense Systems
Chapter 5
Cost Estimating Relationships

5.1 Overview
This chapter discusses an approach to building Cost Estimating Relation-
ships (CERs) and Benchmarks. Figure 5.1 shows the process starting with
SRDR data and other supporting historical data. Defining, collecting, eval-
uating and normalizing historical data were discussed in previous chapters.
Data segmentation will be used to partition the data into groups with common
project characteristics. Each segment will have its own CER and benchmark.
.

DACIMS Historical Project Data

SRDRs

.
Data Segmentation

Estimating
Relationships Benchmarks

FIGURE 5.1: Building CERs and Benchmarks

Each segment is examined for the presence of a relationship between a

single predictor and single response variable. The statistical method for Or-
dinary Least Squares (OLS) regression for one predictor and one response
variable is used to quantify the relationship. This approach simplifies CER
formulation by eliminating possible effects of multicollinearity found in multi-
variable models. In the case of a cost estimating relationship, software size is

45
46 Software Cost Estimation Metrics Manual for Defense Systems

the predictor and effort is the response. Where relationships exist, CERs and
benchmarks are created.

5.2 Data Segmentation

Factors such as application complexity; impact of loss due to reliability;
autonomous modes of operation; constraints on timing, storage, and power;
security requirements; and complex interfaces influence the cost and time to
develop applications. Parametric cost models have a number of adjustable
parameters that account for these factors.
This chapter describes developing simple CERs based on grouping soft-
ware applications with similar characteristics instead of creating CERs with
many parameters. This makes for practical estimation and enables better pre-
cision within application types vs. a calibration to the global dataset of diverse
applications. The groupings have evolved over time starting with a set of 29
software application groups presented at our research workshops. A survey
was conducted across best practices of other software cost model vendors in
an effort to reduce the number of groups. The resulting set of 17 groups pre-
sented in this chapter have also been submitted to the DoD SRDR Working
Group sponsored by DCARC. They are currently being adopted as Applica-
tion Domains for use in filling out the SRDR data collection form.
Two data classification groups are used to develop CERs and Benchmarks.
The Application Type groups are based on similar functional characteris-
tics, while the Operating Environment groups distinguish the environments
in which the applications operate. Both Operating Environment (OpEnv) and
Application Type (AppType) are considered in the CERs and benchmarks.

5.2.1 Operating Environments (OpEnv)

Operating Environments have similar systems, similar products, similar
operational characteristics, and similar requirements. These aspects include:
• High-speed vehicle versus stationary
• Battery operated versus ground power
• Unrecoverable platform versus recoverable, accessible platform
• Limited, non-upgradeable computing processor capacity versus racks of
processors
• Fixed internal and external memory capacity versus expandable capacity
The Operating Environments for Ground, Maritime, Aerial, Space and
Ordnance decompose into 11 groups when fixed versus mobile and manned
versus unmanned are considered. These high-level environments described in
Table 5.1 are useful when there is a lack of data at the lower levels.
 Cost Estimating Relationships 47

TABLE 5.1: Operating Environments Definition

Operating Environment (OpEnv) Examples

Ground Site (GS) Fixed (GSF) Command Post, Ground Operations
Center, Ground Terminal, Test
Faculties

Mobile (GSM) Intelligence gathering stations

mounted on vehicles, Mobile missile
launcher

Ground Vehicle (GV) Manned Tanks, Howitzers, Personnel carrier

(GVM)

Unmanned Robotic vehicles

(GVU)

Maritime Vessel (MV) Manned Aircraft carriers, destroyers, supply

(MVM) ships, submarines

Unmanned Mine hunting systems, Towed sonar

(MVU) array

Aerial Vehicle (AV) Manned Fixed-wing aircraft, Helicopters

(AVM)

Unmanned Remotely piloted air vehicles

(AVU)

Space Vehicle (SV) Manned Passenger vehicle, Cargo vehicle,

(SVM) Space station

Unmanned Orbiting satellites (weather,

(SVU) communications), Exploratory space
vehicles

Ordnance Vehicle Unmanned Air-to-air missiles, Air-to-ground

(OV) (OVU) missiles, Smart bombs, Strategic
missiles
48 Software Cost Estimation Metrics Manual for Defense Systems

5.2.2 Application Types (AppType)

Application types are groups of software applications with similar functions
and attributes. They are characterized by the following aspects:
• Required software reliability
• Database size, if there is a large data processing and storage component
to the software application
• Product complexity
• Integration complexity
• Real-time operating requirements
• Platform volatility or target system volatility
• Special display requirements
• Development re-hosting
• Quality assurance requirements
• Security requirements
• Assurance requirements
• Required testing level

The Application Types are described in Table 5.2.

TABLE 5.2: Application Types Definitions

Application Type
(AppType) Description
Microcode and Microcode and Firmware is software stored on
Firmware (M&F) target hardware devices that do not have hard
disks and use programmable logic devices. It
is a combination of persistent memory and the
program code and data stored in it.

Examples: Field Programmable Gate Ar-

rays (FPGAs); Microwave controllers;
Field Programmable Logic (FPL); Elec-
tronic Programmable Logic Device (EPLD);
Application Specific Integrated Circuit
(ASIC); Programmable Read-Only Memory
(PROM); Erasable Programmable Read-Only
Memory (EPROM); Electrically Erasable
Programmable Read-Only Memory (EEP-
ROM); Complex Programmable Logic Device
(CPLD); Programmable Array Logic (PAL).

(continued )
 Cost Estimating Relationships 49

TABLE 5.2: Application Types Definition (continued )

Application Type
(AppType) Description
Sensor Control and Software that requires timing-dependent
Signal Processing device coding to enhance, transform, filter,
(SCP) convert, or compress data signals.

Examples: Lasers; Sonar; Acoustic; Electro-

magnetic; Signal Processor; Radar Altimeter;
Photographic Sensors; Motion Sensors; In-
frared Sensors; Sensor Assembly; Electronic
Sensors; Seeker Assembly; Signal Electronics;
Optical Assembly; Tracking Sensors; Antenna
Assembly.

Vehicle Control (VC) Software necessary for the control of vehicle

primary and secondary mechanical devices
and surfaces.

Examples: Flight Control; Electrical Power;

Hydraulic; Fuel Subsystem; Propulsion;
Attitude Control System; Structures &
Mechanisms; Bus Flight Software; Thermal
Control; Landing Gear; Controls software;
Thrust Vector Actuation; Executive.

Vehicle Payload (VP) Software which controls and monitors vehicle

payloads and provides communications to
other vehicle subsystems and payloads.

Examples: Fire Control; Mine Warfare;

Electronic Attack subsystem controller;
Weapons Delivery and Control; Gun fire
control system; Missile fire control systems;
Antisubmarine warfare fire control and tor-
pedo fire control systems; Pointing; Command
& Control Interface; Payload Flight Software;
Armament; Survivability Payload; Reconnais-
sance Payload; Electronic Warfare Payload;
Armament/Weapons Delivery; Intelligence;
Surveillance Reconnaissance Payload; Mission
Payload.

(continued )
50 Software Cost Estimation Metrics Manual for Defense Systems

TABLE 5.2: Application Types Definition (continued )

Application Type
(AppType) Description

Real Time Embedded Interrupt-driven, embedded software in mil-

(RTE) itary and consumer appliances, devices, and
products, possibly directing and processing
sensor inputs/outputs, generally with a very
small executive for an operating system
interface to basic processor(s).

Examples: Embedded Electronics/ Appli-

ance; Robotics; PDAs; Telemetry; Tracking
& Command (TT&C); Guidance; Navigation
and Control; Controls and Displays; Data
Links; Radios (device); Remote Control;
Receiver; Transmitter; Exciter; Bombing
Computer; Video and recorders; Telephones
(device); Built-in-Test.

Command and Control Software that allows humans to manage a

(C&C) dynamic situation and respond in real time.

Examples: Mission Management; Mission

Computer Processing; Mission Control; Com-
mand processing; Air traffic control; Data
reduction/ analysis; Telemetry Processing;
Battlefield command; Battle management.

Communications The transmission of information, e.g. voice,

(COM) data, commands, images, and video across
different mediums. Primarily software systems
that control or manage transmitters, receivers
and communications channels.

Examples: Switches; Routers; Integrated

circuits; Multiplexing; Encryption; Broad-
casting; Transfer modes; Radios (networks);
Network management; Network Operations;
Satellite communications; Telecommunica-
tions; Networks (WAN/LAN); Protocols
(VOIP, TCP/IP, PKI, etc.).

(continued )
 Cost Estimating Relationships 51

TABLE 5.2: Application Types Definition (continued )

Application Type
(AppType) Description

System Software Layers of software that sit between the com-

(SYS) puting platform and applications.

Examples: Operating Systems; Infrastruc-

ture; Framework; Middleware; Device Driver;
Display Drivers; File management; Image
Processing; Interface Driver; Utilities.

Process Control (PC) Software that manages the planning, schedul-

ing and execution of a system based on inputs,
generally sensor driven.

Examples: Temperature control; Manufac-

turing process control; Device or instrument
control.

Scientific and Non real time software that involves significant

Simulation (S&S) computations and scientific analysis.

Examples: System Integration Lab (SIL)

Simulation; Simulators; Offline Data Analysis;
Expert Systems; Math & Algorithm Intensive;
Graphics; Statistical Analysis; Artificial Intel-
ligence; Simulation & Modeling; Engineering
& Science; 3D Modeling & Animation; Trainer
Simulations; Computer Aided Design (CAD);
Weather models.

(continued )
52 Software Cost Estimation Metrics Manual for Defense Systems

TABLE 5.2: Application Types Definition (continued )

Application Type
(AppType) Description
Test, Measurement, Software used for testing, measuring, diag-
and Diagnostic nosing, emulating, and evaluating operational
Equipment (TMDE) hardware and software systems

Software necessary to operate and main-

tain systems and subsystems which are not
consumed during the testing phase and are
not allocated to a specific phase of testing.
This does not include built-in-test (BIT).

Examples: Test equipment software; Test

driver; Maintenance and Diagnostic; Fault
Tolerance; Diagnostic; Equipment emulators.

Training (TRN) Software used for educational and training

purposes.

Examples: Computer Based Training (CBT);

Computer Aided Instruction (CAI); Tutorial
Applications; Courseware.

Software Tools Software that is used for analysis, design,

(TOOL) construction, or testing of computer programs.

Examples: Compilers; Linker/loaders; De-

buggers; Editors; Assemblers; Requirements
analysis & design tool aids; Code generators;
Programming aids; Report generators; Code
auditors; Test case data recording; Test case
data reduction/analysis; Test case generation.

(continued )
 Cost Estimating Relationships 53

TABLE 5.2: Application Types Definition (continued )

Application Type
(AppType) Description
Mission Planning Provides the capability to maximize the use
(MP) of the platform. The system supports all the
mission requirements of the platform and
may have the capability to program onboard
platform systems with routing; targeting; and
performance, map, and Intel data.

Examples: Scenario generation; Planning

& Analysis; Target planning; Route planning;
Fuel planning; Cargo load planning.

Custom AIS Software Software needed to build a custom software

(CAS) application to fill a capability gap not cap-
tured by COTS/GOTS software packages.

Examples: Glue code; External system

interfaces; Data transformation; Inter-
COTS/GOTS data exchange; Graphical User
Interface; Internet Server Applet; Website.

Enterprise Service Software needed for developing functionality

Systems (ESS) or a software service that are unassociated,
loosely coupled units of functionality that
have no calls to each other embedded in them.

Examples: Enterprise service management;

Machine-to-machine messaging; Service dis-
covery; People and device discovery; Metadata
discovery; Mediation service; Service security;
Content discovery and delivery; Federated
search; Enterprise catalog service; Data
source integration; Enterprise content delivery
network; Session management; Presence and
awareness; Text collaboration; White boarding
and annotation; Application sharing; Appli-
cation broadcasting; Virtual spaces; Identity
management; Content discovery; Collabora-
tion; User profiling and customization.

(continued )
54 Software Cost Estimation Metrics Manual for Defense Systems

TABLE 5.2: Application Types Definition (continued )

Application Type
(AppType) Description
Enterprise Information Software needed for building an enterprise
Systems (EIS) information system that uses an integrated
database to support typical business pro-
cesses within business/functional areas and
consistent information access across ar-
eas and systems. COTS/GOTS attributed to
a specific software service or bundle of services.

Examples: Enterprise resource planning;

Enterprise data warehouse; General ledger;
Accounts payable; Revenue and accounts
receivable; Funds control and budgetary
accounting; Cost management; Financial
reporting; Real property inventory and man-
agement; Document management; Logistic
or Supply Planning & Control; Transaction
Processing; Management Performance Report-
ing; Office Information System; Reservation
System; Geographic or spatial information
system; Financial Transactions; Database
management; Data Warehousing; Executive
Information System; Internet Server Applet;
Report Generation; Office Automation; Data
Mining.

5.2.3 MIL-STD-881C WBS Mapping to Application Types

It is challenging to determine which application type should be used to es-
timate the cost of an application (that part of the hardware-software complex
which comprise a domain). By using a work breakdown structure (WBS), the
environment and domain are used to determine the application type. Using
the WBS from MIL-STD-881C [39], a mapping is created from environment
to Application Type (AppType):
• Environment
– Subsystem
∗ Sub-subsystem
· Domains → AppType
 Cost Estimating Relationships 55

Starting with the environment, traverse the WBS to the lowest level where
the domain is represented. Each domain is associated with a Application Type
(AppType). In real-world WBSs, the traverse from environment to AppType
will most likely not be the same number of levels. However the 881C WBS
provides the context for selecting the AppType and should be transferable to
other WBSs.
Two examples for finding the application type using the 881C Aerial Ve-
hicle Manned (AVM) and Space Vehicle Unmanned (SVU) WBS elements are
provided below. The highest level WBS element represents the environment.
In the AVM environment there are the Avionics subsystem, Fire-Control sub-
subsystem. The Fire-Control sub-subsystem contains the sensor, navigation,
air data, display, bombing computer and safety domains. Each domain has an
associated application type:
• Air Vehicle Manned (AVM)
– Avionics
∗ Fire Control
· Search, target, tracking sensors → SCP
· Self-contained navigation → RTE
· Self-contained air data systems → RTE
· Displays, scopes, or sights → RTE
· Bombing computer → VP
· Safety devices → RTE
∗ Data Display and Controls
· Multi-function display → RTE
· Control display units → RTE
· Display processors → RTE
· On-board mission planning → MP
For a space system, the highest level 881C WBS element is the Space
Vehicle Unmanned (SVU). The two sub-systems are Bus and Payload. The
domains for Bus address controlling the vehicle. The domains for Payload
address controlling the onboard equipment. Note the system decomposition
has one less level than the decomposition for Air Vehicle Manned. Each sub-
subsystem has an associated application type:
• Space Vehicle Unmanned (SVU)
– Bus
∗ Structures & Mechanisms → VC
∗ Thermal Control → VC
∗ Electrical Power → VC
∗ Attitude Control → VC
∗ Propulsion → VC
∗ Telemetry, Tracking, & Command → RTE
∗ Bus Flight Software → VC
56 Software Cost Estimation Metrics Manual for Defense Systems

– Payload
∗ Thermal Control → RTE
∗ Electrical Power → RTE
∗ Pointing, Command, & Control Interface → VP
∗ Payload Antenna → SCP
∗ Payload Signal Electronics → SCP
∗ Optical Assembly → SCP
∗ Sensor → SCP
∗ Payload Flight Software → VP
The detailed mappings from MIL-STD-881C to Application Types are in
Appendix B. All of the WBS elements from MIL-STD-881C were mapped to
the appropriate application type. MIL-STD-881C covers the following oper-
ating environments:
• Aerial Vehicle Manned (AVM)
• Aerial Vehicle Unmanned(AVU) & Ground Site Fixed (GSF)
• Ordnance Vehicle Unmanned (OVU)-Ordnance
• Ordnance Vehicle Unmanned (OVU)-Missile
• Ordnance Vehicle Unmanned (OVU)-Launch Vehicles
• Maritime Vessel Manned (MVM)
• Maritime Vessel Unmanned (MVU) and Maritime Vessel Manned
(MVM)
• Space Vehicle Manned / Unmanned (SVM/U) & Ground Site Fixed
(GSF)
• Ground Site Fixed (GSF)
• Ground Vehicle Manned & Unmanned (GVM/U)
• Applies to All Environments: Common Elements
After segmenting the data into application types and operating environ-
ments, each group is examined for an estimating relationship between size and
effort. This is discussed next.

5.3 Estimating Relationships

There are many statistical techniques available to explore the relationship
between X and Y. The exploratory technique described here uses the scatter
plot . A scatter plot shows the relationship, if any, between X and Y. Without
doing any number crunching, scatter plots immediately reveal X and Y’s re-
lationship. By convention the predictor, X, is presented on the horizontal axis
and the response, Y, on the vertical axis of the scatter plot. Examples of dif-
ferent relationships are shown next with example (non-real) data. Additional
exploratory data analysis techniques can be found at [42].
 Cost Estimating Relationships 57

5.3.1 Scatter Plots

The plot in Figure 5.2 shows the predictor factor X plotted against the
response factor Y. For the X values there are multiple corresponding values
for Y with a wide variation across the range. This lack of predictability in
determining Y from a given value of X, and the associated amorphous, non-
structured appearance of the scatter plot leads to the conclusion that there is
no relationship.

100

80

60
Y

40

20
.
0.
0 20 40 60 80 100
X

FIGURE 5.2: No Relationship

Scatter plots are extremely insightful for assessing relationships visually.

It should be noted that a Y vs. X relationship could be created from the
data in Figure 5.2 using OLS regression. However, a quick visual check of the
scatter plot between predictor and response shows there is an extremely weak
or no relationship. Therefore the relationship should be discarded and not
considered further.
Note in Figure 5.3 how a straight line comfortably fits through the data.
This indicates a linear relationship between X and Y. The data scatter about
the line is quite small indicating a strong relationship. The slope of the line
shows a positive relationship between X and Y, i.e., as X increases, Y also
increases. Visual inspection of the scatter data shows that a fitted model is
appropriate and further statistical testing is warranted.
Figure 5.4 also shows a straight line comfortably fitting through the scat-
ter data indicating a strong relationship. The negative slope of the line means
there is a negative relationship between X and Y, i.e., as X increases, Y de-
creases. The data scatter about the line is small indicating a strong relation-
ship for prediction.
58 Software Cost Estimation Metrics Manual for Defense Systems

100

80

Y 60

40

20

0 .
.
0 20 40 60 80 100
X

FIGURE 5.3: Strong Positive Relationship

100

80

60
Y

40

20

0 .
.
0 20 40 60 80 100
X

FIGURE 5.4: Strong Negative Relationship

In Figure 5.5, the scatter plot reveals a positive, homoscedastic (homo-sce-

das-tic) linear relationship between X and Y. For a value of X, the value of Y
varies consistently about the predicted value of Y. This is a condition called
homoscedasticity, which is very important as it is an underlying assumption
for the regression analysis technique used to create CERs. Its violation leads
 Cost Estimating Relationships 59

to parameter estimates with inflated variances and CER statistics indicating

poor accuracy. Thus a homogeneity of variance shown in a homoscedastic
linear relationship is assumed for valid CERs.

100

80

60
Y

40

20

0 .
.
0 20 40 60 80 100
X

FIGURE 5.5: Homoscedastic Relationship

The scatter plot in Figure 5.6 shows an approximately positive, het-

eroscedastic (hetero-sce-das-tic) linear relationship between X and Y. How-
ever, for any given value of X, the Y variation range about the predicted
value of Y depends on the value of X. This condition is called heteroscedas-
ticity meaning the non-constant variation in Y over the values of X. In Figure
5.6, small values of X show a small data scatter in Y while large values of X re-
sult in large data scatter in Y. OLS regression would produce larger prediction
errors as X increases.
Figure 5.6 also shows another condition, the non-normal distribution of
data. Normally distributed data is an underlying assumption for producing
valid CERs. In the figure, the non-normal data shows up as clumping of the
data scatter for a range of X values. Often with software size and effort data,
the clumping occurs for lower values of size. Unless this condition is alleviated
CERs should not be created from the data.
One method to overcome the effects of heteroscedasticity is by weighting
of the data with noisier data being weighted less. The weight would depend
on the value of X where heteroscedasticity is observed. Another approach is
to perform a Y variable transformation to achieve homoscedasticity using a
technique such as a Box-Cox transformation. This procedure will transform
data into a normal distribution shape.
60 Software Cost Estimation Metrics Manual for Defense Systems

100

80

Y 60

40

20

0 .
.
0 20 40 60 80 100
X

FIGURE 5.6: Heteroscedastic Relationship

When creating CERs, the two factors of interest in SRDR data are software
size and effort. Size is treated as the predictor variable (X in the discussion
above) and effort is treated as the response variable (Y). Software size in SRDR
data is expressed as ESLOC defined in Section 3.2.2. Effort is expressed as
person hours which is then transformed into person months as described in
Section 3.4.3.
SRDR size and effort data usually have a positive relationship, are non-
normally distributed, and exhibit heteroscedasticity. Both conditions of non-
normality and heteroscedasticity can be reduced in SRDR data by transform-
ing the size and effort data using logarithms. This permits the use of OLS to
derive CER models.

5.3.2 Data Transformation

The technique for creating CERs (discussed next) assumes the data is
normally distributed. In a normal distribution the data is distributed equally
above and below the mean data value. Unfortunately, software engineering cost
data is rarely normally distributed. A contributing factor to this phenomena
is that cost data is bounded on the lower end by zero. Another factor is there
are typically more smaller size CSCIs that larger ones.
To address this issue the data should be transformed to a normal distri-
bution before being analyzed. The Anderson-Darling statistical test can help
determine if data has a normal distribution. If the test indicates non-normality,
the data is transformed using logarithms. Although other transformation can
be used, such as square root or squares, a logarithmic transformation has been
 Cost Estimating Relationships 61

found to be the most effective at transforming non-normal cost data into a

normally distributed data set. Analysis is carried out in log-space and the
results transformed back into unit-space.
Using SRDR data as an example, Figure 5.7 shows non-normally dis-
tributed size data for the RTE AppType using a histogram on the left. When
the data is transformed using logarithms, the distribution is much closer to a
normal distribution per the histogram on the right.

20 10

Frequency
Frequency

10 5

.
0. . 0.
0 50 100 150 0.5 1 1.5 2
KESLOC Log KESLOC

FIGURE 5.7: RTE Size Data Distributions

As discussed next, a logarithmic transformation is also used to convert a

non-linear CER model form into a linear model form that can be analyzed
using advanced statistical techniques.

5.3.3 CER Models

The cost estimating relationship between size and effort can take several
forms. The following two models could be used to produce CERs:

Ef f ort(P M ) = A · (KESLOC)B (5.1)

Ef f ort(P M ) = C + A · (KESLOC)B (5.2)

Each equation has a primary input parameter, KESLOC, with additional
factors. The multiplicative factor, A, roughly represents a production cost in
Person-Months per thousands of lines of code. The C factor represents the
fixed start-up and overhead activity costs expressed in Person-Months.
Both models have an exponent B on the size parameter, which is often
referred to as a scaling factor. The impact of B is greater when size is larger and
reduced when size is smaller. A common issue in modeling software engineering
cost data is whether B is greater than or less then 1.0. If B is greater than one,
larger size systems require more effort per line of code (diseconomies of scale).
If B is less than one, larger sizes require less effort per line of code developed
(economies of scale).
62 Software Cost Estimation Metrics Manual for Defense Systems

Banker and Kemerer [4] provide a survey of reasons for economies and
diseconomies of scale. Their research substantiates both economies and disec-
onomies of scale. The economies of scale were observed on small projects and
diseconomies of scale were observed on large projects. Economies of scale are
attributed to:
• Software development tools that increase productivity
• Specialized personnel that are highly productive
• Fixed overhead that does not increase directly with project size thereby
producing economies of scale in larger projects.
Diseconomies of scale are due to:
• Increased coordination effort required
• Larger systems having more complex interface problems
• Increased effort to obtain agreement or consensus
• Increase in overhead activities at a faster than linear rate as product
size increases.
Equation 5.2 has the advantage of removing fixed start-up costs from the
prediction portion of the equation. During SRDR analysis it was observed
that fixed costs can mask the diseconomy of scale relationship between size
and effort, especially with smaller size projects (less than 50 KESLOC). When
Equation 5.2 was used as the CER model form, the diseconomy of scale was
more often observed.
Equation 5.2 also has the advantage of a user being able to assess the
reasonableness of the C factor with respect to known fixed costs in their
organization and using the other part of the equation for the variable costs.
Alternatively, a user could use the variable cost part and add it to their own
estimate of project fixed start up costs. Both parts of the equation have a
distinct interpretation and can be assessed for their reasonableness. However,
there are difficulties using the model form in Equation 5.2.
Both Equations 5.1 and 5.2 are non-linear due to the scaling factor. If
the equations were linear (e.g., P M = A + B · KESLOC), OLS can be used
to find the values for the model factors. This technique finds the values for
the factors by minimizing the square of the errors between the actual and
predicted effort, PM.
The multiplicative model in Equation 5.1 can be transformed from its non-
linear form into a linear form using logarithms shown in Equation 5.3. The
CERs presented in this chapter use this log transformed linear model.

log(P M ) = log(A) + B · log(KESLOC). (5.3)

OLS is then used to find the estimated values for the A and B factors
using the SRDR data. The resulting value for the A factor is actually Log(A).
It is converted back to unit-number space by raising to the log base (10 or e
if using natural logarithms) to the log value A. B is used as the scale factor
without conversion.
 Cost Estimating Relationships 63

Despite all of its advantages, 5.2 cannot be transformed into a linear model
due to the position of the C factor in the model form. To solve for the three
factors in 5.2, a non-regression iterative search technique must be used. De-
pending on the initial starting conditions, different values can result from
the search. It is recommended to use multiple starting conditions to perform
multiple searches and to select the factor values that result most often. This
requires a specialized software analysis tool to conduct such a search.
Because of the ease of finding the factor values in Equation 5.1 and the
difficulty of finding the factor values in 5.2, 5.1 is chosen as the CER model
form presented in this chapter.

5.3.4 CER Calibration

The A and B factors in the CER model shown in Equation 5.3 are derived
from SRDR data in each of the application types and operating environments.
A and B are observable in scatter plots like those shown in Figures 5.3 and
5.4. The A is the intercept of the trend line through the data if it were ex-
tended until it crossed the vertical axis. The B is the slope of the trend line
representing the average change in the response variable (PM in this analysis)
for one unit of change in the predictor variable (KESLOC), i.e., the average
change in effort for one unit of change in size.
OLS calculates the CER factors (A and B) that minimizes the distance
between the fitted line and all of the data points, i.e. minimizes the error.
It does this by mathematically minimizing the sum of the squared residual
errors. In general, a model fits the data well if the differences between the
observed values and the model’s predicted values (the residual errors) are
small and unbiased. Large outlier data perturbs model results by producing
larger differences.
Statistical software packages have linear regression tools that will analyze
data and produce estimates for the A and B factors. Microsoft Excel also has
a regression tool in the Data Analysis Tool Pack that will also produce these
estimates.
In addition to estimates of the A and B factors, regression tools also pro-
duce statistics on the significance and variability of the estimated factors.
Some of these important statistics are presented next with criteria used to
evaluate the results for each CER model.

5.3.5 CER Evaluation

OLS regression with one predictor and one response variable provides es-
timates of factor values based on minimizing error. Therefore every CER has
some amount of error. CERs are evaluated with a number of different statistics
described in Table 5.3.
The calibration datasets are described by N, Min and Max KESLOC.
Groups of data had to have 10 or more data points to be included in the
64 Software Cost Estimation Metrics Manual for Defense Systems

analysis presented here. The min and max size also set the boundaries for
where the CER can safely be used.
The multiplicative factors A and B are the coefficients derived from the
OLS regression. They are evaluated with the T-Statistic (T-Stat) and P-value
measures for hypothesis testing. Generally a T-stat greater than 2.0 indicates
a significant value and one that can be used in the CER model. A predictor
that has a low P-value is likely to be a meaningful addition to the CER
because changes in the predictor’s value are related to changes in the response
variable. Conversely, a larger (insignificant) P-value suggests that changes in
the predictor are not associated with changes in the response.
The Lower Confidence Interval (LCI) and Upper Confidence Interval (UCI)
specify ranges for the true values of A and B. This is because OLS regression
provides estimates of factor values based on minimizing error. In practice the
confidence interval is set to a desired level of confidence. For instance, we can
be 95% confident that the range of the 95% confidence interval includes the
real population mean of the effort to develop software of a given size. As the
sample size increases, the sampling error decreases and the intervals become
narrower. If one could increase the sample size to equal the population, there
would be no sampling error. The confidence interval would have a width of
zero and be equal to the true population parameter.
The Coefficient of Determination (R2 ) is a primary measure that describes
how much of the variation is explained by the CER regression model. The more
variance that is accounted for by the regression model the closer the data
points will fall to the fitted regression line. However, R2 cannot determine
whether the factor estimates and predictions are biased. Bias is detected in a
scatter plot (called residual plots) of estimated versus actual response values.
Looking at the scatter plot of residuals is necessary to evaluate the CER model
factors.
The Standard Error (SE) is the standard deviation around the regression
line. It is used to assess the precision of the predictions. R2 is relevant mainly
for assessing the fit of the trendline to the data. However, R2 cannot be used
to assess precision which SE is used for.

TABLE 5.3: CER Evaluation Statistics

Statistic Description
N Number of data points used to create the CER.

Min and Max Minimum and maximum ESLOC size used to

KESLOC create the CER.

A Multiplicative factor derived from OLS regres-

sion.

(continued )
 Cost Estimating Relationships 65

TABLE 5.3: CER Evaluation Statistics (continued )

Statistic Description

B Scaling factor derived from OLS regression.

T-Stat The T-Statistic (T-Stat) indicates the signifi-

cance of the regression derived factors (multi-
plicative and scaling). It tells how many stan-
dard deviations the factors are from zero.

P-Value The P-value for each factor tests the null hy-
pothesis that the factor is equal to zero and
has no effect. A low p-value less than α (typi-
cally < 0.05) indicates that one can reject the
null hypothesis. The level of significance α is
the probability of achieving data at least as
extreme.

LCI and UCI Lower Confidence Interval (LCI) and Upper

Confidence Interval (UCI) specify a range of
values for the CER factors (A and B) that are
likely to contain the true value of the unknown
population parameter (the real A and B) at a
given confidence level.

R2 The Coefficient of Determination (R2 ) rep-

resents the percentage of total variation ex-
plained by the regression model. It provides a
measure of how well actual values of effort are
estimated by the CER model factors A and B.
R2 ranges between 0 and 100% and is the same
in log-space as in unit-space.

SE Standard Error (SE) represents the average

distance that the observed values fall from the
regression line (estimated values). It indicates
how wrong the regression model is on average
using the units of the response variable,
effort. Smaller values indicate that the obser-
vations are closer to the fitted line.

(continued )
66 Software Cost Estimation Metrics Manual for Defense Systems

TABLE 5.3: CER Evaluation Statistics (continued )

Statistic Description
PRED(30) Percent of estimates where the PREDiction
is within 30% of actuals. For parametric cost
models a PRED(30) of 75% is very good.

Scatter plots are also used in CER assessment. Two plots are useful, a plot
of residuals discussed in Section 5.3 and a plot of KESLOC versus PM with
the CER model trend line displayed through the data scatter. Residual plots
show the scatter of errors and should be checked before evaluating statistical
measures for goodness-of-fit. They can reveal unwanted residual patterns that
indicate biased results more effectively than numbers. When the residual plots
show a random distribution of errors, the goodness-of-fit statistics are valid.
Both CER evaluation statistics and evaluation scatter plots are provided
for each SRDR CER presented in this chapter.

5.3.6 CER Example

Creating CERs starts with segmenting the data into similar productivity
groups. Scatter plots are used as a means of visually determining if a rela-
tionship exists between a predictor, X, and response variable, Y. The data
is transformed into log-number space to make is normally distributed and
homoscedastic. Factors for each CER parameter are estimated using simple
linear regression. Finally the regression results are evaluated with statistics
and scatter plots. This section presents an example that will be followed for
each SRDR CER presented later.
The CER for the Real-Time Embedded (RTE) application type is pre-
sented in Equation 5.4. The effort (PM) and size (KESLOC) were both trans-
formed from unit-number space into log-number space by taking natural logs
(log) of all values. A regression tool was given the response values (log PM)
and the corresponding predictor values (log KESLOC) for the RTE data. The
resulting CER to predict effort in Person-Months (PM) for the RTE Applica-
tion Type is:

P M = 13.16 · KESLOC 0.84 (5.4)

Because the RTE data is only a sample of the RTE population of data,
the factors, 13.16 and 0.84, are estimates of the true population parameters
and each has an associated error. Tables 5.4 and 5.5 show the CER evaluation
statistics.
Table 5.4 shows that the number of data points was 57. The minimum size
in the data was 1.8 KESLOC and the maximum was 200.6 KESLOC. The
 Cost Estimating Relationships 67

estimated factors are 13.16 for A and 0.84 for B. Each factors has a very high
T-Stat (good) and an extremely low P-Value (good). The confidence interval
for A is ± 3.9 and for B is ±0.11. The R-squared is 83% which is judged to be
strong. The SE is 211 PM of effort which isn’t so good. PRED(30) is 56.4%
which is isn’t good.

TABLE 5.4: RTE CER Regression Statistics

Statistic Value
N 57
Min KESLOC 1.8
Max KESLOC 200.6
A 13.16
B 0.84
R2 83%
SE (PM) 211
PRED(30) 54%

TABLE 5.5: RTE CER Coefficient Statistics

Coefficient Value T-Stat P-Value LCI UCI
A 13.16 14.7 1.00E-20 9.26 18.69
B 0.84 16.2 1.48E-22 0.73 0.94

Figure 5.8 shows the RTE CER Residuals scatter plot which is checked
for any appearance of patterns. The scatter of errors appears to be randomly
distributed, which indicates a good regression fit.

0.5
Residuals

.
−0.5 .
0.5 1 1.5 2 2.5

FIGURE 5.8: Residuals Scatter Plot

68 Software Cost Estimation Metrics Manual for Defense Systems

Figure 5.9 shows the RTE CER scatter plot with ± 1 Standard Error in log
space, the Upper and Lower Prediction Intervals, and CER error distribution
for a value of 10 KESLOC. The solid trendline for the CER is enclosed by the
Standard Error and 95% prediction intervals representing estimation error.
The ± 1 Standard Error around the line shows the average error around the
mean for the regression, and the CER error distribution is a normal distribu-
tion with a mean of zero across the prediction interval.

. .
P M = 13.20 · KESLOC 0.84
. .
Upper/Lower Prediction Interval
. ± Standard Error .
. .
CER Error Distribution @10 KESLOC

1000
Person-Months

100

10
.
10 100
KESLOC
.
FIGURE 5.9: RTE CER Prediction Interval and Error Distribution

A prediction interval represents the range where the response value is

likely to fall given a single new predictor value. Prediction intervals account
for the variability around the mean response inherent in any prediction. Like
a confidence interval, prediction interval is set by the level of confidence that
a new single observation will predict the correct response.
The prediction intervals contain 95% of the actual result values for the log
of PM (effort). For instance, we can be 95% confident that this range includes
the effort to develop software given a size. Unlike confidence intervals, predic-
tion intervals predict the spread for response values given a predictor
value rather than the mean response value. The prediction interval is always
wider than the corresponding confidence interval of the prediction because
of the added uncertainty involved in predicting a single response versus the
mean response.
Using the CER formula in Equation 5.4 for an estimated 10 KESLOC
CSCI, the estimated effort is 91 Person Months with the normally distributed
 Cost Estimating Relationships 69

CER error spread shown in Figure 5.9. The error range covers approximately
30-250 Person-Months.

5.4 SRDR-Derived CERs

5.4.1 Model Coverage
The coverage of model-based CERs by operating environment and appli-
cation type are in Table 5.6 showing the number of project data points. The
operating environment acronyms in the columns are defined in Table 5.1 The
application types in rows are defined in Table 5.2. Not all application types
and environments were covered due to lack of enough data in each group (at
least 10 data points are required). The populated cells denote a CER and the
number in a cell is the number of data points used to create the CER. The
Total column and row are for all operating environments or application types.

TABLE 5.6: Model Coverage

AppType AV GF GV MV OV Total
C&C 9 16 3 5 33
CAS 1 13 2 16
COM 1 22 2 22 47
MP 20 20
RTE 23 21 3 5 5 57
S&S 6 8 1 1 1 17
SCP 11 14 1 3 6 36
SYS 8 13 3 3 27
TMDE 1 6 4 11
VC 10 14 3 27
VP 10 1 5 18
Total 87 134 26 42 25 306
70 Software Cost Estimation Metrics Manual for Defense Systems

5.4.2 Operating Environment CERs

The CERs presented in this and following sections follow the example in
Section 5.3.6 and show the statistical criteria discussed in Section 5.3.5. Each
CER is presented on its own page. The Operating Environment CERs in this
section are derived from the number of data points shown in the Total row
at the bottom of Table 5.6. The CER results for Operating Environment are
summarized in Table 5.7.

TABLE 5.7: CERs by Operating Environment

Op KESLOC
Env Effort Equation N R2 SE PRED Min Max
AV P M = 9.41 · KESLOC 0.93 87 77% 516 39 0.7 329.9
GS P M = 12.6 · KESLOC 0.79 134 74% 478 56 0.8 842.1
GV P M = 15.1 · KESLOC 0.82 26 74% 178 54 7.2 283.1
MV P M = 5.44 · KESLOC 1.12 42 87% 206 33 0.4 123.8
OV P M = 27.45 · KESLOC 0.71 25 79% 11 48 0.9 221.1
 Cost Estimating Relationships 71

5.4.2.1 Aerial Vehicle (AV)

CC 9
CAS 1
P M = 9.41 · KESLOC 0.93

App Type
COM 1
MP 0
RTE 23
Data Points = 87 SS 6
SCP 11
KESLOC Min = 0.7 SYS 8
KESLOC Max = 329.9 TMDE 1
VC
. 10
VP
. 10
0 5 10 15 20 25

# Data Points

0.5
Residuals

R2 = 77%
SE = 516 PM 0
PRED = 39% −0.5
.
.
0 0.5 1 1.5 2 2.5
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 9.41 12.3 1.54E-20 6.55 13.53
B 0.93 17.0 4.58E-29 0.82 1.03

. .
P M = 9.41 · KESLOC 0.93
. .
± Standard Error
1000
Person-Months

100

10

. .
1 10 100
KESLOC

FIGURE 5.10: AV Dataset and CER Summary

72 Software Cost Estimation Metrics Manual for Defense Systems

5.4.2.2 Ground Site (GS)

CC 16
CAS 13
P M = 12.6 · KESLOC 0.79

App Type
COM 22
MP 20
RTE 21
Data Points = 134 SS 8
SCP 14
KESLOC Min = 0.8 SYS 13
KESLOC Max = 842.1 TMDE 6
VC
VP . 00
.
0 5 10 15 20

# Data Points

0.5
Residuals

R2 = 74%
SE = 478 PM 0
PRED = 56%
−0.5
.
.
0 0.5 1 1.5 2 2.5 3
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 12.6 15.84 2.51E-32 9.18 17.27
B 0.79 19.56 8.05E-41 .71 0.87

. .
P M = 12.6 · KESLOC 0.79
. .
± Standard Error

1000
Person-Months

100

10

.
1 10 100
.
KESLOC

FIGURE 5.11: GS Dataset and CER Summary

 Cost Estimating Relationships 73

5.4.2.3 Ground Vehicle (GV)

CC 0
CAS 2
P M = 15.1 · KESLOC 0.82

App Type
COM 2
MP 0
RTE 3
Data Points = 26 SS 1
SCP 1
KESLOC Min = 7.2 SYS 3
KESLOC Max = 283.1 TMDE 0
VC
.0 14
VP
.
0 2 4 6 8 10 12 14

# Data Points
0.4
Residuals

2
R = 74% 0.2
SE = 178 PM 0
PRED = 54% −0.2
−0.4 .
.
1 1.5 2 2.5
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 15.1 7.67 6.56E-08 7.3 31.5
B 0.82 8.18 2.09E-08 .61 1.02

. .
P M = 15.1 · KESLOC 0.82
. .
± Standard Error

100
Person-Months

10

.
1 10
KESLOC

FIGURE 5.12: GV Dataset and CER Summary

.
74 Software Cost Estimation Metrics Manual for Defense Systems

5.4.2.4 Maritime Vessel (MV)

CC 3
CAS 0
P M = 5.44 · KESLOC 1.12

App Type
COM 22
MP 0
RTE 5
Data Points = 42 SS 1
SCP 3
KESLOC Min = 0.4 SYS 3
KESLOC Max = 123.8 TMDE 4
VC
VP . 01
.
0 5 10 15 20

# Data Points

0.5
Residuals

R2 = 87%
SE = 206 PM
0
PRED = 33%
.
.
−0.5 0 0.5 1 1.5 2
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 5.44 9.73 4.28E-12 3.83 7.74
B 1.12 16.61 1.51E-19 .98 1.26

. .
P M = 5.44 · KESLOC 1.12
. .
± Standard Error
Person-Months

1000

100

10 .
.
1 10 100
KESLOC

FIGURE 5.13: MV Dataset and CER Summary

 Cost Estimating Relationships 75

5.4.2.5 Ordnance Vehicle (OV)

CC 5
CAS 0
P M = 27.45 · KESLOC 0.71

App Type
COM 0
MP 0
RTE 5
Data Points = 25 SS 1
SCP 6
KESLOC Min = 0.9 SYS 0
KESLOC Max = 221.1 TMDE 0
VC
. 3
VP
. 5
0 1 2 3 4 5 6

# Data Points
0.5
Residuals

2
R = 79%
SE = 11 PM 0
PRED = 48%
−0.5 .
.
0 0.5 1 1.5 2 2.5
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 27.45 11.88 2.70E-11 15.42 48.88
B 0.71 9.35 2.67E-09 0.55 0.87

. P M = 27.45 · .KESLOC 0.71

. .
± Standard Error
1000
Person-Months

100

10 .
1 10 100
KESLOC

FIGURE 5.14:
. OV Dataset and CER Summary
76 Software Cost Estimation Metrics Manual for Defense Systems

5.4.3 Application Types CERs

The CERs in this section are derived from the number of data points in
Table 5.6 listed as the AppType Total in the right column. The CER results
across all environments are summarized in Table 5.8.

TABLE 5.8: CERs by AppType Across All Environments

App KESLOC
Type Effort Equation N R2 SE PRED Min Max
CAS P M = 2.64 · KESLOC 1.02 16 97% 135 88 2.4 417.1
COM P M = 7.30 · KESLOC 0.91 47 88% 253 60 0.4 532.4
C&C P M = 6.60 · KESLOC 1.05 33 88% 449 52 0.8 229.0
MP P M = 6.14 · KESLOC 0.86 20 77% 220 55 9.6 570.0
RTE P M = 13.20 · KESLOC 0.84 57 83% 211 54 1.8 200.6
SCP P M = 26.5 · KESLOC 0.87 36 91% 346 50 0.8 192.9
S&S P M = 7.43 · KESLOC 0.91 17 85% 176 53 4.4 225.8
SYS P M = 5.06 · KESLOC 0.98 27 93% 585 48 0.8 842.1
TMDE P M = 7.42 · KESLOC 1.00 11 92% 454 27 0.5 313.0
VC P M = 9.05 · KESLOC 1.02 27 92% 303 48 0.7 329.9
VP P M = 22.27 · KESLOC 0.81 18 89% 111 56 1.1 221.1
 Cost Estimating Relationships 77

5.4.3.1 Command and Control (C&C)

P M = 6.60 · KESLOC 1.05 16

Data Points
15
10 9
Data Points = 33 5
KESLOC Min = 0.8 5 3
. . 0
KESLOC Max = 229.0 0
Aerial Ground Ground Maritime Ordnance
Vehicle Site Vehicle Vessel Vehicle
Operating Environment

R2 = 83% 0.4
Residuals

SE = 211 PM 0.2
PRED = 54%
0

−0.2
.
.
0 0.5 1 1.5 2 2.5
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 6.6 7.15 4.89E-08 3.85 11.3
B 1.05 15.2 6.42E-16 0.91 1.19

. .
P M = 6.60 · KESLOC 1.05
. .
± Standard Error
1000
Person-Months

100

10

.
1. 10 100
KESLOC

FIGURE 5.15: CC Dataset and CER Summary

78 Software Cost Estimation Metrics Manual for Defense Systems

5.4.3.2 Custom AIS Software (CAS)

P M = 2.64 · KESLOC 1.02 13

Data Points
10
Data Points = 16
5 2
KESLOC Min = 2.4 1 0 0
0 . .
KESLOC Max = 417.1
Aerial Ground Ground Maritime Ordnance
Vehicle Site Vehicle Vessel Vehicle
Operating Environment

R2 = 97% 0.1
Residuals

SE = 135 PM
0
PRED = 88%
−0.1

−0.2 .
.
0.5 1 1.5 2 2.5
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 2.64 5.45 8.51E-05 1.80 3.87
B 1.02 23.1 1.47E-12 0.93 1.12

. .
P M = 2.64 · KESLOC 1.02
. .
± Standard Error
1000
Person-Months

100

10

.
10 100
. KESLOC

FIGURE 5.16: CAS Dataset and CER Summary

 Cost Estimating Relationships 79

5.4.3.3 Communication (COM)

P M = 7.30 · KESLOC 0.91 22 22

Data Points
20
Data Points = 47 10
KESLOC Min = 0.4 1 2 0
0 . .
KESLOC Max = 532.4
Aerial Ground Ground Maritime Ordnance
Vehicle Site Vehicle Vessel Vehicle
Operating Environment

1
R2 = 88%
Residuals

SE = 253 PM 0.5
PRED = 60%
0

.
−0.5 .
0 1 2 3
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 7.3 5.45 2.37E-17 1.80 3.87
B .91 23.1 3.15E-22 0.93 1.12

. .
P M = 7.30 · KESLOC 0.91
. .
± Standard Error
1000
Person-Months

100

10

1. .
1 10 100 1000
KESLOC

FIGURE 5.17: COM Dataset and CER Summary

80 Software Cost Estimation Metrics Manual for Defense Systems

5.4.3.4 Mission Planning (MP)

P M = 6.14 · KESLOC 0.86 20

Data Points
20
15
Data Points = 20 10
KESLOC Min = 9.6 5
0 . 0. 0 0 0
KESLOC Max = 570.0
Aerial Ground Ground Maritime Ordnance
Vehicle Site Vehicle Vessel Vehicle
Operating Environment

0.6
R2 = 77% 0.4
Residuals

SE = 220 PM
0.2
PRED = 55%
0
−0.2
.
−0.4 .
1 1.5 2 2.5
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 6.14 3.68 0.0017 2.18 17.31
B 0.86 7.74 3.93E-07 0.63 1.09

. .
P M = 6.14 · KESLOC 0.86
. .
± Standard Error
1000
Person-Months

100

.
10 100
KESLOC

FIGURE 5.18: MP Dataset and CER Summary

.
 Cost Estimating Relationships 81

5.4.3.5 Real Time Embedded (RTE)

P M = 13.20 · KESLOC 0.84 23 21

Data Points
20
Data Points = 57
10 5 5
KESLOC Min = 2 3
0 . .
KESLOC Max = 201
Aerial Ground Ground Maritime Ordnance
Vehicle Site Vehicle Vessel Vehicle
Operating Environment

0.5
R2 = 83%
Residuals

SE = 211 PM
PRED = 54% 0

.
−0.5 .
0.5 1 1.5 2 2.5
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 13.16 14.7 1.00E-20 9.26 18.69
B 0.84 16.2 1.48E-22 0.73 0.94

. P M = 13.20 · .KESLOC 0.84

. .
± Standard Error
1000
Person-Months

100

10
.
10 100
KESLOC

. 5.19: RTE Dataset and CER Summary

FIGURE
82 Software Cost Estimation Metrics Manual for Defense Systems

5.4.3.6 Scientific and Simulation (S&S)

P M = 7.43 · KESLOC 0.91 8

Data Points
8 6
6
Data Points = 17 4 3 3
KESLOC Min = 4.4 2 0
0 . .
KESLOC Max = 225.8
Aerial Ground Ground Maritime Ordnance
Vehicle Site Vehicle Vessel Vehicle
Operating Environment

0.4
R2 = 85%
Residuals

SE = 176 PM 0.2
PRED = 53% 0

−0.2

−0.4 .
.
0.5 1 1.5 2 2.5
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 7.4 5.2 0.00011 3.26 16.9
B 0.91 9.1 1.66E-07 0.69 1.17

. .
P M = 7.43 · KESLOC 0.91
. .
± Standard Error
100
Person-Months

10

.
10 100
KESLOC

FIGURE 5.20: S&S Dataset and CER Summary

.
 Cost Estimating Relationships 83

5.4.3.7 Sensor Control and Signal Processing (SCP)

P M = 26.5 · KESLOC 0.87 14

15

Data Points
11
10
Data Points = 36 6
KESLOC Min = 0.8 5 3
1
0 . .
KESLOC Max = 192.9
Aerial Ground Ground Maritime Ordnance
Vehicle Site Vehicle Vessel Vehicle
Operating Environment

R2 = 91% 0.2
Residuals

SE = 346 PM
PRED = 50% 0

−0.2
.
.
0 0.5 1 1.5 2 2.5
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 26.5 21.7 1.72E-21 19.5 36.0
B 0.87 18.5 2.43E-19 0.77 0.96

. .
P M = 26.5 · KESLOC 0.87
. .
± Standard Error
Person-Months

1000

100 .
1 10 100
KESLOC

FIGURE 5.21:
. SCP Dataset and CER Summary
84 Software Cost Estimation Metrics Manual for Defense Systems

5.4.3.8 System Software(SYS)

P M = 5.06 · KESLOC 0.98 13

Data Points
10 8
Data Points = 27
5 3 3
KESLOC Min = 0.8 0
0 . .
KESLOC Max = 842.1
Aerial Ground Ground Maritime Ordnance
Vehicle Site Vehicle Vessel Vehicle
Operating Environment

0.6
R2 = 93%
0.4
Residuals

SE = 585 PM
PRED = 48% 0.2

−0.2 .
.
0 0.5 1 1.5 2 2.5 3
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 5.1 7.1 1.55E-07 3.18 8.05
B 0.98 17.6 1.35E-15 0.86 1.10

10000 . .
P M = 5.06 · KESLOC 0.98
. .
± Standard Error

1000
Person-Months

100

10

. .
1 10 100 1000
KESLOC

FIGURE 5.22: SYS Dataset and CER Summary

 Cost Estimating Relationships 85

5.4.3.9 Test, Measurement and Diagnostic Equipment (TMDE)

P M = 7.42 · KESLOC 1.00 6

Data Points
6
4
Data Points = 11 4
KESLOC Min = 0.5 2 1
. . 0 0
KESLOC Max = 313.0 0
Aerial Ground Ground Maritime Ordnance
Vehicle Site Vehicle Vessel Vehicle
Operating Environment

0.4
R2 = 92% 0.2
Residuals

SE = 454 PM
PRED = 27% 0

−0.2

−0.4 .
.
−0.5 0 0.5 1 1.5 2 2.5
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 7.42 5.69 0.000297 3.34 16.5
B 1.00 10.11 3.26E-06 0.78 1.23

. .
P M = 7.42 · KESLOC 1.00
. .
± Standard Error
Person-Months

1000

100

10 . .
1 10 100
KESLOC

FIGURE 5.23: TMDE Dataset and CER Summary

86 Software Cost Estimation Metrics Manual for Defense Systems

5.4.3.10 Vehicle Control (VC)

P M = 9.05 · KESLOC 1.02 14

15

Data Points
10
10
Data Points = 27
KESLOC Min = 0.7 5 3
. . 0 0
KESLOC Max = 329.9 0
Aerial Ground Ground Maritime Ordnance
Vehicle Site Vehicle Vessel Vehicle
Operating Environment

0.4
R2 = 92%
0.2
Residuals

SE = 303 PM
PRED = 48% 0

−0.2

−0.4
.
.
0 0.5 1 1.5 2 2.5
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 9.05 11.0 4.52E-11 6.0 13.7
B 1.02 17.22 2.23E-15 0.90 1.14

. .
P M = 9.05 · KESLOC 1.02
. .
± Standard Error

1000
Person-Months

100

10

.
1. 10 100
KESLOC

FIGURE 5.24: VC Dataset and CER Summary

 Cost Estimating Relationships 87

5.4.3.11 Vehicle Payload (VP)

P M = 22.27 · KESLOC 0.81 10

Data Points
10
Data Points = 18 5
5
KESLOC Min = 1.1 0 0 1
0 . .
KESLOC Max = 221.1
Aerial Ground Ground Maritime Ordnance
Vehicle Site Vehicle Vessel Vehicle
Operating Environment

0.6
R2 = 89% 0.4
Residuals

SE = 111 PM
PRED = 56% 0.2

0
−0.2 .
.
0 0.5 1 1.5 2 2.5
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 22.27 13.8 2.62E-10 13.8 35.9
B 0.81 11.7 3.10E-09 0.66 0.95

. P M = 22.27 · .KESLOC 0.81

. .
± Standard Error
1000
Person-Months

100

10 .
1 10 100
KESLOC

FIGURE .5.25: VP Dataset and CER Summary

88 Software Cost Estimation Metrics Manual for Defense Systems

5.4.4 Application Type and Operating Environment CERs

The CERs in this section are derived from the number of data points in
Table 5.6 where the AppType and OpEnv intersect and the number of data
points is at least 10. These CER results are summarized in Table 5.9.

TABLE 5.9: CERs by AppType and OpEnv

KESLOC
Group Effort Equation N R 2 SE PRED Min Max
Aerial Vehicle (AV)
RTE P M = 15.09 · KESLOC 0.81 23 82% 293 52 2.0 164.2
SCP P M = 28.26 · KESLOC 0.86 11 92% 528 55 1.1 178.8
VC P M = 8.09 · KESLOC 1.05 10 96% 328 50 0.7 330.0
VP P M = 13.38 · KESLOC 0.91 10 95% 85 70 1.1 86.7

Ground System (GS)

CC P M = 6.66 · KESLOC 1.01 16 91% 257 63 5.0 229.0
CAS P M = 2.57 · KESLOC 1.03 13 79% 146 48 2.4 417.1
COM P M = 10.80 · KESLOC 0.83 22 82% 435 36 1.2 532.4
MP P M = 6.14 · KESLOC 0.86 20 77% 221 55 9.6 570.0
RTE P M = 8.71 · KESLOC 0.92 21 83% 149 48 1.8 112.7
SCP P M = 31.88 · KESLOC 0.79 14 86% 346 43 0.8 193.0
SYS P M = 7.93 · KESLOC 0.89 13 85% 941 31 10.7 842.1

Ground Vehicle (GV)

VC P M = 7.81 · KESLOC 1.10 14 78% 149 43 10.6 74.7

Maritime Vessel (MV)

COM P M = 5.77 · KESLOC 0.96 22 90% 17 68 0.4 27.1
 Cost Estimating Relationships 89

5.4.4.1 Command & Control - Ground Site

P M = 6.66 · KESLOC 1.01

Data Points = 16
KESLOC Min = 5.0
KESLOC Max = 229.0

0.2
Residuals

R2 = 91% 0
SE = 257 PM
PRED = 63%
−0.2 .
.
0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 6.66 5.73 5.24E-05 3.27 13.56
B 1.01 12.01 9.28E-09 0.83 1.19

. .
P M = 6.66 · KESLOC 1.01
. .
± Standard Error
100
Person-Months

10

.
1 10
KESLOC

FIGURE 5.26: CCGS Dataset and CER Summary

.
90 Software Cost Estimation Metrics Manual for Defense Systems

5.4.4.2 Custom AIS Software - Ground Site

P M = 2.57 · KESLOC 1.03

Data Points = 13
KESLOC Min = 2.4
KESLOC Max = 417.1

0.1
Residuals

R2 = 79% 0
SE = 146 PM
PRED = 48% −0.1
.
−0.2 .
0.5 1 1.5 2 2.5
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 2.57 4.66 6.89E-04 1.64 4.00
B 1.03 20.70 3.69E-10 0.92 1.13

. .
P M = 2.57 · KESLOC 1.03
1000 . .
± Standard Error
Person-Months

100

10

.
1 10
. KESLOC

FIGURE 5.27: CASGS Dataset and CER Summary

 Cost Estimating Relationships 91

5.4.4.3 Communications - Ground Site

P M = 10.80 · KESLOC 0.83

Data Points = 22
KESLOC Min = 1.2
KESLOC Max = 532.4

0.5
Residuals

2
R = 82%
SE = 435 PM 0
PRED = 36%
−0.5 .
.
0 0.5 1 1.5 2 2.5
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 10.80 7.75 1.90E-07 5.69 20.49
B 0.83 9.50 7.43E-09 0.65 1.01

. P M = 10.80 · .KESLOC 0.83

. .
± Standard Error
1000
Person-Months

100

10

.
1 10 100
.
KESLOC

FIGURE 5.28: COMGS Dataset and CER Summary

92 Software Cost Estimation Metrics Manual for Defense Systems

5.4.4.4 Communications - Maritime Vessel

P M = 5.77 · KESLOC 0.96

Data Points = 22
KESLOC Min = 0.4
KESLOC Max = 27.1

0.4
Residuals

0.2
R2 = 90%
SE = 17 PM 0
PRED = 68%
−0.2 .
.
−0.4−0.2 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6

Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 5.77 12.57 5.94E-11 4.31 7.71
B 0.96 13.40 1.88E-11 0.81 1.11

. .
P M = 5.77 · KESLOC 0.96
. .
± Standard Error
100
Person-Months

10

. .
1 10
KESLOC

FIGURE 5.29: COMMV Dataset and CER Summary

 Cost Estimating Relationships 93

5.4.4.5 Mission Planning - Ground Site

P M = 6.14 · KESLOC 0.86

Data Points = 20
KESLOC Min = 9.6
KESLOC Max = 570.0

0.6
0.4
Residuals

R2 = 77% 0.2
SE = 221 PM 0
PRED = 55% −0.2
.
−0.4 .
1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 6.14 3.68 0.0017 2.18 17.31
B 0.86 7.74 3.93E-07 0.63 1.09

. .
P M = 6.14 · KESLOC 0.86
. .
± Standard Error
100
Person-Months

10

.
1 10
KESLOC

FIGURE 5.30: MPGS Dataset and CER Summary

.
94 Software Cost Estimation Metrics Manual for Defense Systems

5.4.4.6 Real Time Embedded - Aerial Vehicle

P M = 15.09 · KESLOC 0.81

Data Points = 23
KESLOC Min = 2.0
KESLOC Max = 164.2

0.4

0.2
Residuals

2
R = 82% 0
SE = 293 PM −0.2
PRED = 52%
−0.4 .
.
0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4

Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 15.09 10.28 1.20E-09 8.71 26.14
B 0.81 9.95 2.13E-09 0.64 0.98

. P M = 15.09 · .KESLOC 0.81

. .
± Standard Error
100
Person-Months

10

.
1 10
KESLOC

FIGURE 5.31: RTEAV Dataset and CER Summary

.
 Cost Estimating Relationships 95

5.4.4.7 Real Time Embedded - Ground Site

P M = 8.71 · KESLOC 0.92

Data Points = 21
KESLOC Min = 1.8
KESLOC Max = 112.7

0.6

0.4
Residuals

0.2
R2 = 83%
SE = 149 PM 0
PRED = 48% −0.2
.
−0.4 .
0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 8.71 6.85 1.56E-06 4.49 16.88
B 0.92 9.63 9.60E-09 0.72 1.12

1000 . .
P M = 8.71 · KESLOC 0.92
. .
± Standard Error
Person-Months

100

10
.
1 10
KESLOC

. 5.32: RTEGS Dataset and CER Summary

FIGURE
96 Software Cost Estimation Metrics Manual for Defense Systems

5.4.4.8 Sensor Control and Signal Processing - Ground Vehicle

P M = 28.26 · KESLOC 0.86

Data Points = 11
KESLOC Min = 1.1
KESLOC Max = 178.8

0.2
Residuals

R2 = 92% 0
SE = 528 PM
PRED = 55%
−0.2 .
.
0 0.5 1 1.5 2
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 28.26 10.87 1.78E-06 14.10 56.66
B 0.86 9.92 3.81E-06 0.67 1.06

. P M = 28.26 · .KESLOC 0.86

. .
± Standard Error
100
Person-Months

10

.
1 10 100
KESLOC

FIGURE 5.33: SCPAV Dataset and CER Summary

.
 Cost Estimating Relationships 97

5.4.4.9 Sensor Control and Signal Processing - Ground Site

P M = 31.88 · KESLOC 0.79

Data Points = 14
KESLOC Min = 0.8
KESLOC Max = 193.0

0.2
Residuals

2
R = 86%
0
SE = 346 PM
PRED = 43% −0.2
.
.
0 0.5 1 1.5 2 2.5
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 31.88 12.02 4.74E-08 17.02 59.70
B 0.79 8.64 1.69E-06 0.59 0.99

. P M = 31.88 · .KESLOC 0.79

. .
± Standard Error
1000
Person-Months

100

10 .
1 10 100
KESLOC

FIGURE 5.34: SCPGS Dataset and CER Summary

.
98 Software Cost Estimation Metrics Manual for Defense Systems

5.4.4.10 System Software - Ground Site

P M = 7.93 · KESLOC 0.89

Data Points = 13
KESLOC Min = 10.7
KESLOC Max = 842.1

0.4
Residuals

R2 = 85% 0.2
SE = 941 PM 0
PRED = 31%
−0.2 .
.
1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8 3
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 7.93 3.78 0.0030 2.38 26.46
B 0.89 7.94 7.01E-06 0.64 1.14

1000
. .
P M = 7.93 · KESLOC 0.89
. .
± Standard Error
Person-Months

100

10

.
1 10 100
KESLOC

FIGURE 5.35: SYSGS Dataset and CER Summary

.
 Cost Estimating Relationships 99

5.4.4.11 Vehicle Control - Aerial Vehicle

P M = 8.09 · KESLOC 1.05

Data Points = 10
KESLOC Min = 0.7
KESLOC Max = 330.0

0.4

0.2
Residuals

R2 = 96% 0
SE = 328 PM
PRED = 50% −0.2
.
−0.4 .
0 0.5 1 1.5 2 2.5
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 8.09 8.80 2.18E-05 4.68 13.99
B 1.05 13.71 7.71E-07 0.87 1.22

. .
P M = 8.09 · KESLOC 1.05
. .
± Standard Error
1000
Person-Months

100

10

.
1. 10 100
KESLOC

FIGURE 5.36: VCAV Dataset and CER Summary

 100 Software Cost Estimation Metrics Manual for Defense Systems

5.4.4.12 Vehicle Control - Ground Vehicle

P M = 7.81 · KESLOC 1.10

Data Points = 14
KESLOC Min = 10.6
KESLOC Max = 74.7

0.2
Residuals

R2 = 78%
SE = 149 PM 0
PRED = 43%
−0.2 .
.
1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9

Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 7.81 3.72 0.0029 2.35 26.00
B 1.10 6.58 2.60E-05 0.73 1.46

. .
P M = 7.81 · KESLOC 1.10
100 . .
± Standard Error
Person-Months

10

.
1 10 100
KESLOC

FIGURE 5.37: VCGV Dataset and CER Summary

.
 Cost Estimating Relationships 101

5.4.4.13 Vehicle Payload - Aerial Vehicle

P M = 13.38 · KESLOC 0.91

Data Points = 10
KESLOC Min = 1.1
KESLOC Max = 86.7

0.2
Residuals

2
R = 95%
SE = 85 PM 0
PRED = 70%
−0.2 .
.
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
Log KESLOC

Coefficient Value T-Stat P-Value LCI UCI

A 13.38 12.31 1.76E-06 8.23 21.74
B 0.91 12.69 1.40E-06 0.74 1.07

1000 . P M = 13.38 · .KESLOC 0.91

. .
± Standard Error
Person-Months

100

10
.
1 10 100
KESLOC

FIGURE 5.38:
. VPAV Dataset and CER Summary
102 Software Cost Estimation Metrics Manual for Defense Systems

5.4.5 CER Effort Distribution

Using a CER produces an effort estimate given the size of software devel-
opment. The estimate is for all activities covered by the data on which the
CER was derived (see Section 4.5.2 on standardizing effort). The total effort
estimate can be distributed across different development activities.
Average effort percentages for each activity consistently covered by SRDR
data were derived for each Application Type with 10 or more data points.
The total estimated effort is distributed across development activities using
the percentages in Table 5.10.

TABLE 5.10: Effort Distributions

Application Requirement Architecture Code & Integration
Type Analysis & Design Unit Test & QT
CAS 11.6% 27.8% 35.6% 25.0%
C&C 20.6% 15.8% 28.9% 34.8%
MP 16.2% 12.3% 50.8% 20.7%
RTE 15.5% 26.7% 26.7% 31.2%
S&S 7.4% 39.9% 32.1% 20.7%
SCP 10.8% 45.2% 20.3% 23.7%
SYS 17.6% 21.1% 28.8% 32.5%
VC 18.5% 23.6% 31.3% 26.6%

5.5 SRDR-Derived Benchmarks

5.5.1 Benchmark Model and Evaluation
The purpose of software benchmarks are for comparisons between groups
of software, and not estimation. The benchmarks provide a sense of which
groups are more or less expensive to produce. The benchmarks presented here
are expressed as productivity numbers.
Software productivity is the ratio of outputs to inputs per Equation 5.5.
The size, expressed as equivalent SLOC, represents the amount of functionality
implemented. The effort represents the amount of labor required to implement
the functionality as influenced by the inputs discussed above.

P roductivity = Outputs/Inputs = ESLOC/P M (5.5)

The metric used to express software productivity is Equivalent Source

Lines of Code (ESLOC) per Person-Month (PM) of effort. While many other
measures exist, ESLOC / PM will be used because most of the data collected
 Cost Estimating Relationships 103

by the DoD on past projects is captured using these two measures. While con-
troversy exists over whether or not ESLOC / PM is a good measure, consistent
use of this metric (see Metric Definitions) provides for meaningful comparisons
of productivity.
Table 5.6 shows the coverage by the number of data points analyzed by Op-
erating Environment and Application Type. A minimum of 10 or more projects
were required to derive a productivity benchmark. Groups with smaller sample
sizes weren’t considered large enough to draw conclusions from. The produc-
tivity results contain the statistics defined in Table 5.11.

5.5.2 Operating Environment Benchmarks

Table 5.12 shows the productivity statistics for the Operating Environ-
ments (OpEnv). They are sorted alphabetically with the measures per Table
5.11 for assessment.
For example, the Aerial Vehicle (AV) Operating Environment in Table 5.12
has 80 data points for analysis. The mean productivity was 137 ESLOC/PM.
The lower 95% confidence interval is 119 ESLOC/PM and the upper is 156
ESLOC/PM. This means that with 95% confidence, the true population mean
for Aerial Vehicle productivity is between 119 and 156 ESLOC/PM. The SE,
standard error, is 9.4 ESLOC/PM and is used to calculate the confidence
intervals. The Standard Deviation (SD) shows the range where 68% of the data
points lie above or below the mean, 54 - 222 ESLOC/PM. Smaller Standard
Deviations are desirable to draw benchmarking conclusions.
The median productivity is 118 ESLOC/PM. This is less than the mean
productivity and indicates a non-normal data distribution. In situations where
the data is not normally distributed, the median productivity is a more
reliable value. The first and third quartiles, Q1 and Q3, show the range
where 50% of the data points lie above and below the median, 78 to 172
ESLOC/PM. Narrower ranges are desirable.
A conclusion that may be made for the Aerial Vehicle environment is the
mean or median productivity is based on a wide range of individual data
point productivities. The large spread of productivities indicates the mean
and median productivity are not highly reliable to meaningfully benchmark
this environment.
The benchmark values in Table 5.12 are further visualized in box and
whisker plots in Figure 5.39. The boxes are bounded by the 1st and 3rd quar-
tiles, Q1 and Q3, with the median shown as a line in the box. The tails
represent values beyond the 1st and 3rd quartiles. The lower whisker on each
is the smallest data value which is larger than the lower quartile. The upper
whisker is the largest data value which is smaller than the upper quartile.
Data outliers beyond these ranges are indicated with marks.
The widths of the boxes in Figure 5.39 are also proportional to the sample
sizes for the groups. Operating Environments with wider boxes had more data
used which is desirable.
104 Software Cost Estimation Metrics Manual for Defense Systems

TABLE 5.11: Productivity Statistic Definitions

Statistic Description
N Number of data points

Min ESLOC Minimum size in thousands of equivalent source lines of

code

Max ESLOC Maximum size in thousands of equivalent source lines of

code

LCI Lower Confidence Interval is an estimate of an interval be-

low the sample mean within which the population mean is
estimated to lie

Mean Estimated sample value representing the population central

value; equal to the sum of the values divided by the number
of values, i.e., arithmetic mean

UCI Upper Confidence Interval is an estimate of an interval

above the sample mean within which the population mean
is estimated to lie

SE Standard Error of the mean is an estimate computed from

the sample of data being analyzed of the standard devia-
tion of sample means over all possible samples (of a given
size) drawn from the population. It is found by dividing the
StdDev by the square root of N.

StdDev Standard Deviation measures the amount of variation or

dispersion from the mean in a sample. Plus or minus one
standard deviation from the mean is a range that includes
about 68% of the data.

Q1 Numerical value for the lower 25% of ranked data (1st Quar-
tile), i.e., the value half way between the lowest value and
the median in a set of ranked values

Median Numerical value separating the higher half of a sample from

the lower half, i.e., the middle value in a set of ranked values

Q3 Numerical value for the lower 75% of ranked data (3rd

Quartile), i.e. the value half way between the median and
the highest value in a set of ranked values
 Cost Estimating Relationships 105

TABLE 5.12: Productivity Benchmarks by Operating Environment

Productivity (ESLOC/PM)
OpEnv N LCI Mean UCI SE SD Q1 Mdn. Q3
AV 80 118.9 137.6 156.3 9.4 84.1 77.7 117.6 172.0
GF 133 186.5 208.2 230.0 11.0 126.8 118.2 192.0 280.0
GV 26 110.2 140.6 171.0 14.8 75.3 89.9 121.0 169.5
MV 42 138.9 163.0 187.2 12.0 77.5 88.1 178.8 217.3
OV 25 82.8 122.5 162.1 19.2 96.2 59.1 95.4 161.4

Figure 5.39 shows that while the operating environments are physically
distinct, the productivities are indistinguishable. By this, productivity is not
influenced by the operating environment. There are two caveats to this con-
clusion:
1. There is the lack of data for the space vehicle, SV, environment. Initial
indications are the SV environment has the lowest productivity.
2. There was not enough data to separate each environment into manned
and unmanned sub-environments. Some cost estimators feel the manned
environment has a lower productivity that an unmanned one.

AV
GF
GV
MV
OV .
.
0 200 400 600 800 1,000
ESLOC/PM

FIGURE 5.39: Operating Environment Productivities Boxplot

5.5.3 Application Type Benchmarks

Table 5.13 shows the mean and median productivity across Application
Types (AppType). To be included in the table, there had to be 10 or more
data points in a application type group. The rows are sorted alphabetically.
Figure 5.40 shows a boxplot comparison of the productivities across appli-
cation types summarized in Table 5.13. The box widths are proportional to
the sample size for each group. Plot interpretation is described in the previous
section.
106 Software Cost Estimation Metrics Manual for Defense Systems

TABLE 5.13: Productivity Benchmarks by Application Type

Productivity (ESLOC/PM)
AppType N LCI Mean UCI SE SD Q1 Mdn. Q3
C&C 33 120.5 140.7 160.8 9.9 56.8 103.1 128.2 177.8
CAS 16 309.6 353.4 396.9 20.5 82.2 292.1 323.0 406.6
COM 47 167.8 189.5 211.2 10.8 74.0 139.6 185.0 243.1
MP 20 263.4 335.0 406.3 34.2 153.0 207.5 328.7 427.1
RTE 57 125.1 142.7 160.2 8.7 66.1 84.0 141.1 171.6
S&S 17 159.6 211.5 263.1 24.5 100.9 129.5 209.8 259.9
SCP 36 51.6 60.2 68.8 4.2 25.5 40.1 55.7 79.5
SYS 27 196.3 230.8 265.2 16.8 87.2 162.8 235.4 298.5
TMDE 11 88.3 160.5 231.9 32.4 107.5 83.7 118.3 205.6
VC 27 94.5 115.2 135.9 10.1 52.4 70.2 110.1 126.4
VP 18 68.0 88.5 108.8 9.7 41.1 43.0 90.9 119.5

C&C
CAS
COM
MP
RTE
S&S
SCP
SYS
TMDE
VC
VP .
.
0 100 200 300 400 500 600 700 800
ESLOC/PM

FIGURE 5.40: Application Type Productivities Boxplot

 Cost Estimating Relationships 107

The productivities of some of the Application Types are indistinguishable

from each other. Yet the SCP, VP, and VC productivities are different than
the productivities for CAS, MP, SYS and S&S. This suggests that groups of
application types may be distinguished by productivity.

5.5.4 AppType & OpEnv Benchmarks

Table 5.14 shows the productivity means, medians and related statistics
across 13 Application Types and Operating Environment subgroups. These
are sorted by Operating Environment for each subgroup with 10 or more data
points. These benchmarks are shown in the Figure 5.41 boxplot.

Aerial Vehicle
RTE
SCP
VC
VP

Ground System
C&C
CAS
COM
MP
RTE
SCP
SYS

Ground Vehicle
VC

Maritime Vehicle
COM .

.
0 100 200 300 400 500 600 700 800
ESLOC/PM

FIGURE 5.41: Application Type and Operating Environment Productivities

Boxplot
108 Software Cost Estimation Metrics Manual for Defense Systems

TABLE 5.14: Productivity Benchmarks by Operating Environment and Ap-

plication Type

Productivity (ESLOC/PM)
Subgroup N LCI Mean UCI SE SD Q1 Mdn. Q3
Aerial Vehicle (AV)
RTE 23 102.1 133.7 165.2 15.2 73.1 79.6 124.0 164.7
SCP 11 42.3 60.1 77.7 8.0 26.5 41.0 51.2 94.0
VC 10 82.2 122.2 161.6 17.7 55.9 76.4 110.9 169.3
VP 10 75.3 99.7 123.8 10.8 34.1 80.8 92.7 121.7

Ground System (GS)

C&C 16 126.2 153.6 180.9 12.9 51.4 105.4 143.4 199.8
CAS 13 309.9 363.4 416.5 24.6 88.6 288.1 353.1 425.9
COM 22 146.5 186.7 226.8 19.3 90.6 137.3 175.3 243.2
MP 20 263.4 335.0 406.3 34.2 153.0 207.5 328.7 427.1
RTE 21 133.4 162.4 191.3 13.9 63.7 133.4 158.6 198.3
SCP 14 46.1 63.9 81.6 8.2 30.8 31.4 64.8 85.7
SYS 13 174.8 235.6 295.8 27.9 100.6 144.5 246.2 310.2

Ground Vehicle (GV)

VC 14 79.8 97.9 116.0 8.4 31.4 65.6 104.0 120.7

Maritime Vessel (MV)

COM 22 169.7 193.8 218.0 11.6 54.6 172.3 190.0 224.8

5.6 Limitations and Future Work

All the CERs are amenable to change with new data. They reflect only the
current dataset and for continuous CER improvement they will be reexamined
and derived anew when more SRDR data is available. Future work will also
expand the number of CERs for application types and operating environments.
A major limitation of the CERs presented here is the lack of a correspond-
ing schedule estimating relationship, SER. The presented CERs represent the
nominal effort required to develop a CSCI of a specific size. But there is no
information on what development duration is assumed by the nominal effort
estimate.
The amount of allotted development time impacts the amount of required
 Cost Estimating Relationships 109

effort and therefore cost. Development time can be shortened within reason
with more people working on the project. Future work will focus on developing
SERs to show a nominal duration for a nominal effort estimate and the impact
on effort when duration is shortened.
The set of CERs presented are based on software size expressed as esti-
mated SLOC. As discussed in Chapter 1, these CERs would be most useful in
the early acquisition phase of a program. Therefore it would be useful to use
a size metric that is available in the early phases such as requirements counts.
Future work will investigate the use of requirements counts, as reported in the
initial SRDR, as a size metric in CERs.
110 Software Cost Estimation Metrics Manual for Defense Systems
Chapter 6
Modern Estimating Challenges

Several trends present significant future challenges for the sizing and cost
estimation of 21st century software systems. Prominent among these trends
are:
• Rapid change, emergent requirements, and evolutionary development
• Net-centric systems of systems
• Model-Driven and Non-Developmental Item (NDI)-intensive systems
• Ultrahigh software system assurance
• Legacy maintenance and brownfield development
• Agile and Kanban development
This chapter summarizes each trend and elaborates on its challenges for
software sizing and cost estimation.

6.1 Changing Objectives, Constraints and Priorities

6.1.1 Rapid Change, Emergent Requirements, and Evolu-
tionary Development
21st century software systems will encounter increasingly rapid change
in their objectives, constraints, and priorities. This change will be necessary
due to increasingly rapid changes in their competitive threats, technology,
organizations, leadership priorities, and environments. It is thus increasingly
infeasible to provide precise size and cost estimates if the systems’ require-
ments are emergent rather than pre-specifiable. This has led to increasing
use of strategies such as incremental and evolutionary development, and to
experiences with associated new sizing and costing phenomena such as the
Incremental Development Productivity Decline. It also implies that measur-
ing the system’s size by counting the number of source lines of code (SLOC)
in the delivered system may be an underestimate, as a good deal of software
may be developed and deleted before delivery due to changing priorities.
There are three primary options for handling these sizing and estimation
challenges. The first is to improve the ability to estimate requirements volatil-
ity during development via improved data collection and analysis, such as the

111
112 Software Cost Estimation Metrics Manual for Defense Systems

use of code counters able to count numbers of SLOC added, modified, and
deleted during development [37] .
If such data is unavailable, the best one can do is to estimate ranges of
requirements volatility . For uniformity, Table 6.1 presents a recommended
set of Requirements Volatility (RVOL) ranges over the development period
for rating levels of 1 (Very Low) to 5 (Very High), such as in the DoD SRDR
form [44].

TABLE 6.1: Recommended RVOL Rating Levels

Rating Level RVOL Range RVOL Average

Very Low 0-6% 3%
Low 6-12% 9%
Nominal 12-24% 18%
High 24-48% 36%
Very High >48% 72%

For incremental and evolutionary development projects, the second option

is to treat the earlier increments as reused software, and to apply reuse factors
to them (such as the percent of the design, code, and integration modified,
perhaps adjusted for degree of software understandability and programmer
unfamiliarity [8].
This can be done either uniformly across the set of previous increments,
or by having these factors vary by previous increment or by subsystem. This
will produce an equivalent-SLOC (ESLOC) size for the effect of modifying
the previous increments, to be added to the size of the new increment in
estimating effort for the new increment. In tracking the size of the overall
system, it is important to remember that these ESLOC are not actual lines
of code to be included in the size of the next release.
The third option is to include an Incremental Development Productivity
Decline (IDPD) factor, or perhaps multiple factors varying by increment or
subsystem. Unlike hardware, where unit costs tend to decrease with added
production volume, the unit costs of later software increments tend to in-
crease, due to previous-increment breakage and usage feedback, and due to
increased integration and test effort. Thus, using hardware-driven or tradi-
tional software-driven estimation methods for later increments will lead to
underestimates and overruns in both cost and schedule.
A relevant example was a large defense software system that had the fol-
lowing characteristics:

• 5 builds, 7 years, $100M

• Build 1 productivity over 300 SLOC / person-month
• Build 5 productivity under 150 SLOC / person-month
– Including Build 1-4 breakage, integration, rework
 Modern Estimating Challenges 113

– 318% change in requirements across all builds

A factor-of-2 decrease in productivity across four new builds corresponds

to an average build-to-build IDPD factor of 19%. A recent quantitative IDPD
analysis of a smaller software system yielded an IDPD of 14%, with significant
variations from increment to increment [54]. Similar IDPD phenomena have
been found for large commercial software such as the multi-year slippages in
the delivery of Microsoft’s Word for Windows [20] and Windows Vista, and
for large agile-development projects that assumed a zero IDPD factor [16].
Based on experience with similar projects, the following impact causes
and ranges per increment are conservatively stated in the Table 6.2. The de-
creased effort due to more experienced personnel assumes a reasonable initial
experience level.

TABLE 6.2: IDPD Effort Drivers

Less effort due to more experienced personnel
Variation depending on personnel turnover rates 5-20%

More effort due to code base growth

Breakage, maintenance of full code base 20-40%
Diseconomies of scale in development, integration 10-25%
Requirements volatility, user requests 10-25%

In the best case, there would be 20% more effort (from above -
20+20+10+10); for a 4-build system, the IDPD would be 6%. In the worst
case, there would be 85% more effort (from above 40+25+25-5); for a 4-build
system, the IDPD would be 23%.
With fixed staff size, there would be either a schedule increase or incom-
plete builds in any case. The difference between 6% and 23% may not look
too serious, but the cumulative effects on schedule across a number of builds
is very serious.
A simplified illustrative model relating productivity decline to number of
builds needed to reach 4M ESLOC across 4 builds follows. Assume that the
two-year Build 1 production of 1M SLOC can be developed at 200 SLOC /
PM for a total of 5000 PM. This means it will need 208 developers (5000 PM
/ 24 Months), assuming a constant staff size for all builds.
The analysis shown in Figure 6.1 shows the impact on the amount of
software delivered per build and the resulting effect on the overall delivery
schedule as a function of the IDPD factor. Many incremental development
cost estimates assume an IDPD of zero, and an on-time delivery in 4 builds.
However, as the IDPD factor increases and the staffing level remains constant,
the productivity decline per build stretches the schedule out to twice as long
for an IDPD of 20%.
Thus, it is important to understand the IDPD factor and its influence
114 Software Cost Estimation Metrics Manual for Defense Systems

4M

Cumulative ESLOC 3M

. .
0% productivity decline
2M
. .
10% productivity decline
. .
15% productivity decline
. .
20% productivity decline
1M .
.
1 2 3 4 5 6 7 8
Build

FIGURE 6.1: Effects of IDPD on Multiple Builds

when doing incremental or evolutionary development. Ongoing research [35]

indicates that the magnitude of the IDPD factor may vary by type of applica-
tion (infrastructure software having higher IDPDs since it tends to be tightly
coupled and touches everything; applications software having lower IDPDs if
it is architected to be loosely coupled), or by recency of the build (older builds
may be more stable). Research in [35] and [36] determined that IDPD factors
were generally not constant, due to personnel turnover, COTS volatility, or
pro-active productivity enablers. Further data collection and analysis would
be very helpful in improving the understanding of the IDPD factor.

6.1.2 Net-centric Systems of Systems (NCSoS)

If one is developing software components for use in a NCSoS, changes in
the interfaces between the component systems and independently-evolving
NCSoS-internal or NCSoS-external systems will add further effort. The
amount of effort may vary by the tightness of the coupling among the systems;
the complexity, dynamism, and compatibility of purpose of the independently-
evolving systems; and the degree of control that the NCSoS protagonist has
over the various component systems. The latter ranges from Directed SoS
(strong control), through Acknowledged (partial control) and Collaborative
(shared interests) SoSs , to Virtual SoSs (no guarantees) [40].
For estimation, one option is to use requirements volatility as a way to
assess increased effort. Another is to use existing models such as COSYSMO
[55] to estimate the added coordination effort across the NCSoS [27]. A third
 Modern Estimating Challenges 115

approach is to have separate models for estimating the systems engineering,

NCSoS component systems development, and NCSoS component systems in-
tegration to estimate the added effort [28].

6.1.3 Model-Driven and Non-Developmental Item (NDI)-

Intensive Development
Model-driven development and Non-Developmental Item (NDI)-intensive
development are two approaches that enable large portions of software-
intensive systems to be generated from model directives or provided by NDIs
such as Commercial-Off-The-Shelf (COTS) components, open source compo-
nents, and purchased services such as Cloud services.
Such applications are highly cost-effective, but present several sizing and
cost estimation challenges:
• Model directives generate source code in Java, C++, or other third-
generation languages, but unless the generated SLOC are going to be
used for system maintenance, their size as counted by code counters
should not be used for development or maintenance cost estimation. In
general, auto-generated code is not efficient, does not have meaningful
variable names for the internal logic, is relatively cryptic with no com-
ments unless the developer adds them.
• Counting model directives is possible for some types of model-driven
development, but presents significant challenges for others (e.g., GUI
builders).
• Except for customer-furnished or open-source software that is expected
to be modified, the size of NDI components should not be used for
estimating.
• A significant challenge is to find appropriately effective size measures
for such NDI components. One approach is to use the number and com-
plexity of their interfaces with each other or with the software being
developed. Another is to count the amount of glue-code SLOC being
developed to integrate the NDI components, with the proviso that such
glue code tends to be about 3 times as expensive per SLOC as regularly-
developed code [9]. A similar approach is to use the interface elements
of function points for sizing [17].
• A further challenge is that much of the effort in using NDI is expended in
assessing candidate NDI components and in tailoring them to the given
application. Some initial guidelines for estimating such effort are pro-
vided in the COnstructive Commercial-off-the Shelf (COCOTS) model
[1].
• Another challenge is that the effects of COTS and Cloud-services evolu-
tion are generally underestimated during software maintenance. COTS
116 Software Cost Estimation Metrics Manual for Defense Systems

products generally provide significant new releases on the average of

about every 10 months, and generally become unsupported after three
new releases. With Cloud services, one does not have the option to de-
cline new releases, and updates occur more frequently. One way to es-
timate this source of effort is to consider it as a form of requirements
volatility.

• Another serious concern is that functional size measures such as function

points, use cases, or requirements will be highly unreliable until it is
known how much of the functionality is going to be provided by NDI
components or Cloud services.

6.1.4 Ultrahigh Software Systems Assurance

The increasing criticality of software to the safety of transportation ve-
hicles, medical equipment, or financial resources; the security of private or
confidential information; and the assurance of 24 / 7 Internet, web, or Cloud
services will require further investments in the development and certification
of software than are provided by most current software-intensive systems.
While it is widely held that ultrahigh-assurance software will substantially
raise software project cost, different models vary in estimating the added cost.
For example, the 1990’s Softcost-R model estimates a factor of 3.43 [49]; the
SEER model uses a similar value of 3.47 [17].
A recent experimental extension of the COCOMO II model called
COSECMO used the 7 Evaluated Assurance Levels (EALs) in the ISO Stan-
dard Common Criteria for Information Technology Security Evaluation (CC)
[24], and quoted prices for certifying various EAL security levels to provide an
initial estimation model in this context [15]. Its added-effort estimates were a
function of both EAL level and software size: its multipliers for a 5000-SLOC
secure system were 1.50 for EAL 4 and 8.8 for EAL 7.
A further sizing challenge for ultrahigh-assurance software is that it re-
quires more functionality for such functions as security audit, communication,
cryptographic support, data protection, etc. These may be furnished by NDI
components or may need to be developed for special systems.

6.1.5 Legacy Maintenance and Brownfield Development

Fewer and fewer software-intensive systems have the luxury of starting
with a clean sheet of paper or whiteboard on which to create a new Greenfield
system. Most software-intensive systems are already in maintenance; [12] esti-
mated in 2009 that there were roughly 200 billion SLOC in service worldwide.
Also, most new applications need to consider continuity of service from the
legacy system(s) they are replacing. Many such applications involving incre-
mental development have failed because there was no way to separate out the
incremental legacy system capabilities that were being replaced. Thus, such
 Modern Estimating Challenges 117

applications need to use a Brownfield development approach that concurrently

architects the new version and its increments, while re-engineering the legacy
software to accommodate the incremental phase-in of the new capabilities [23]
[29] [6].
Traditional software maintenance sizing models have determined an equiv-
alent SLOC size by multiplying the size of the legacy system by its Annual
Change Traffic (ACT) fraction (% of SLOC added + % of SLOC modified)
/ 100. The resulting equivalent size is used to determine a nominal cost of
a year of maintenance, which is then adjusted by maintenance-oriented effort
multipliers. These are generally similar or the same as those for development,
except for some, such as required reliability and degree of documentation, in
which larger development investments will yield relative maintenance savings.
Some models such as SEER [17] include further maintenance parameters such
as personnel and environment differences. An excellent summary of software
maintenance estimation is in Stutzke [53].
However, as legacy systems become larger and larger (a full-up BMW con-
tains roughly 100 million SLOC [13], the ACT approach becomes less stable.
The difference between an ACT of 1% and an ACT of 2% when applied to
100 million SLOC is 1 million SLOC. A recent revision of the COCOMO II
software maintenance model sizes a new release as

ESLOC = 2·(M odif ied SLOC)+Added SLOC+0.5·(Deleted SLOC) (6.1)

The coefficients are rounded values determined from the analysis of data
from 24 maintenance activities [37], in which the modified, added, and deleted
SLOC were obtained from a code counting tool. This model can also be used
to estimate the equivalent size of re-engineering legacy software in Brown-
field software development. At first, the estimates of legacy SLOC modified,
added, and deleted will be very rough, and can be refined as the design of the
maintenance modifications or Brownfield re-engineering is determined.

6.1.6 Agile and Kanban Development

The difficulties of software maintenance estimation can often be mitigated
by using workflow management techniques such as Kanban [3]. In Kanban,
individual maintenance upgrades are given Kanban cards (Kanban is the
Japanese word for card; the approach originated with the Toyota Produc-
tion System). Workflow management is accomplished by limiting the number
of cards introduced into the development process, and pulling the cards into
the next stage of development (design, code, test, release) when open capacity
is available (each stage has a limit of the number of cards it can be processing
at a given time). Any buildups of upgrade queues waiting to be pulled forward
are given management attention to find and fix bottleneck root causes or to
rebalance the manpower devoted to each stage of development. A key Kanban
principle is to minimize work in progress.
118 Software Cost Estimation Metrics Manual for Defense Systems

An advantage of Kanban is that if upgrade requests are relatively small,

non-critical and uniform, that there is no need to estimate their required effort;
they are pulled through the stages as capacity is available, and if the capacities
of the stages are well-tuned to the traffic, work gets done on schedule. However,
if a too-large upgrade is introduced into the system, it is likely to introduce
delays as it progresses through the stages. Thus, some form of estimation is
necessary to determine right-size upgrade units, but it does not have to be
precise as long as the workflow management pulls the upgrade through the
stages. For familiar systems, performers will be able to right-size the units.
For Kanban in less-familiar systems, and for sizing builds in agile methods
such as Scrum, group consensus techniques such as Planning Poker [14] or
Wideband Delphi [5] can generally serve this purpose.
The key point here is to recognize that estimation of knowledge work
can never be perfect, and to create development approaches that compen-
sate for variations in estimation accuracy. Kanban is one such; another is the
agile methods’ approach of timeboxing or schedule-as-independent-variable
(SAIV), in which maintenance upgrades or incremental development features
are prioritized, and the increment architected to enable dropping of features
to meet a fixed delivery date (With Kanban, prioritization occurs in determin-
ing which of a backlog of desired upgrade features gets the next card). Such
prioritization is a form of value-based software engineering, in that the higher-
priority features can be flowed more rapidly through Kanban stages [3], or in
general given more attention in defect detection and removal via value-based
inspections or testing [11], [30]. Another important point is that the ability to
compensate for rough estimates does not mean that data on project perfor-
mance does not need to be collected and analyzed. It is even more important
as a sound source of continuous improvement and change adaptability efforts.

6.1.7 Putting It All Together at the Large-Project or

Enterprise-Level
The biggest challenge of all is that the six challenges above need to be
addressed concurrently. Suboptimizing on individual-project agility runs the
risks of easiest-first lock-in to unscalable or unsecurable systems, or of produc-
ing numerous incompatible stovepipe applications. Suboptimizing on security
assurance and certification runs the risks of missing early-adopter market win-
dows, of rapidly responding to competitive threats, or of creating inflexible,
user-unfriendly systems.
One key strategy for addressing such estimation and performance chal-
lenges is to recognize that large systems and enterprises are composed of
subsystems that have different need priorities and can be handled by different
estimation and performance approaches. Real-time, safety-critical control sys-
tems and security kernels need high assurance, but are relatively stable. GUIs
need rapid adaptability to change, but with GUI-builder systems, can largely
compensate for lower assurance levels via rapid fixes. A key point here is that
 Modern Estimating Challenges 119

for most enterprises and large systems, there is no one-size-fits-all method of

sizing, estimating, and performing.

6.2 Estimation Approaches for Different Processes

This diversity of application complexities, objectives and constraints im-
plies a need for guidance on what kind of process to use for what kind of
system or subsystem, and on what kinds of sizing and estimation capabilities
fit what kinds of processes. A start toward such guidance is provided in [10].
Figure 6.2 summarizes the traditional single-step waterfall process plus
several forms of incremental development, each of which meets different com-
petitive challenges and which are best served by different cost estimation ap-
proaches. The time phasing of each form is expressed in terms of the increment
1, 2, 3, ... content with respect to the Rational Unified Process (RUP) phases
of Inception (I), Elaboration (E), Construction (C), and Transition (T).
The “Single Step” model is the traditional waterfall model, in which the
requirements are pre-specified, and the system is developed to the require-
ments in a single increment. Single-increment parametric estimation models,
complemented by expert judgment, are best for this process.
The “Pre-specified Sequential” incremental development model is not evo-
lutionary. It just splits up the development in order to field an early Initial Op-
erational Capability, followed by several Pre-Planned Product Improvements
(P3Is). When requirements are pre-specifiable and stable, it enables a strong,
predictable process. When requirements are emergent and / or rapidly chang-
ing, it often requires expensive rework to undo architectural commitments.
Cost estimation can be performed by sequential application of single-step
parametric models plus the use of an IDPD factor, or by parametric model
extensions supporting increments, including options for increment overlap and
breakage of existing increments, such as the COCOMO II Incremental Devel-
opment Model described in [8].
The “Evolutionary Sequential” model rapidly develops an initial opera-
tional capability and upgrades it based on operational experience. Pure agile
software development fits this model: if something is wrong, it will be fixed
in 30 days in the next release. Rapid fielding also fits this model for larger or
hardware-software systems. Its strength is getting quick-response capabilities
into the field. For pure agile, it can fall prey to an easiest-first set of archi-
tectural commitments which break when, for example, it tries to add security
or scalability as a new feature in a later increment. For rapid fielding, it may
be expensive to keep the development team together while waiting for usage
feedback, but it may be worth it. For small agile projects, group consensus
techniques such as Planning Poker are best; for larger projects, parametric
models with an IDPD factor are best.
120 Software Cost Estimation Metrics Manual for Defense Systems

Single Step

C
E T
I

Prespecified Sequential

C1 C2 C3
E1 T1 T2 T3 ...
I1

Evolutionary Sequential

C1 C2 C3
E1 T1 E2 T2 E3 T3 ...
.I1 I2 I3

Evolutionary Overlapped

C1
E1 T1
I1

C2
E2 T2
I2

C3
E3 T3 ...
I3

Evolutionary Concurrent

C1 C2 C3
E1 T1 T2 T3 ...
I1

I2 E2 I3 E3 I4 E4

FIGURE 6.2: Summary of Different Processes

 Modern Estimating Challenges 121

“Evolutionary Overlapped” covers the special case of deferring the next

increment until critical enablers such as desired new technology, anticipated
new commercial product capabilities, or needed funding become available or
mature enough to be added.
“Evolutionary Concurrent” has the systems engineers handling the change
traffic and re-baselining the plans and specifications for the next increment,
while keeping the development stabilized for the current increment. Its exam-
ple and pros and cons are provided Table 6.3. All Cost Estimation approaches
also include an expert-judgment cross-check.

TABLE 6.3: Situation-Dependent Processes and Estimation Approaches

Type Examples Pros Cons Cost Estimation

Single Step Stable; High Pre- Emergent Single-increment
Assurance specifiable requirements parametric
full- or rapid estimation models
capability change
requirements

Pre-specified Platform Pre- Emergent COINCOMO or

Sequential base plus specifiable requirements repeated
P3Is full- or rapid single-increment
capability change parametric model
requirements estimation with
IDPD

Evolutionary Small: Agile Adaptability Easiest-first; Small: Planning-

Sequential to change late, costly poker-type
Large: breakage
Evolutionary Large: Parametric
Development with IDPD and
Requirements
Volatility

Evolutionary COTS- Immaturity Delay may Parametric with

Overlapped intensive risk be noncom- IDPD and
systems avoidance petitive Requirements
Volatility

Evolutionary Mainstream High Highly COINCOMO with

Concurrent product lines assurance coupled IDPD for
with rapid systems with development;
Systems of change very rapid COSYSMO for
systems change re-baselining
122 Software Cost Estimation Metrics Manual for Defense Systems

Table 6.4 provides criteria for deciding which of the five classes of incre-
mental and evolutionary acquisition (EvA) defined in Table 6.3 to use, plus
the choice of non-incremental, single-step development.

TABLE 6.4: Process Selection Criteria

Process Stable pre- OK to wait Need to Need to
Type specifiable for full wait for wait for
require- system to be next- next-
ments? developed? increment increment
priorities? enablers?
Single Step Yes Yes - -

Pre- Yes No - -
specified
Sequential

Evolutionary No No Yes -
Sequential

Evolutionary No No No Yes
Overlapped

Evolutionary No No No No
Concurrent

The “Single-Step-to-Full-Capability” process exemplified by the tradi-

tional waterfall or sequential Vee model is appropriate if the products require-
ments are pre-specifiable and have a low probability of significant change; and
if there is no value in or opportunity to deliver a partial product capability. A
good example would be the hardware portion of a geosynchronous satellite.
The “Pre-specified Sequential” process is best if the product’s require-
ments are pre-specifiable and have a low probability of significant change;
and if waiting for the full system to be developed incurs a loss of impor-
tant and deliverable incremental mission capabilities. A good example would
be a well-understood and well-prioritized sequence of software upgrades to a
programmable radio.
The “Evolutionary Sequential” process is best when there is a need to
get operational feedback on a quick-response capability before defining and
developing the next increment’s content. Agile methods fit into this category,
as do systems undergoing rapid competitive change.
The “Evolutionary Overlapped” process is best when one does not need
to wait for operational feedback, but may need to wait for next-increment
enablers such as technology maturity, external system capabilities, or needed
resources. A good example is the need to wait for a mature release of an
 Modern Estimating Challenges 123

anticipated commercial product. The Evolutionary Concurrent process is best

when the enablers are available, but there is a great deal of change traffic to
be handled that would destabilize the team developing the current increment.
Examples may be new competitive threats, emergent user capability needs,
external system interface changes, technology matured on other programs, or
COTS upgrades.
As is frequently stated, “Prediction is difficult, especially about the fu-
ture.” Emerging trends such as Internet of Things, Big Data Analytics, social
networking and crowdsourcing, self-modifying autonomic systems, megacore
parallel processing chips, and human-capability augmentation devices will
present further challenges in estimating costs of their software. Even more
challenging game-breakers lie ahead.
124 Software Cost Estimation Metrics Manual for Defense Systems
Chapter 7
Estimation Process

7.1 Overview
The software estimation process described in this chapter is based on
the U.S. Government Accountability Office Cost Estimating and Assessment
Guide [18]. The recommended phases and steps within each phase are ex-
plained in terms of software cost estimation. Not all of the steps are described,
as some of the steps are specific to how an organization organizes, executes
and presents estimates. Where applicable, references to other sections of this
manual are used to provide examples of employing metrics data assessment
and analysis.
A system level cost estimate involves many cost elements, and the software
cost estimate is only a part. This chapter discusses a software estimation
process in the context of a system estimate. The following generic system cost
element list shows where this software estimation process applies:
Elements covered by this estimation process:

• Integration, assembly, test, and checkout

All effort of technical and functional activities associated with software
required to assemble equipment (hardware and software) elements into
mission equipment (hardware and software). This includes the activ-
ities of software requirements analysis, software architecting, software
design, coding and unit testing, software integration and testing, and
qualification testing.

Elements not covered by this estimation process:

• System engineering The technical and management efforts of direct-

ing and controlling a totally integrated engineering effort of a system or
program.
• Program management
The business and administrative planning, organizing, directing, coordi-
nating, controlling, and approval actions designated to accomplish over-
all program objectives not associated with specific hardware elements
and not included in systems engineering.
• Training

125
126 Software Cost Estimation Metrics Manual for Defense Systems

Deliverable training services, devices, accessories, aids, equipment, and

parts used to facilitate instruction in which personnel will learn to op-
erate and maintain the system with maximum efficiency
• Data
The deliverable data that must be on a contract data requirements list,
including technical publications, engineering data, support data, and
management data needed to configure management, cost, schedule, con-
tractual data management, and program management.
• System test and evaluation
The use of prototype, production, or specifically fabricated hardware
and software to obtain or validate engineering data on the performance
of the system in develop (in DoD, normally funded from research, de-
velopment, test, and evaluation appropriations); also includes all effort
associated with design and production of models, specimens, fixtures,
and instrumentation in support of the system-level test program.
• Peculiar support equipment
Equipment uniquely needed to support the program: vehicles, equip-
ment, tools, and the like to fuel, service, transport, hoist, repair, over-
haul, assemble and disassemble, test, inspect, or otherwise maintain mis-
sion equipment, as well as equipment or software required to maintain
or modify the software portions of the system.
• Common support equipment
Operational and site activation; facilities equipment not unique to the
program and available in inventory for use by many programs; and in-
stallation of mission and support equipment in the operations or support
facilities and complete system checkout or shakedown to ensure opera-
tional status.
• Initial spares and repair parts
Includes the deliverable spare components, assemblies, and subassem-
blies used for initial replacement purposes in the materiel system equip-
ment end item.

7.1.1 Four Estimation Stages

There are four major stages of estimation as shown in Figure 7.1: Initiation
and Research, Assessment, Analysis, and Documentation and Presentation.
Each stage is described briefly.
Initiation and Research Assessment Analysis Presentation

Determine Identify
Define the the Esti- Ground
Program mating Rules and Update the
Develop Conduct a Present
Define the Structure Assumptions Estimate to
the Conduct Risk and Document Estimate to
Estimate’s Reflect
Estimating Sensitivity Uncertainty the Estimate Management
Purpose Develop the Point Actual
Plan Analysis for Approval
Obtain Estimate and Compare it costs/changes
the Data to an Independent Cost
Estimate
Estimation Process

FIGURE 7.1: GAO’s Cost Estimation Process

127
128 Software Cost Estimation Metrics Manual for Defense Systems

7.1.1.1 Initiation and Research

This describes to the recipient of the estimate what is being estimating,
and why it is being estimated. There are two steps in this stage:

1. Define the estimate purpose. The purpose of the estimate defines the
scope of the estimate, i.e., what is in and what is out of scope. More on
this is in Section 7.2.

2. Develop the estimating plan. This step defines the estimation team, the
schedule/timeline of estimation activities, the estimation methodology
and the alternative methodology to be used. This step is organization
specific and is not discussed here.

7.1.1.2 Assessment
The steps in cost assessment are iterative and can be accomplished in
varying order or concurrently. These steps are:

1. Define the program (software) that is to estimated. The description of

the software program under estimation is important for correctly clas-
sifying the software configuration items by Operating Environment and
Application Type, discussed earlier in the manual. More on this is in
Section 7.3.

2. Determine the structure of the estimate. The structure addresses how

the different software configuration items are related, e.g., sequential,
overlapped, and how their individual estimates will be aggregrated into
an overall estimate. More on this is in Section 7.4.2.

3. Identify the ground rules and assumptions. Every estimate is based on

some rationale, e.g., analogy, data, cost estimating relationships (as pre-
sented in this manual), and, because the estimate is for future develop-
ment, assumptions about what is going to happen. More on this is in
Section 7.4.1.

4. Obtain the data. Estimates should be based data either collected imme-
diately from past project data (if the estimate is an update to an ongoing
project) or from historical data from similar projects. The SRDR data
discussed in Chapter 2 of this manual is an example of the latter. More
on this is in Section 7.5.

5. Develop the point estimate and compare it to an independent estimate.

The cost estimating relationships described in this manual produce point
estimates. A second estimation method should be used to compare to
the point estimate. The sources of the second estimate can be analogous
projects or commercial software cost estimate models. More on this is
in Section 7.6.
 Estimation Process 129

This is an important stage where the relevant cost modeling method(s) are
chosen before the estimates are calculated. Simple or detailed CERs such as
those provided in this manual may apply here, depending on the program
context and acquisition phase.

7.1.1.3 Analysis

Analysis of the estimate expresses the confidence in the point or range of

the estimate. There are two steps in this phase:

1. Conduct sensitivity analysis. Once the point estimate is developed, deci-

sion makers need to understand how sensitive the total cost estimate is
to changes in the data input. Sensitivity analyses should be performed
to identify the major cost drivers for the estimate. It determines how
the different input ranges affect the point estimates. Cost drivers are
varied to see which ones impact the greatest changes in cost. More on
this is in Section 7.7.

2. Conduct risk and uncertainty analysis. This is a critical review of the

estimate considering the estimation methodology, e.g. cost estimating
relationships and assumptions. More on this is in Section 7.8.

7.1.1.4 Documentation and Presentation

Documentation and presentation are the final products of an estimate.

This stage has three steps:

1. Document the estimate. This is an important step because it serves

to record how the estimate was constructed. This will enable others
to review the estimate. It will also be an important input to future
estimates. More on this is in Section 7.9.

2. Present the estimate to management for approval. This step is a manage-

ment presentation of the life cycle cost estimate including a description
of the baseline, estimation assumptions, comparison of the estimate to
the allocated budget and schedule, discussion of the largest cost elements
and cost drivers, all presented in an clear logical manner. This step is
organization specific and is not discussed here.

3. Update the estimate to reflect actual costs/changes. Perhaps the biggest

mistake is to create an estimate once and never update it with changes
in the project. The estimate should be a continuously evolving chronicle
of forecasted project costs. The last step above begins with the first step,
the purpose of the estimate.
130 Software Cost Estimation Metrics Manual for Defense Systems

7.2 Estimation Purpose

This is not a large step but it is important because it sets the context of
what the estimate is being done. The purpose of the software cost estimate
reflects its intended use and drives the amount of resources required to create
the estimate. There are many purposes for a software estimate including:

• Evaluation of software affordability (the funds available)

• Analysis of alternatives
• Estimation of the funding required to efficiently execute a program.
• Provide an independent review of software costs
• Assess the cost impact of software baseline changes
The purpose of an estimate is also influenced by two other factors:

1. The software projects acquisition lifecycle stage. See Figure 1.2 on the
acquisition lifecycle. Early lifecycle stage estimates will rely mostly on
analogous projects. A detailed estimate will be difficult to create with
early lifecycle stage projects. Later lifecycle stage estimates will have
historical data on which to base the estimate providing more insight
into development activity cost and durations.
2. The timeframe covered by the estimate. Different timeframes include:

• Full lifecycle cost of the software (development and maintenance)

• Total development cost (requirements analysis to qualification test-
ing for all deliveries)
• Incremental cost (the cost of the next increment to be delivered)
• Annual cost

7.3 Program Definition

This is an important step because it is information the estimator will use
in creating the estimate. The goal is to define the program in enough detail to
understand what software configuration items are going to be developed and
in which categories of operating environments and productivity types they
belong. As with the purpose of the estimate section above, the description of
the program is an important documentation piece explaining what software is
to be estimated.
Information on the following should be collected (depending on the soft-
wares acquisition lifecycle phase, some of this information may not be avail-
able):
 Estimation Process 131

• Name of the program

• Programs purpose, its performance characteristics and all system con-
figurations. This is an indication of complexity.
• The programs acquisition strategy and schedule including the deploy-
ment and maintenance plan if that will be required in the estimate.
• Technical details such as:
– Key functional requirements and performance characteristics for
the software
– The number of platforms the software runs on
– Who will develop, operate, and maintain the system
– Descriptions of software components (including interactions, tech-
nical maturity of critical components, and standards). A high-level
system architecture diagram is very helpful in identifying compo-
nents and interfaces.
– The programs relationship to other existing systems, including pre-
decessor or similar legacy systems;
– Software architecture and the different required configurations
• How the software will interface to other systems
• Information assurance requirements
• Requirements for reliability, security and safety
• Integration test and evaluation concepts and plans.

The technical details also need to provide the estimator with enough in-
formation to understand the software application boundaries and interfaces.
Application boundaries are one part of defining the estimation scope. Known
risks to software development should be collected, e.g.,

• Hard real-time execution requirements

• COTS product incorporation
• Number of interfaces these CSCIs depend upon
• Technical readiness of algorithms
• Security requirements complexity
• Data volume throughput
• Possible external stakeholders contention

7.4 Estimation Scope

It is very important to make explicit what is being estimated. The estima-
tion purpose and program definition drive the estimation scope. The ground
rules and assumptions bound what is being estimated and capture uncertainty.
The structure of the estimate shows how the estimate will be composed.
At the conclusion of this step, it is highly recommended that the recipients
132 Software Cost Estimation Metrics Manual for Defense Systems

of the estimate be presented with the estimation scope. This will serve to
validate and verify the software estimate.

7.4.1 Ground Rules and Assumptions

The ground rules and assumptions clearly state up front the conditions of
the software estimate. Multiple items should be addressed:

• Estimation boundaries, i.e., what the estimate includes and excludes

• Global and program-specific assumptions (these may be expanded later
after reconstructing historical development performance)
• Program acquisition lifecycle phase
• Planned build or release schedule
• Schedule or budget constraints
• List of software configuration items in each build or release
• Legacy software that will be used in the development (if applicable)
• Software the government is to furnish as well as the use of existing unique
software development facilities (if applicable)
• COTS software that will be used in the development (if applicable)
• Prime contractors and major subcontractors
• Technology refresh strategy, i.e., what and how often refresh will occur
• New technology to be developed, e.g. new algorithms.

Some of this information may only be available from the Program Office. If
this is the case, during the data collection step discussed in Section 7.5, the
missing information should be collected.

7.4.2 Estimation Structure

Based on the ground rules and assumptions, an estimating structure is
developed. Figure 7.2 shows an example decomposition of an aircraft system
down to the software elements. The estimating structure resembles these el-
ements and will include all software elements in the aircraft. The structure
serves to decompose the estimate into its individual components. It clearly
shows how estimates are rolled-up into an aggregate estimates.
Using the WBS in Figure 7.2, the Radar Software would be the top level
of the estimation structure in Table 7.1. Radar is decomposed into multiple
builds and each build is decomposed into its constituent configuration items.
The CSCIs may include COTS software, GOTS software, adapted software
(from internal or external sources), auto-generated software and sofware ser-
vices in addition to custom-built software.
Each configuration item should have a function description and an appli-
cation type. The operating environment should be confirmed as there may be
mix of environments, e.g. an aircraft could have ground-based software.
 Estimation Process 133

Receiver

Transmitter
Comm/ID CSCI 1...n
Radar Antennae Build 1
Guidance &
Navigation CSCI-CSCI Int
Platform Radar
Mission Int. Software
Airframe CSCI 1...n
Computer Int, Test &
SEPM Build 2
Display & Checkout CSCI-CSCI Int
Avionics
Control
Sys T&E
SEPM Fire Control
. Aerial Training
Vehicle
SYS T&E Survivability
Data
Automatic
Data
Fight Ctl Support
Equipment
Training
Initial
Repair Parts

FIGURE 7.2: High-Level Aircraft WBS Decomposed to Software Elements

TABLE 7.1: Radar Software WBS Decomposition

Radar Software
• Build-1
– CSCI-1 Control Unit (activity estimation for Requirements,
Architecture, Design, Implementation and CSCI testing)
– CSCI-2 Item Processing
– CSCI-3 Track Database
– CSCI-CSCI Integration
– Hardware Installation and Checkout
– Acceptance Testing
• Build-2
– CSCI-1 Control Unit
– CSCI-2 Item Processing
– CSCI-4 Display Manager
– CSCI-5 Built-In-Test
– CSCI-CSCI Integration
– Hardware Installation and Checkout
– Acceptance Testing
134 Software Cost Estimation Metrics Manual for Defense Systems

7.5 Data Collection and Normalization

In this step, create a data collection plan and collect historical data with
actual results. Much of the data will come from the Program Office. An inter-
view with the Program Office should be part of this plan. The data collection
plan would include:

• All of the data not found during the Program Definition step
• Software size from previous builds or releases.
• Staffing levels and/or effort for each build or release
• Software build or release durations with start/end dates
• Document the start/end criteria for each build or release
• Cost model parameters, such as those in Appendix A, if they are
going to be used in creating the estimate
• Other cost model calibration data if required
• Average labor rate ($) to convert estimated effort to dollars
• Number of labor hours in a year
• Software risks, e.g., requirements stability, algorithm maturity.

There are at least two sources of historical data.

1. Historical data from this program. This is the best source of data. The
size, effort and duration data show actual development performance.
2. Similar historical SRDR data for each CSCI’s application type. This
manual discusses SRDR data in Chapter 2 and what is available.

Document all pertinent information, including an assessment of data reli-

ability and accuracy, discussed in Chapter 4. This will be an input to docu-
menting the estimate.
After the data is collected it must be normalized, i.e., converted to the
same measurement units and cover the same activities. Chapter 4 discusses
data normalization.

• Use the size definition checklists in Tables 3.2 and 3.3 to collect size data
and normalize it to a logical count.
• Use the definition of labor activities in Table 3.9 and the guidelines in
section 3.4 to document the data collected. SRDR activities and labor
categories are discussed in Section 4.5.2.
• Match duration with effort activities above, e.g. requirements, architect-
ing, detailed design, implementation, etc.

Analyze the data for anomalies and trends. Compare results against the
application type benchmarks in section 5.5.1 provided in this manual. Finally,
store the collected data for model calibration and for future estimates.
 Estimation Process 135

7.6 Estimate Creation

This step starts with either developing productivity benchmarks and cost
estimating relationships as discussed in Chapter 5, using the benchmarks and
CERs contained in this manual, or calibrating existing cost estimation models.
Commercial cost models in Appendix A have their own calibration require-
ments and are not discussed here. The use of productivity benchmarks, cost
estimating relationships or commercial cost models are referred to here as
estimating methodologies.

7.6.1 Historical Reconstruction

As a final step in calibration, perform a historical reconstruction of past
software builds or releases for both effort and duration. This important activity
informs the estimator about the suitability of the estimation methodology to
predict past software development activities. A large difference between the
estimation methodology output and past performance may require further
calibration.
Historical reconstruction should be reviewed with the Program Office for
feedback and insights. Questions and assumptions required to recreate the past
development effort and duration should be discussed. These discussions may
lead to another round of calibration of the estimation methodology. Feedback
from the Program Office should be documented.
Another benefit of historical reconstruction and Program Office feedback
is the identification of addition assumptions not apparent from earlier analysis
of the program. All or some of these assumptions may be present in creating
the future estimate. These need to be documented.

7.6.2 Estimate Construction

To develop a software estimate for future development, select an estimation
methodology suitable for each CSCI in the estimation structure discussed
earlier. This usually requires a mixture of methodologies similar to those used
in reconstruction.
The estimated effort should be distributed over the development activity
durations for each configuration item in the estimation structure, see Sec-
tion 5.4.5. Then the durations for each configuration item are overlaid by
their planned start/end dates. Then the builds or releases are overlaid by
their planned start/end dates. This can be accomplished using a spreadsheet.
Commercial models do this task automatically.
The overlaid effort of the configuration items and the builds or releases
creates an overall effort profile across the software development (similar to
the cover page of this manual). A cost estimate can now be created from this.
136 Software Cost Estimation Metrics Manual for Defense Systems

7.6.3 Estimate Cost Analysis

Convert the effort estimate for each month in the effort profile into dollars
using the average labor rate from the data collection step. Be sure to express
costs in base-year dollars. A base-year is a selected year to which all costs are
deflated or inflated (using an approved index) to normalize the dollar values
so comparisons can be made across years. Past projects and historical costs
would be inflated. Future costs would be deflated if they were adjusted for
inflation in another estimate.
Compare the effort, adjusted base-year cost and duration estimate against
analogous programs. Examine where and why there are differences. Look for
errors like double counting and omitted costs.
After comparisons have been made, the estimate should be adjusted for
then-year dollars. Then-year dollars are the costs for future years adjusted
for inflation. The inflation rates are obtained from an approved index such as
from the Office of Management and Budget.
Use a cost-estimating checklist, such as the the Software Engineering In-
stitute’s A Manager’s Checklist for Validating Software Cost and Schedule
Estimates in [48], to review the estimate. At a high level, there are seven
parts to this checklist:
1. Are the objectives of the estimate clear and correct?
2. Has the task been appropriately sized?
3. Is the estimated cost and schedule consistent with demonstrated accom-
plishments on other projects?
4. Have the factors that affect the estimate been identified and explained?
5. Have steps been taken to ensure the integrity of the estimating process?
6. Is the organization’s historical evidence capable of supporting a reliable
estimate?
7. Has the situation changed since the estimate was prepared?
Each of these parts are discuss in more detail in the SEI report.

7.6.4 Alternate Estimate

It is considered a best practice to create a second independent estimate
using a different estimation methodology. The value in creating a second inde-
pendent estimate comes from comparing the two results. Estimation method-
ology inputs and assumptions will be highlighted and errors/omissions will be
discovered. This exercise will improve the overall quality of the estimate. The
goal should be to get both estimates to within 10% of each other. As an exam-
ple of comparing different methodologies, Appendix A contains a comparison
of several cost model parameters and assumptions.
Software cost estimation should not be a one-and-done activity. The esti-
mation methodologies should be updated as more data becomes available or if
conditions change on the program. The applicable estimate steps should then
be repeated.
 Estimation Process 137

7.7 Sensitivity Analysis

A simple approach to associating cost to risk is to create different devel-
opment scenarios based on the highest probable set of risks occurring. This
consists of changing the estimated software size (based on a growth factor
and/or uncertainty) and other model inputs. Identify the effects to the overall
estimate. Determine which risks have the highest impact on effort and dura-
tion and which estimation structure elements are affected most by changes.

7.8 Risk and Uncertainty Analysis

A software cost estimate is uncertain. Risk analysis captures the uncer-
tainty about the point estimate (discussed in the previous step) and the an-
ticipated growth in cost and duration. Risk is expressed as a distribution
around the estimate.
There are several approaches to quantifying risk and creating a distribution
around a cost estimate. Two simple approaches are discussed here. Additional
approaches and more detailed discussion on the topic can be found in the GAO
Guidebook [18], Air Force Cost Risk and Uncertainty Analysis Handbook [2],
and NASAs Probabilistic Risk Assessment Procedures Guide [52].
One way to determine whether a program is realistically budgeted is to per-
form an uncertainty analysis, so that the probability associated with achieving
its budget point estimate can be determined. A cumulative probability distri-
bution (CDF), also known as an S-curve, is particularly useful in portraying
the uncertainty implications of cost estimates. It shows the probability of
occurrence, or the percent confidence of achieving particular costs.
The CDF is usually derived from a simulation technique such as Monte
Carlo analysis using random samples from input probability distributions.
Monte Carlo simulation produces a CDF per Figure 7.3. This example shows
a cost estimate mapped to probability levels. The point estimate created in
the previous step can be found on the distribution curve with its probability
of occurrence in Figure 7.3.
For instance, if a point estimate was 235 Person-Months of effort, Figure
7.3 shows a 25% probability of occurrence. A 300 Person-Month estimate has
50% probability, and there is a 70% probability of achieving a cost of less than
or equal to 350 Person-Months.
The basic tasks for carrying out a risk analysis are:
1. Determine program risks. This is a subjective task based on interviews
with technical experts familiar with the program. Possible categories of
risk for software are
138 Software Cost Estimation Metrics Manual for Defense Systems

100%
90%
80%
70%
Probability
60%
50%
40%
30%
20%
10%
0% . .
0 100 200 300 400 500
Person-Months

FIGURE 7.3: Cost Cumulative Probability Distribution

• Stakeholder contention over software functionality, funding and

schedule
• Requirements growth and volatility
• Subcontractor performance
• Legacy software function shortfall
• Technology immaturity
• Unplanned rework
• Inadequate staffing or the wrong skill set
• Not enough allocated schedule, i.e., the estimated build duration is
longer than the allocated schedule.
• Inefficient production processes.

Identified risks should be given a scored as to their impact on cost, e.g.

1 = low impact, 5 = medium impact, and 10 = high impact.

2. Select the probability distributions to model uncertainty in the cost

inputs. Some simple, useful common probability distributions are de-
scribed next.

7.8.1 Common Probability Distributions

Some common probability distributions used in cost estimation are

shown in Figure 7.4. These include the uniform, triangular, normal and
lognormal distributions to model uncertainty. Each of these is described
 Estimation Process 139

next in terms of their characteristics and applications.

Uniform
A uniform distribution represents a range of equally likely outcomes with
an equal number of measures in each interval. As shown in Figure 7.4,
all values in between the range endpoints of a and b are equally likely.
It can be used as an initial model for a quantity that is felt to be ran-
domly varying between a and b but about which little else is known. It
can be used:

• in the absence of distribution data

• to represent equally likely outcomes (e.g. decision choices)
• for many software cost drivers.

The uniform is often used with engineering data or analogy estimates

[18].

f (x) f (x)

1 2
(b−a) (b−a)

. .
a b x a c b x
Uniform Triangle
f (x) f (x)

. ..
µ x x
Normal Lognormal

FIGURE 7.4: Common Probability Distributions

Triangular
The triangular distribution is used when the minimum, maximum and
140 Software Cost Estimation Metrics Manual for Defense Systems

most likely values of a random variable are known but no other infor-
mation is available. It is characterized by three points, can be skewed or
symmetric and is easy to understand because it is intuitive. A triangular
distribution is often used as a rough model in the absence of data. It ac-
counts for a tapered distribution with the values being centered around
the c location parameter, the most likely value. The limiting cases where
c = b and c = a are called right triangular and left triangular distribu-
tions respectively.
The triangular distribution is commonly used to express technical uncer-
tainty in software cost estimation because of its use of minimum, most
likely and maximum inputs. It is often used as an initial rough model
when the minimum, most likely and maximum are known but no other
distribution shape data is known. Software cost drivers, particularly size,
are frequently represented with triangular distributions.

Normal
A normal distribution is symmetric around the mean and bell-shaped
(also called a bell curve) in which most measures occur in the closest
to the center of the distribution and fewer in the intervals farther from
the center. Figure 7.4 shows a normal distribution centered around it’s
mean µ.
The symmetric normal distribution is used for outcomes likely to occur
on either side of the average value. For software these may include
• personnel factors representing a normal spread of capability
• size, effort, complexity, other project characteristics
Besides technical uncertainty it is used to assess the uncertainty of cost
estimating relationships, as presented in Chapter 5.

Lognormal
The lognormal is a continuous distribution positively skewed with a
limitless upper bound and known lower bound. It is skewed to the right
to reflect the limitless upper bound.
In the skewed distribution the measurements cluster around a certain
value, but that value is not in the center of the distribution. In the log-
normal distribution ln(x) is normally distributed. The distribution mean
is the mean of ln(x), and the standard deviation is the standard devi-
ation of ln(x). Figure 7.4 shows a sample lognormal distribution. The
dispersion around the mean is characterized by the standard deviation.
The lognormal distribution is often a good one to model the overall size
or complexity of a software project, as one tends to underestimate the
maximum possible values. It can be used:
• for uncertainty in system size, effort, complexity
 Estimation Process 141

• accounting for typical underestimation of scope and system-wide

effects

One drawback of a lognormal function is its extremely long tail. For this
a truncated lognormal function bounds the distribution to more realistic
end values. The truncated lognormal prevents non-positive values or
other inconsistent values from a standard lognormal (e.g. when the mean
is close enough to zero that the spread goes negative).
The lognormal function can take on shapes similar to the gamma func-
tion or the Weibull distribution (the Rayleigh curve is actually a special
case of the Weibull distribution). The lognormal is also used characterize
uncertainty in nonlinear cost estimating relationships.
See [18] for additional probability distributions, their descriptions and
typical applications.

3. Continue the risk analysis by acccounting for correlation between cost

elements. This identifies correlations between different elements in the
estimation structure, e.g. when one element increases in cost the corre-
lated element also increases in cost. Care must be taken when sampling
that when one element has a high cost value the correlated element does
not have a low cost value producing inconsistency.

4. Perform uncertainty analysis with a Monte Carlo simulation. CER pa-

rameters are modeled as distributions according to Step 2 resulting in a
cost cumulative distribution function per Figure 7.3. This technique is
illustrated for a CER in Section 7.8.2.

5. Identify the probability associated with the point estimate. The output
from the Monte Carlo simulation will produce a cost cumulative prob-
ability distribution, see Figure 7.3. Locate the point estimate on the
cumulative curve.

6. Recommend sufficient contingency reserves. Identify the amount of con-

tingency funding and add this to the point estimate to determine the
risk-adjusted cost estimate.

7. Allocate risk-adjusted cost across development builds or releases.

7.8.2 Monte Carlo Analysis with a CER

An illustration of the Monte Carlo simulation technique will be applied to
the RTE CER, equation 5.4 derived in Chapter 5:

P M = 13.16 · KESLOC 0.84 (7.1)

We choose a normal distribution of size represented in Figure 7.5 with a
mean value of 100 and standard deviation of 20 EKSLOC.
142 Software Cost Estimation Metrics Manual for Defense Systems

40%

30%

Probability
20%

10%

0% .
0 25 50 75 100 125 150 175 200
KESLOC

FIGURE 7.5: Size Input Normal Probability Distribution

The RTE CER is computed many times at each random value of KESLOC
drawn from the normal distribution in Figure 7.5. The Monte Carlo simulation
results are based on 1000 runs. The probability distribution function shows
the simulated effort intervals in Figure 7.6, and the frequencies are summed
for increasing size in the cumulative distribution function in Figure 7.7.
The smoothed line in Figure 7.7 is the continuous CDF corresponding to
earlier Figure 7.3 where the probability of achieving a given estimate can be
determined. For example, a safe 80% confidence level estimate corresponds to
about 700 Person-Months.

200

150
Frequency

100

50

0. .
200 300 400 500 600 700 800 900 1,000
Person-Months

FIGURE 7.6: RTE Effort Probability Distribution Function

 Estimation Process 143

100%
90%
80%
Probability 70%
60%
50%
40%
30%
20%
10%
0% .
200 300 400 500 600 700 800 900 1,000
Person-Months

FIGURE 7.7: RTE Effort Cumulative Distribution Function

This Monte Carlo simulation example covered the technical uncertainty

associated with the size. A more thorough simulation would also account for
the CER error distribution as shown in Figure 5.9. Both sources of uncertainty
would be reflected in the simulation output.

7.9 Estimate Documentation and Packaging

The documentation step provides a record of the software estimate. There
should be enough detail that a cost analyst unfamiliar with the program can
recreate the estimate and produce the same result. This will require the doc-
umentation to include background information as well as information on the
estimation methodology and results. Background documentation should in-
clude:
• The purpose of the estimate, the team that prepared it, and who ap-
proved the estimate and on what date
• Description of the program, performance characteristics and system con-
figurations
• Program acquisition strategy including deployment and maintenance
• Program funding profile and schedule
• Program technical details such as key functional requirements, number
of platforms, development contractor, and a high-level architecture dia-
gram
• Programs relationship to other existing systems
144 Software Cost Estimation Metrics Manual for Defense Systems

• Software architecture for different configurations.

Estimation methodology documentation should include:

• Ground rules and assumptions

• Estimation structure with a description and classification of each cost
element
• Data sources for each cost element
• How the data were normalized
• Past program technical data: software size, effort, build or release dura-
tions with an assessment of the data quality and reliability
• Program office feedback and insights on the historical reconstruction of
previous builds or releases.
Estimation results documentation should include:

• Estimating methodology and rationale used to derive each estimation

structure element’s cost
• Programs time-phased life-cycle cost containing the effort profile and
duration across all builds or releases.
• Results of the risk, uncertainty, and sensitivity analyses
• Comparison of how the estimate compares to the funding profile
• Differences in how this estimate compares to any previous estimates.

The software estimation process described in this chapter is based on GAO

guidance [18]. Where applicable, references to other sections of this manual are
used to provide examples of employing metrics data to make cost estimates.
A software cost estimate is only part of the system level estimate. This
chapter focused on estimating the cost of software activities associated with
development, i.e., requirement analysis through acceptance activities. As
pointed out there are may be other activities requiring a cost estimate, e.g.,
training, data, support for system test and evaluation. The estimation struc-
ture can be expanded to handle these other sources of cost.
Appendix A
Cost Model Descriptions

A.1 Introduction
This appendix summarizes and compares the parametric software cost
models most frequently used in the DoD. Transformations between the re-
spective models are provided so projects can be represented in all models in
a consistent manner and to help understand why estimates may vary. These
descriptions are subject to change, and the reader should check for updates
and more complete details as necessary.
The cost models used in sea, space, ground, and air platforms by the ser-
vices are generally based on the common effort formula shown in Equation
A.1. Size of the software is provided in a number of available units, cost fac-
tors describe the overall environment and calibrations may take the form of
coefficients adjusted for actual data or other types of factors that account for
domain-specific attributes [31], [32]. The total effort is calculated and then de-
composed by phases or activities according to different schemes in the models.

Ef f ort = A · SizeB · EM (A.1)

where
• Effort is in Person Months
• A is a constant derived from historical project data
• Size is software size in source lines of code or other size measures
• B is a scale factor
• EM is an effort multiplier from cost factors.
The models can use size expressed as lines of code, function points, object-
oriented metrics and other measures. Each model has its own respective cost
factors for the linear effort multiplier term, and each model specifies the B scale
factor in slightly different ways (either directly or through other factors). Some
models use project type or application domain to improve estimating accuracy.
Others use alternative mathematical formulas to compute their estimates.
Sizing for the models is summarized in Table A.1. A comparative analysis of
cost model factors is listed in Table A.2. The model WBS phases and activities
are in Tables A.3 and A.4 respectively.

145
146 Software Cost Estimation Metrics Manual for Defense Systems

A.2 Cost Models

The models covered include COCOMO II, SEER-SEM, SLIM, and True
Planning. They were selected because they are the most frequently used mod-
els for estimating DoD software effort, cost and schedule. A comparison of
the COCOMO II, SEER-SEM and True Planning models for NASA projects
is described in [32]. A previous study analyzed the same three models with
respect to flight and ground projects [31], and another comparative survey of
software cost models can also be found in [33].
COCOMO II is a public domain model and is implemented in several com-
mercial tools. True Planning and SEER-SEM are both proprietary commercial
tools with unique features but also share some aspects with COCOMO. SLIM
is another parametric tool that uses a different approach to effort and schedule
estimation.
Any of the models can be used effectively if properly calibrated. Each
model has strengths and weaknesses, thus the studies recommend using at
least two models to estimate costs whenever possible to provide added assur-
ance that you are within an acceptable range of variation.

A.2.1 COCOMO II
The COCOMO (COnstructive COst MOdel) cost and schedule estimation
model was originally published in [5] and the model continues to be updated at
USC. The COCOMO II model defined in [8] has three submodels: Applications
Composition, Early Design and Post-Architecture. They can be combined in
various ways to deal with different software environments. The Application
Composition model is used to estimate effort and schedule on projects typi-
cally done as rapid application development. The Early Design model involves
the exploration of alternative system architectures and concepts of operation.
Typically, not enough is known to make a detailed fine-grained estimate. This
model is based on function points (or lines of code when available) and a set
of five scale factors and seven effort multipliers.
The most frequently used is the Post-Architecture model for when top level
design is complete and detailed information about the project is available and
the software architecture is well defined. It uses Source Lines of Code and\or
Function Points for the sizing parameter, adjusted for reuse and breakage; a
set of 17 effort multipliers and a set of five scale factors that determine the
economies\diseconomies of scale of the software under development. The effort
formula is:

∏
N
Ef f ort = A · SizeB · EMi (A.2)
i=1
 Appendix A - Cost Model Descriptions 147

Where

• Effort is Person Months

• A is a constant derived from historical project data
• Size is in KSLOC (thousand source lines of code), or converted from
other size measures
• B is an exponent for the diseconomy of scale dependent on additive scale
drivers
• EMi is an effort multiplier for the ith cost driver. The geometric product
of N multipliers is an overall effort adjustment factor to the nominal
effort.

The COCOMO II effort is decomposed by lifecycle phase and activity as

detailed in Section A.4.

A.2.2 True Planning

True Planning is offered by PRICE Systems, and initially was called
PRICE S. It fits into a composite modeling system and can be used to es-
timate more than just software costs. Many of the models central algorithms
were published in [45]. For more details on the model and the modeling system
see the PRICE Systems website at http://www.pricesystems.com.
The True Planning model consists of three submodels that enable esti-
mating costs and schedules for the development and support of computer
systems. The model covers business systems, communications, command and
control, avionics, and space systems. It includes features for reengineering,
code generation, spiral development, rapid development, rapid prototyping,
object-oriented development, and software productivity measurement. Size in-
puts include SLOC, function points and/or Predictive Object Points (POPs).
The True Planning system also provides a COCOMO II capability. Some of
the descriptions herein may retain the old PRICE S terminology for the cost
factors, such as the Rosetta Stone mapping between models.

A.2.3 SEER-SEM
SEER-SEM is a product offered by Galorath, Inc. This model is based on
the original Jensen model [26], which derives from COCOMO and other mod-
els in its mathematical formulation. However, its parametric modeling equa-
tions are proprietary. SEER-SEM estimates can be used as part of a composite
modeling system for hardware/software systems. Descriptive material about
the model can be found in [17].
The scope of the model covers all phases of the project life-cycle, from
early specification through design, development, delivery and maintenance. It
handles a variety of environmental and application configurations, and mod-
els different development methods and languages. Development modes covered
148 Software Cost Estimation Metrics Manual for Defense Systems

include object oriented, reuse, COTS, spiral, waterfall, prototype and incre-
mental development. Languages covered are 3rd and 4th generation languages
(C++, FORTRAN, COBOL, Ada, etc.), as well as application generators.
The SEER-SEM cost model allows probability levels of estimates, con-
straints on staffing, effort or schedule, and it builds estimates upon a knowl-
edge base of existing projects. Estimate outputs include effort, cost, sched-
ule, staffing, and defects. Sensitivity analysis is also provided as is a risk
analysis capability. Many sizing methods are available including lines of code
and function points. For more information, see the Galorath Inc. website at
http://www.galorath.com.

A.2.4 SLIM
The Software LIfecycle Management ( SLIM) model is offered by Quanti-
tative Software Management. It was developed by Putnam based on a Nor-
den/Rayleigh manpower distribution and his finding in analyzing many com-
pleted projects [46]. The central part of Putnam’s model called the software
equation is:

1 4
S = Ck · Ef f ort 3 · td3 (A.3)

Where

• S is Size of the software

• Effort is Person Months
• td is the software delivery time
• Ck is a productivity environment factor

The productivity environment factor reflects the development capability de-

rived from historical data using the software equation.
Another relation found by Putnam is

Ef f ort = D0 · t3d (A.4)

where D0 is the manpower build-up parameter which ranges from 8 (entirely

new software with many interfaces) to 27 (rebuilt software). The SLIM soft-
ware tool is based on this model for cost estimation and manpower scheduling
[34].
 Appendix A - Cost Model Descriptions 149

A.3 Cost Model Input Factors

A.3.1 Software Size
A.3.1.1 Overview and Sizing Units
This section describes software sizing inputs in the models. The primary
unit of software size is Thousands of Source Lines of Code (KSLOC). The
models also support Function Point-based sizing, conversions from from other
size measures to KSLOC, or additional size units can be used directly in the
models. User-defined proxy sizes can also be developed for any of the models.
All models support size inputs for new and adapted software, and some
support automatically translated or generated code. The models differ with
respect to their detailed parameters for the developed categories of software.
The size inputs for the models are described in subsequent sections. The re-
spective software adaptation models for determining equivalent size are also
elaborated. Table A.1 summarizes the size factors and maps them between
the models.

COCOMO II
The COCOMO II size model is based on SLOC or function points con-
verted to SLOC, and can be calibrated and used with other software size units.
Examples include use cases, use case points, object points, physical lines, and
others. Alternative size measures can be converted to lines of code and used
directly in the model or it can be independently calibrated directly to different
measures.
COCOMO II uses Unadjusted Function Points for sizing, and applies its
reuse factors, cost drivers, and scale drivers to this sizing quantity to account
for the effects of reuse, distribution, etc. on project effort. The COCOMO II
procedure for determining Unadjusted Function Points follows the definitions
in [22] in both the Early Design and the Post-Architecture models. The Un-
adjusted Function Points are converted to ESLOC for the effort and schedule
formulas using default or user-supplied conversion ratios [8].

SEER-SEM
In SEER-SEM several sizing units can be used alone or in combination.
It can use SLOC, function points and custom proxies. SEER provides several
variations of function point counts. Its Function-Based Sizing is consistent
with IFPUG counting rules, with the addition of Internal Functions for algo-
rithmic processes. It also has proxy sizing for Mark II Function Points and
Function Points for direct input of a total function point count.
SEER allows custom proxies as a flexible way to estimate software size.
Any countable artifact can be established as a measure. Custom proxies can
be used with other size measures in a project.
150 Software Cost Estimation Metrics Manual for Defense Systems

SEER converts all size data into internal size units, also called effort units,
Users can combine or select a single metric for any project element or for the
entire project.
The SEER function-based sizing uses traditional function point types de-
scribed in Section 3.3.1:
• External Inputs (EI)
• External Outputs (EO)
• Internal Logical Files (ILF)
• External Interface Files (EIF)
• External Inquiries (EQ)
• Internal Functions (IF) any functions that are neither data nor trans-
actions.
Additional pre-defined proxy measures for size include:
• Web Site Development
• Mark II Function Points
• Function Points (for direct IFPUG-standard function points)
• Object-Oriented Sizing.
COTS sizing elements available in SEER are in section A.3.1.6.

True Planning
The True Planning software cost model size measures may be expressed in
different size units including source lines of code (SLOC), function points, pre-
dictive object points (POPs) or Use Case Conversion Points (UCCPs). True
Planning also differentiates executable from non-executable software sizes.
Functional Size describes software size in terms of the functional requirements
that you expect a Software COTS component to satisfy. The True Planning
software cost model size definitions for all of the size units are listed below.

SLIM
SLIM uses effective system size composed of new, modified and reused
code. Deleted code is not considered in the model. If there is reused code than
the Productivity Index (PI ) factor may be adjusted to add in time and effort
for regression testing and integration of the reused code.
SLIM provides different sizing techniques including:
• Sizing by history
• Total system mapping
• Sizing by decomposition
• Sizing by module
• Function point sizing.
Alternative sizes to SLOC such as use cases or requirements can be used in
Total System Mapping. The user defines the method and quantitative mapping
factor.
 Appendix A - Cost Model Descriptions 151

TABLE A.1: Cost Model Size Inputs

COCOMO II SEER-SEM True Planning

New Software
New Size New Size New Size

New Size
Non-executable

Modified Software
Adapted Size Pre-exists Size Adapted Size

Adapted Size
Non-executable

Amount of
Modification %
% Design Modified Redesign Required % % of Design Adapted
% Code Modified Reimplementation % of Code Adapted
Required %
% Integration Retest Required % % of Test Adapted
Required
Assessment and - -
Assimilation
Software - -
Understanding
Programmer - -
Unfamiliarity
- Deleted Size Deleted Size

Code Removal
Complexity

Reused Software
Reused Size Pre-exists Size Reused Size

Reused Size
Non-executable
- Redesign Required % % of Design Adapted
- Reimplementation % of Code Adapted
Required %

(continued )
152 Software Cost Estimation Metrics Manual for Defense Systems

TABLE A.1: Cost Model Size Inputs (continued )

COCOMO II SEER-SEM True Planning

% Integration Retest Required % % of Test Adapted
Required
Assessment and - -
Assimilation
- Deleted Size Deleted Size

Code Removal
Complexity

Generated Software
- - Auto Generated Code
Size

Auto Generated Size

Non-executable

Automatically Translated Software

Adapted SLOC - Auto Translated Code
Size

Auto Translated Size

Non-executable
Automatic Translation - -
Productivity
% of Code - -
Reengineered

A.3.1.2 New Software

New software is interpreted the same in all the cost models. The New Size
parameter is used in COCOMO II to designate new software. New software
in SEER is represented by New Lines of Code or New Functions. In True
Planning, New Code Size is the amount of new code that does not reuse
any design, code, or test artifacts. New Size Non-executable is the percentage
of the New Code Size that is non-executable (such as data statements, type
declarations, and other non-procedural statements). New software is also used
in SLIM to designate software created for the first time.

A.3.1.3 Adapted Software (Modified and Reused)

All the software adaptation models involve estimating AAF as previously
described for modified and reused code. The linear approximation is used to
convert reused software into equivalent lines of code in the models. However,
 Appendix A - Cost Model Descriptions 153

the models have other adjustment factors or parameterizations for AAF as

described next.

COCOMO II
The COCOMO II treatment of software adaptation includes a nonlinear
estimation model with the following parameters [8]:
• Software Understanding (SU) is an effort increment as a percent-
age based on the software structure, application clarity and self-
descriptiveness.
• Unfamiliarity (UNFM) is the programmer’s relative unfamiliarity with
the software which is applied multiplicatively to the Software Under-
standing increment.
• Assessment and Assimiliaton (AA) measures how much relative effort is
needed to determine whether a reused software module is appropriate to
the application, and to integrate its description into the overall product
description. AA is expressed as an effort percentage increment.
The overall Adaptation Adjustment Multiplier (AAM) in COCOMO II uses
the factors for AAF, SU, UNFM and AA as:

AA + AAF (1 + .02 · SU · U N F M )
AAM = , for AAF ≤ 50%
100
AA + AAF + SU · U N F M
AAM = , for AAF > 50%
100
SEER
The software adaption parameters in SEER are applied to the category
Pre-Existing software which is modified to fit into a new system. There are
two categories of pre-existing software:
• Pre-existing, Designed for Reuse
• Pre-existing, Not Designed for Reuse.
Each category is then composed of:

• Pre-existing lines of code which is the number of lines from a previous

system
• Lines to be Deleted are those lines deleted from a previous system.

With these additional inputs SEER will automatically assign lower rework
percentages to software that is designed for reuse. The adaption parameters
applied to the pre-existing software to determine equivalent size are:

• Redesign Required is the percentage of existing code that must be re-

designed to meet new system requirements.
154 Software Cost Estimation Metrics Manual for Defense Systems

• Reimplementation Required is the percentage of existing code that must

be re-implemented, physically recoded, or reentered into the system,
such as code that will be translated into another language.

• Retest Required is the percentage of existing code that must be retested

to ensure that it is functioning properly in the new system.
SEER then uses different proportional weights with these parameters in
their AAF equation according to

P re–existing Ef f ective Size = 0.4 · A + 0.25 · B + 0.35 · C.

Where A, B and C are the respective percentages of code redesign, code

reimplementation, and code retest required. In the standard AAF for-
mula, DM is from Redesign Required, CM is Reimplementation Required
and IM is Retest Required.

True Planning
The True Planning software adapted size parameters are detailed below.

• Adapted Code Size - The amount of existing code that must be changed,
deleted, or adapted for use in the new software project. When the value
is zero (0.00), the value for New Code Size or Reused Code Size must
be greater than zero.

• Adapted Size Non-executable - The percentage of the adapted code size

that is non-executable (such as data statements, type declarations, and
other non-procedural statements). Typical values for fourth generation
languages range from 5% to 30%. When a value cannot be obtained by
any other means, the suggested nominal value for non-executable code
is 15%.

• Amount for Modification - The percent of the component functional-

ity that you plan to modify, if any. The Amount for Modification value
(like Glue Code Size) affects the effort calculated for the Software De-
sign, Code and Unit Test, Perform Software Integration and Test, and
Perform Software Qualification Test activities.

• Percent of Code Adapted - The percentage of the adapted code that must
change to enable the adapted code to function and meet the software
project requirements.

• Percent of Design Adapted - The percentage of the existing (adapted

code) design that must change to enable the adapted code to function
and meet the software project requirements. This value describes the
planned redesign of adapted code. Redesign includes architectural design
changes, detailed design changes, and any necessary reverse engineering.
 Appendix A - Cost Model Descriptions 155

• Percent of Test Adapted - The percentage of the adapted code test ar-
tifacts that must change. Test plans and other artifacts must change to
ensure that software that contains adapted code meets the performance
specifications of the Software Component cost object.
• Reused Code Size - The amount of pre-existing, functional code that
requires no design or implementation changes to function in the new
software project. When the value is zero (0.00), the value must be greater
than zero for New Code Size or Adapted Code Size.
• Reused Size Non-executable - The percentage of the Reused Code Size
that is non-executable (such as, data statements, type declarations, and
other non-procedural statements). Typical values for fourth generation
languages range from 5% to 30%. If a value cannot be obtained by any
other means, the suggested nominal value for non-executable code is
15%.
• Equivalent Source Lines of Code - The ESLOC ( Equivalent Source
Lines of Code) value describes the magnitude of a selected cost object
in equivalent Source Lines of Code size units. True Planning does not use
ESLOC in routine model calculations, but provides an ESLOC value for
any selected cost object. Different organizations use different formulas
to calculate ESLOC. The True Planning calculation for ESLOC is

ESLOC = N ew Code + .70 · Adapted Code + .10 · Reused Code.

• Code Removal Complexity - This value describes the difficulty of delet-

ing code from the adapted code. Two things need to be considered when
deleting code from an application or component: the amount of func-
tionality being removed and how tightly or loosely this functionality is
coupled with the rest of the system. Even if a large amount of function-
ality is being removed, if it accessed through a single point rather than
from many points, the complexity of the integration will be reduced.
• Deleted Code Size - This describes the amount of pre-existing code that
you plan to remove from the adapted code during the software project.
The Deleted Code Size value represents code that is included in Adapted
Code Size, therefore, it must be less than, or equal to, the Adapted Code
Size value.

SLIM
SLIM uses effective system size composed of new, modified and reused
code. Modified software designates changed software and Reused is unchanged.
The calculations for effective size used in SLIM involve the PI factor to adjust
for the relative adaptation effort. If there is reused code than the Productivity
Index (PI ) factor may be adjusted to add in time and effort for regression
testing and integration of the reused code.
156 Software Cost Estimation Metrics Manual for Defense Systems

A.3.1.4 Generated Software

COCOMO II
Generated software is not designated as a separate entity in COCOMO II.
The generator statements, or lines developed as input to the generators, are
counted as new lines of code.

SEER
Generated software is not designated as a separate entity in SEER. The
generator statements are counted as new lines of code. In maintenance, the
generated code may be used instead according the rules in Section Error! Ref-
erence source not found. for maintenance sizing. Function-Based Sizing is also
available in SEER to handle generated software.

True Planning
In True Planning, the parameter Auto Gen Size Non-executable represents
the percentage of the Auto Generated Code Size that is non-executable (such
as, data statements, type declarations, and other non-procedural statements).
Typical values for fourth generation languages range from 5% to 30%. If a
value cannot be obtained by any other means, the suggested nominal value for
non-executable code is 15%. Auto Generated Code Size describes the amount
of code generated by an automated design tool for inclusion in a component.

SLIM
Generated code is not treated as a separate category in SLIM. It could be
improvised through a user-defined size measure and gearing factor.

A.3.1.5 Automatically Translated Software

COCOMO II
The COCOMO II re-engineering and conversion estimation approach in-
volves estimation of an additional factor, AT, the percentage of the code that is
re-engineered by automatic translation. The productivity for automated trans-
lation is designated as another factor called ATPROD. Automated translation
is considered to be a separate activity from development. Thus, its Adapted
SLOC are not included in ESLOC, and its automated translation effort is not
included in estimating the projects schedule in COCOMO II.

SEER
Translated code is not covered in SEER.

True Planning
In True Planning, Auto Trans Size Non-executable is the percentage of the
Auto Translated Code Size that is non-executable (such as, data statements,
type declarations, and other non-procedural statements). Typical values for
fourth generation languages range from 5% to 30%. If a value cannot be ob-
 Appendix A - Cost Model Descriptions 157

tained by any other means, the suggested nominal value for non-executable
code is 15%. Auto Translated Code Size describes the amount of code trans-
lated from one programming language to another by using an automated
translation tool (for inclusion in a component). Auto Translation Tool Effi-
ciency is the percentage of code translation that is actually accomplished by
the tool. More efficient auto translation tools require more time to configure
the tool to translate. Less efficient tools require more time for code and unit
test on code that is not translated.

SLIM
Translated code is not covered in SLIM.

A.3.1.6 Commercial Off-The-Shelf Software (COTS)

The sizing of COTS software sizing is not addressed fully here since COTS
source code is not available to be modified nor measured. However, there are
instances when size proxies are used for effort related to COTS in some mod-
els. COTS can sometimes be handled as reused software or be estimated with
other COTS-specific models. Others have more extensive COTS models built
in such as True Planning.

COCOMO II
COTS is not directly represented in the COCOMO II model. However,
COTS can be treated as reused software or be used in conjunction with the
COCOTS model. See further guidance for COTS estimation in [8].

SEER-SEM
COTS Elements available for sizing in SEER include:
• Quick Size
• Application Type Parameter
• Functionality Required Parameter
• Features
• Number of Features Used
• Unique Functions
• Data Tables Referenced
• Data Tables Configured
True Planning
The COTS estimation model in True Planning uses the following:

• Functional Size - Software size in terms of the functional requirements

that you expect a Software COTS component to satisfy. When you select
Functional Size as the unit of measure (Size Units value) to describe
a Software COTS component, the Functional Size value represents a
conceptual level size that is based on the functional categories of the
158 Software Cost Estimation Metrics Manual for Defense Systems

software (such as Mathematical, Data Processing, or Operating System).

A measure of Functional Size can also be specified using Source Lines of
Code, Function Points, Predictive Object Points or Use Case Conversion
Points if one of these is the Size Unit selected.

• Glue Code Size - The amount of glue code that will be written. Glue
Code holds the system together, provides interfaces between Software
COTS components, interprets return codes, and translates data into
the proper format. Also, Glue Code may be required to compensate
for inadequacies or errors in the COTS component selected to deliver
desired functionality.

To calculate ESLOC for a Software COTS, True Planning first converts

Functional Size and Glue Code Size inputs to SLOC using a default set of con-
version rates. New Code includes Glue Code Size and Functional Size when
the value of Amount for Modification is greater than or equal to 25%. Adapted
Code includes Functional Size when the value of Amount for Modification is
less than 25% and greater than zero. Reused Code includes Functional Size
when the value of Amount for Modification equals zero.

SLIM
COTS is not explicitly modeled in SLIM.

A.3.2 Software Cost Drivers

This section summarizes the mappings, or transformations between cost
factors in different models. With this information estimate inputs can be con-
verted between the models. It also illustrates differences in the models to help
understand why estimates may vary. A top-level Rosetta Stone for common
factors in COCOMO II, SEER-SEM and True Planning factors is shown in
Table A.2.
Most of the mappings between factors are one to one, but some are one to
many (e.g. SEER-SEM has platform factor ratings split into target and host).
In the case of True Planning, many of the COCOMO II factors have direct
corollaries to sub-factors in aggregate True Planning factors. For example the
COCOMO personnel factors are represented as sub-factors under the aggre-
gate True Planning factor for Development Team Complexity. Note there are
additional factors in SEER-SEM and True Planning for which there are no
analogs in COCOMO II.
See [32] for additional details and comparisons of the models. Also refer
to current documentation of the respective cost models for full details and
updates of the cost drivers.
 Appendix A - Cost Model Descriptions 159

TABLE A.2: Cost Model Factors

COCOMO II SEER-SEM True Planning

Scale Drivers
Precedentedness none none
Development none Operating
Flexibility Specification
Architecture/Risk none none
Resolution
Team Cohesion none Development Team
Complexity
Process Maturity none 1 Organization
Productivity
- CMM Level

Product Attributes
Required Software Specification Level - Operating
Reliability Reliability Specification
Data Base Size none Code Size non
Executable
Product Complexity - Complexity (Staffing) Functional Complexity

- Application Class
Complexity
Required Reusability - Reusability Level Design for Reuse
Required

- Software Impacted
by Reuse
Documentation Match none Operating
to Lifecycle Needs Specification

Platform Attributes
Execution Time Time Constraints Project Constraints
Constraint - Communications and
Timing
Main Storage Memory Constraints Project Constraints
Constraint - Memory and
Performance
Platform Volatility - Target System Hardware Platform
Volatility Availability

(continued )
160 Software Cost Estimation Metrics Manual for Defense Systems

TABLE A.2: Cost Model Factors (Continued)

COCOMO II SEER-SEM True Planning

- Host System -
Volatility

Personnel Attributes
Analyst Capability Analyst Capability Development Team
Complexity
- Capability of
Analysts and
Designers
Programmer Programmer Development Team
Capability Capability Complexity
- Capability of
Programmers
Personnel Continuity none Development Team
Complexity
- Team Continuity
Application Application Development Team
Experience Experience Complexity
- Familiarity with
Platform
Platform Experience - Development System Development Team
Experience Complexity
- Familiarity with
- Target System Product
Experience
Language and Toolset Programmer’s Development Team
Experience Language Experience Complexity
- Experience with
Language

Project Attributes
Use of Software Tools Software Tool Use Design Code and Test
Tools
Multi-site Multiple Site Multi Site
Development Development Development
Required Development none Start and End Date
Schedule
 Appendix A - Cost Model Descriptions 161

A.4 Cost Model Lifecycles and Work Breakdown Struc-

tures
The phases covered in the models are summarized in Table 1, activities in
Table 2 and their categories of cost sources in Table 3. Note that these are
the primary defaults and the models provide customization.
COCOMO II allows effort and schedule to be allocated to either a waterfall
or MBASE lifecycle which is an iterative and incremental lifecycle model like
the Rational Unified Process (RUP) or the Incremental Commitment Spiral
Model (ICSM). In True Planning and SEER-SEM the standard lifecycle activ-
ities may be defined differently across development organizations and mapped
to other designations. The phase names, activity descriptions and deliverables
can also be changed in SLIM.
The tools provide utilities to import or define alternative work breakdown
structures and activities that are decomposed by effort and schedule percent-
ages. The SEER-SEM and True Planning tool suites cover other discipline
such as systems and hardware engineering, and COCOMO II has a suite of
other models covering systems engineering and more.
Future work will update the cost model descriptions and expand them to
more comprehensive.
162 Software Cost Estimation Metrics Manual for Defense Systems

TABLE A.3: Model Lifecycle Phases

Model Phases
COCOMO II Inception
Elaboration
Construction
Transition
Maintenance

SEER-SEM System Requirements Design

Software Requirements Analysis
Preliminary Design
Detailed Design
Code / Unit Test
Component Integrate and Test
Program Test
System Integration Through OT&E

True Planning Concept

System Requirements
Software Requirements
Preliminary Design
Detailed Design
Code / Unit Test
Integration and Test
Hardware / Software Integration
Field Test
System Integration and Test
Maintenance

SLIM Concept Definition

Requirements and Design
Construction and Testing
Maintenance
Appendix A - Cost Model Descriptions 163

TABLE A.4: Model Cost Activities

Model Activities
COCOMO II Management
Environment / CM
Requirements
Design
Implementation
Assessment
Deployment

SEER-SEM Management
Software Requirements
Design
Code
Data Programming
Test
CM
QA

True Planning Design

Programming
Data
SEPGM
QA
CFM

SLIM Concept Definition

Requirements and Design
Construct and Test
Perfective Maintenance
164 Software Cost Estimation Metrics Manual for Defense Systems

TABLE A.5: Model Cost Categories

Model Categories
COCOMO II Software Engineering Labor

SEER-SEM Software Engineering Labor

Purchases

True Planning Software Engineering Labor

Purchased Good
Purchased Service
Other Cost

SLIM Software Engineering Labor

Appendix B
MIL-STD-881C WBS Mapping to
Application Types

B.1 Overview
The Work Breakdown Structures were adapted from MIL-STD-881C to
assist in identifying the appropriate Application Type (AT). Each System
from 881C is listed with the associated one of more Metrics Manual Operating
Environments. Within the environments, look through the Subsystems to find
one that matches the component being estimated. Each Subsystem or Sub-
Subsystem has a matching AT.

B.n Environment (WBS Level 1) (this will appear as a section heading)

• SubSystem (WBS Level 2)

– Sub-Subsystem (WBS Level 3)
∗ Domain (WBS Level 4) →AT
Use the AT to lookup the associated Model-based CER or Productivity
Benchmark for that AT.

165
166 Software Cost Estimation Metrics Manual for Defense Systems

B.2 Aerial Vehicle Manned (AVM)

Source: MIL-STD-881C Appendix-A: Aircraft Systems

• Air Vehicle

– Flight Control Subsystem →VC

– Auxiliary Power Subsystem →VC
– Hydraulic Subsystem →VC
– Electrical Subsystem →VC
– Crew Station Subsystem →VC
– Environmental Control Subsystem →VC
– Fuel Subsystem →VC
– Landing Gear →VC
– Rotor Group →VC
– Drive System →VC

• Avionics

– Communication / Identification
∗ Intercoms →RTE
∗ Radio System(S) →RTE
∗ Identification Equipment (IFF) →RTE
∗ Data Links →RTE
– Navigation / Guidance
∗ Radar →SCP or RTE depending on decomposition
∗ Radio →SCP or RTE depending on decomposition
∗ Other Essential Nav Equipment →RTE
∗ Radar Altimeter →SCP
∗ Direction Finding Set →RTE
∗ Doppler Compass →RTE
– Mission Computer / Processing →MP
– Fire Control
∗ Search, Target, Tracking Sensors →SSP
∗ Self-Contained Navigation →RTE
∗ Self-Contained Air Data Systems →RTE
∗ Displays, Scopes, Or Sights →RTE
∗ Bombing Computer →MP
∗ Safety Devices →RTE
– Data Display and Controls
∗ Multi-Function Displays →RTE
 Appendix B - MIL-STD-881C WBS Mapping to Application Types 167

∗ Control Display Units →RTE

∗ Display Processors →MP
∗ On-Board Mission Planning →RTE if it is embedded
– Survivability
∗ Ferret And Search Receivers →RTE
∗ Warning Devices →RTE
∗ Electronic Countermeasures →RTE
∗ Jamming Transmitters →RTE
∗ Chaff →RTE
∗ Infra-Red Jammers →RTE
∗ Terrain-Following Radar →SCP or RTE depending on decom-
position
– Reconnaissance
∗ Photographic Sensors →SCP
∗ Electronic Sensors →SCP
∗ Infrared Sensors →SCP
∗ Search Receivers →RTE
∗ Recorders →RTE
∗ Warning Devices →RTE
∗ Magazines →RTE
∗ Data Link →RTE
– Automatic Flight Control
∗ Flight Control Computers →MP
∗ Signal Processors →SCP
∗ Data Formatting →MP
∗ Interfaces To Other Systems →VC
∗ Pressure Transducers →RTE
∗ Rate Gyros →RTE
∗ Accelerometers →RTE
∗ Motion Sensors →SCP
– Health Monitoring System →SYS
– Stores Management →VP

B.3 Aerial Vehicle Unmanned(AVU) & Ground Site

Fixed (GSF)
Source: MIL-STD-881C Appendix-H: Unmanned Air Vehicle Systems

• Air Vehicle
168 Software Cost Estimation Metrics Manual for Defense Systems

– Vehicle Subsystems
∗ Propulsion →VC
∗ Flight Control Subsystem →VC
∗ Auxiliary Power Subsystem →VC
∗ Hydraulic Subsystem →VC
∗ Electrical Subsystem →VC
∗ Environmental Control →VC
∗ Subsystem Fuel Subsystem →VC
∗ Landing Gear →VC
∗ Rotor Group →VC
∗ Drive System →VC
– Avionics
∗ Communication / Identification →RTE
∗ Navigation / Guidance →RTE
∗ Automatic Flight Control →VC
∗ Health Monitoring System →SYS
∗ Stores Management →VP
∗ Mission Processing →MP
∗ Fire Control →RTE

• Payload

– Survivability Payload →VP

– Reconnaissance Payload →VP
– Electronic Warfare Payload →VP
– Armament / Weapons Delivery →VP

• Ground / Host Segment (GSF)

– Ground Control Systems →RTE

– Command and Control Subsystem →MP
– Launch and Recovery Equipment →RTE

B.4 Ordnance Vehicle Unmanned (OVU)-Ordnance

Source: MIL-STD-881C Appendix-D: Ordnance Systems

• Munition

– Guidance
∗ Seeker Assembly →SCP
 Appendix B - MIL-STD-881C WBS Mapping to Application Types 169

∗ Guidance Software →RTE

– Navigation
∗ Sensor Assembly →SCP
∗ Navigation Software →RTE
– Payload
∗ Target Defeat Mechanism →RTE
∗ Target Detection Device →RTE
∗ Fuze →SCP
∗ Payload software →VP
– Power and Distribution
∗ Primary Power →VC
∗ Power Conditioning Electronics →VC
∗ Power and distribution software →VC
– Communications
∗ Antenna Assembly →SCP
∗ Communications software →RTE
– Propulsion Subsystem
∗ Motor Engine →VC
∗ Fuel / Oxidizer Liquid Management →VC
∗ Arm / Fire Device →VC
∗ Thrust Vector Actuation →VC
∗ Flight Termination/Mission Termination >> RTE
∗ Propulsion software →VC
– Controls
∗ Controls software →VC
– On Board Test Equipment →TST
– On Board Training Equipment →TRN
– Auxiliary Equipment →SYS
– Munition Software →MP

• Launch System

– Fire Control →RTE

B.5 Ordnance Vehicle Unmanned (OVU)-Missile

Source: MIL-STD-881C Appendix-C: Missile Systems

• Air Vehicle
170 Software Cost Estimation Metrics Manual for Defense Systems

– Guidance
∗ Seeker Assembly →SCP
∗ Guidance Software →RTE
– Navigation
∗ Sensor Assembly →SCP
∗ Navigation Software →RTE
– Payload
∗ Target Defeat Mechanism →RTE
∗ Target Detection Device →SCP
∗ Fuze →SCP
∗ Payload-specific software →VP
– Power and Distribution
∗ Primary Power →VC
∗ Power Conditioning Electronics →VC
∗ Power and distribution software →VC
– Communications
∗ Antenna Assembly →SCP
∗ Communications software →RTE
– Propulsion Subsystem
∗ Motor Engine →VC
∗ Thrust Vector Actuation →VC
∗ Attitude Control System →VC
∗ Fuel / Oxidizer Liquid Management →VC
∗ Arm / Fire Device →VC
∗ Flight Termination/Mission Termination >> RTE
∗ Propulsion software →VC
– Controls
∗ Controls software →VC
– Reentry System →VC
– Post boost System →VC
– On Board Test Equipment →TST
– On Board Training Equipment →TRN
– Auxiliary Equipment →SYS
– Air Vehicle Software →VC or MP depending on decomposition

• Encasement Device

– Encasement Device Software →RTE

• Command & Launch

– Surveillance, Identification, and Tracking Sensors →SCP

 Appendix B - MIL-STD-881C WBS Mapping to Application Types 171

– Launch & Guidance Control →RTE

– Communications →RTE
– Launcher Equipment →RTE
– Auxiliary Equipment →SYS

B.6 Ordnance Vehicle Unmanned (OVU)-Launch Vehi-

cles
Source: MIL-STD-881C Appendix-J: Launch Vehicles

• Launch Vehicle
– Stage(s)
∗ Propulsions System →VC
∗ Reaction Control System →VC
∗ Recovery System →VC
∗ Environmental Control System →VC
∗ Stage Peculiar Avionics →RTE
– Avionics
∗ Guidance Navigation and Control →RTE
∗ Power →VC
∗ Data Acquisition and Telemetry →RTE
∗ Range Tracking & Safety (Airborne) →RTE
∗ Flight Software →VC

• Flight Operations
– Real-time mission control
∗ Telemetry processing →RTE
∗ Communications →RTE
∗ Data reduction and analysis →IIS

B.7 Maritime Vessel Manned (MVM)

Source: MIL-STD-881C Appendix-E: Sea Systems

• Ship
– Command, Communication & Surveillance
172 Software Cost Estimation Metrics Manual for Defense Systems

∗ Sensing and data →SCP

∗ Navigation equipment →RTE
∗ Interior communication →RTE
∗ Gun fire control system →RTE
∗ Non-electronic & electronic countermeasure →RTE
– Command, Communication & Surveillance
∗ Missile fire control systems →RTE
∗ Antisubmarine warfare fire control and torpedo fire control sys-
tems →RTE
∗ Radar systems →SCP or RTE depending on decomposition
∗ Radio communication systems →RTE
∗ Electronic navigation systems →RTE
∗ Space vehicle electronic tracking systems →RTE
∗ Sonar systems →SCP or RTE depending on decomposition
∗ Electronic tactical data systems →MP
∗ Fiber optic plant →IIS
∗ Inter / intranet →IIS
∗ Entertainment systems →IIS

B.8 Maritime Vessel Unmanned (MVU) and Maritime

Vessel Manned (MVM)
Source: MIL-STD-881C Appendix-I: Unmanned Maritime Vessel Systems

• Maritime Vehicle

– Energy Storage / Conversion

∗ Energy Storage And Conversion Monitoring And Control Sys-
tem →VC
– Electrical Power
∗ Electric Power Monitoring And Control System →VC
– Vehicle Command and Control
∗ Mission Control →RTE
∗ Navigation →RTE
∗ Guidance And Control →RTE
∗ Health Status Monitoring →SYS
∗ Rendezvous, Homing And Docking Systems →SYS
∗ Fire Control →RTE
– Surveillance →RTE
 Appendix B - MIL-STD-881C WBS Mapping to Application Types 173

– Communications / Identification →RTE

– Ship Control Systems
∗ Hovering And Depth Control →VC
∗ Ballast And Trim →VC
∗ Maneuvering System →VC
– Auxiliary Systems
∗ Emergency Systems →MP
∗ Launch And Recovery System →MP
∗ Environmental Control System →MP
∗ Anchoring, Mooring And Towing →MP
∗ Miscellaneous Fluid Systems →MP

• Payload
– Survivability Payload →VP
– Intelligence, Surveillance Reconnaissance Payload →VP
– Armament / Weapons Delivery Payload →VP
– Mission Payload →VP

• Shipboard Segment (MVM)

– Shipboard UM Command and Control Subsystem →RTE
– Shipboard Communication Subsystem →TEL
– Shipboard Power Subsystem →VC
– Launch and Recovery Equipment →RTE

B.9 Space Vehicle Manned / Unmanned (SVM/U) &

Ground Site Fixed (GSF)
Source: MIL-STD-881C Appendix-F: Space Systems

• Bus
– Structures & Mechanisms (SMS) →VC
– Thermal Control (TCS) →VC
– Electrical Power (EPS) →VC
– Attitude Control (ACS) →VC
– Propulsion →VC
– Telemetry, Tracking, & Command (TT&C) →RTE
– Bus Flight Software →VC

• Payload
174 Software Cost Estimation Metrics Manual for Defense Systems

– Thermal Control →VP

– Electrical Power →VP
– Pointing, Command, & Control Interface →VP
– Payload Antenna →SCP
– Payload Signal Electronics →SCP
– Optical Assembly →SCP
– Sensor →SCP
– Payload Flight Software →VP

• Ground Operations & Processing Center (GSF)

– Mission Management →MP

– Command and Control →MP
– Mission Data Processing →MP
– Mission Data Analysis →MP
– Collection Management →MP
– Infrastructure & Framework →SYS

• Ground Terminal / Gateway (GSF)

– Ground Terminal Software

∗ Application Specific Integrated Circuit →RTE
∗ Field Programmable Gate Array →RTE

B.10 Ground Site Fixed (GSF)

Source: MIL-STD-881C Appendix-K: Automated Information Systems

• Custom Application Software

– Subsystem Software CSCI →depends on decomposition

• Enterprise Service Element

– Software COTS / GOTS

∗ Component identification →IIS
∗ Assessment and Selection →IIS
∗ Prototyping →IIS
∗ Glue code development →IIS
∗ Tailoring and configuration →IIS

• Enterprise Information System

 Appendix B - MIL-STD-881C WBS Mapping to Application Types 175

– Business Software COTS / GOTS

∗ Component identification →IIS
∗ Assessment and Selection →IIS
∗ Prototyping →IIS
∗ Glue code development →IIS
∗ Tailoring and configuration →IIS

B.11 Ground Vehicle Manned & Unmanned(GVM/U)

Source: MIL-STD-881C Appendix-G: Surface Vehicle Systems

• Primary Vehicle
– System Survivability →RTE
– Turret Assembly →RTE
– Suspension / Steering →RTE
– Vehicle Electronics
∗ Computers And Other Devices For Command And Control
→RTE
∗ Data Control And Distribution →MP
∗ Controls And Displays →RTE
∗ Power Distribution And Management →VC
∗ Health Management Systems →VC
– Power Package / Drive Train
∗ Controls And Instrumentation →VC
∗ Power Transmission, Final Drivers, And Power Takeoffs →VC
∗ Brakes And Steering When Integral To Power Transmission
→VC
∗ Hybrid Electric Drive Systems →VC
∗ Energy Storage Systems →VC
– Fire Control
∗ Radars And Other Sensors →SCP
∗ Controls And Displays →RTE
∗ Sights Or Scopes →RTE
∗ Range Finders, Gun Drives And Stabilization Systems →VP

– Armament
∗ Main Gun And Secondary Guns →VP
∗ Missile Launchers →VP
∗ Non-Lethal Weapons →VP
176 Software Cost Estimation Metrics Manual for Defense Systems

∗ Other Offensive Weapon Systems →VP

– Automatic Ammunition Handling →MP
– Navigation and Remote Piloting →RTE
– Communications →RTE

• Remote Control System (GVU specific)

– Ground Control Systems →RTE
– Command and Control Subsystem →MP
– Remote Control System Software →RTE

B.12 Applies to ALL Environments

Source: MIL-STD-881C Appendix-L: Common Elements

• System Integration Lab (SIL)

– SIL Software - SIL Operations →TST
– SIL Software - Simulation →SCI
• Test and Evaluation Support
– Test Software →TST
• Automated Test Equipment
– Equipment Software →TST
• Training
– Equipment →TRN
– Simulators →SCI
– Computer Based-Application →IIS
– Computer Based-Web →IIS
• Support Equipment
– Software →SYS
• Test and Measurement Equipment
– Equipment Software →TST
• Data Migration
– Software Utilities →SYS
Appendix C
CER Nomograms

C.1 Overview
Nomograms can be used for rapid, graphical computation of cost models
without computers or manual calculations. They are the most concise repre-
sentation to visualize and quantify CER model relationships. They provide
instant results when approximate answers are appropriate and useful. The
graphs can be also used as crosschecks of other estimation methods and tools,
or help flag their erroneous inputs. A couple examples for the types shown in
this appendix are illustrated.
An example of using a simple nomogram is demonstrated in Figure C.1
for the Real Time Embedded - Aerial Vehicle CER. The corresponding effort
for any size on the left axis is adjacent on the right side.

KESLOC PM
250 245 1300
240 235 1250
230 225
220 215
1200
210 205
1150
200 195 1100
190 185 1050
180 175 1000
170 165 950
160 155 900
150 145 850
140 135 800
130 125 750
120 115 700
110 105 650
100 95 600
90 85 550
80 75 500
70 65 450
60 55 400
50 45 350
40 35
300
30 250
20
25 200
15 150
10 100
5 50
0 0

P M = 15.09 · KESLOC 0.81

FIGURE C.1: Simple CER Nomogram Sensitivity Analysis

177
178 Software Cost Estimation Metrics Manual for Defense Systems

An estimate for 100 KESLOC is 630 Person-Months per Figure C.1. A

sensitivity analysis that varies size is displayed as the colored range. E.g., for
the range of size between 90-130 KESLOC the corresponding effort range is
530-780 Person-Months. This is a form of uncertainty analysis.
A more complex CER is shown in Figure C.2 for the COCOMO cost
model that incorporates the Effort Adjustment Factor (EAF) (see Appendix
A). With this type of nomogram an isopleth line is drawn connecting values
on any two axes to compute the third variable. The example shown indicates
the COCOMO estimate for 100 KSLOC and EAF of 1 corresponds to 465
Person-Months. Sensitivity analysis and tradeoffs can be visualized by moving
the calculation line between values.

KSLOC EAF
250
PM 10
10000 8
200 7
6000
6
150 4000
5
3000 4.5
2000 4
100 3.5
1500
90 3
80 1000 2.5
70
600 2
60
400
50 1.5
45 300
40 200
35 150 1
30
100 0.8
25 0.7
60
20 0.6
40
0.5
30
15 0.4
20
0.35
15
0.3
10 10
0.25

FIGURE C.2: COCOMO Nomogram Estimate

C.2 Operating Environment CERs

 Appendix C - CER Nomograms 179

KESLOC PM
250
245
1550
240
235 1500
230
225 1450
220
215 1400
210 1350
205
200 1300
195
1250
190
185 1200
180
175 1150
170 1100
165
160 1050
155
150 1000
145 950
140
135 900
130
125 850
120 800
115
110 750
105
700
100
95 650
90
85 600
80 550
75
70 500
65 450
60
55 400
50 350
45
40 300
35 250
30
25 200
20 150
15
100
10
5 50
0 0

P M = 9.41 · KESLOC 0.93

FIGURE C.3: Aerial Vehicle - CER Nomogram

180 Software Cost Estimation Metrics Manual for Defense Systems

KESLOC PM
250
245
240 950
235
230
225
220 900
215
210 850
205
200
195
190 800
185
180
175 750
170
165
160 700
155
150 650
145
140
135
600
130
125
120 550
115
110
105 500
100
95
450
90
85
80 400
75
70 350
65
60
55 300
50
45 250
40
35
200
30
25
150
20
15
100
10
5 50

0 0

P M = 12.6 · KESLOC 0.79

FIGURE C.4: Ground Site - CER Nomogram

 Appendix C - CER Nomograms 181

KESLOC PM
250
245
240 1350
235
230 1300
225
220 1250
215
210 1200
205
200 1150
195
190
185 1100
180
175 1050
170
165 1000
160
155 950
150
145 900
140
135 850
130
125 800
120 750
115
110 700
105
100 650
95
90 600
85
80 550
75
70 500
65
450
60
55 400
50
45 350
40 300
35
30 250
25
200
20
15 150

10 100
5 50

0 0

P M = 15.1 · KESLOC 0.82

FIGURE C.5: Ground Vehicle - CER Nomogram

182 Software Cost Estimation Metrics Manual for Defense Systems

KESLOC PM
250
245 2600
2550
240 2500
235 2450
230 2400
225 2350
220 2300
2250
215
2200
210 2150
205 2100
200 2050
195 2000
190 1950
185 1900
1850
180 1800
175 1750
170 1700
165 1650
160 1600
155 1550
150 1500
145 1450
140 1400
1350
135
1300
130 1250
125 1200
120 1150
115 1100
110 1050
105 1000
100 950
95 900
90 850
85 800
80 750
75 700
70 650
65 600
60 550
55 500
50 450
45 400
40 350
35 300
30 250
25 200
20 150
15 100
10 50
5
0 0

P M = 5.44 · KESLOC 1.12

FIGURE C.6: Maritime Vessel - CER Nomogram

 Appendix C - CER Nomograms 183

KESLOC PM
250
245
240 1350
235
230 1300
225
220 1250
215
210
205 1200
200
195 1150
190
185
180 1100
175
170 1050
165
160 1000
155
150 950
145
140
135 900
130
125 850
120
115 800
110
105 750
100
95 700
90
85 650
80
75 600
70 550
65
60 500
55
50 450
45
400
40
35 350
30 300
25
250
20
15 200

10 150

5
100
50

0 0

P M = 27.45 · KESLOC 0.71

FIGURE C.7: Ordnance Vehicle - CER Nomogram

184 Software Cost Estimation Metrics Manual for Defense Systems

C.3 CERs by Application Type within Operating Envi-

ronments
C.3.1 Ground Site Operating Environment
 Appendix C - CER Nomograms 185

KESLOC PM
250 1750
245
240 1700
235 1650
230 1600
225
220 1550
215 1500
210
205 1450
200 1400
195
1350
190
185 1300
180 1250
175
170 1200
165 1150
160
155 1100
150 1050
145
1000
140
135 950
130 900
125
120 850
115 800
110 750
105
100 700
95 650
90
85 600
80 550
75
70 500
65 450
60 400
55
50 350
45 300
40
35 250
30 200
25
150
20
15 100
10 50
5
0 0

P M = 6.66 · KESLOC 1.01

FIGURE C.8: Ground Site - Command and Control (C&C) CER Nomogram
186 Software Cost Estimation Metrics Manual for Defense Systems

KESLOC PM
250 750
245
240
235
230 700
225
220
215 650
210
205
200 600
195
190
185 550
180
175
170
165 500
160
155
150 450
145
140
135 400
130
125
120 350
115
110
105
100 300
95
90
85 250
80
75
70 200
65
60
55
150
50
45
40
35 100
30
25
20 50
15
10
5
0 0

P M = 2.57 · KESLOC 1.03

FIGURE C.9: Ground Site - Custom AIS Software (CAS) CER Nomogram
 Appendix C - CER Nomograms 187

KESLOC PM
250 1050
245
240
235 1000
230
225
220 950
215
210
205 900
200
195
850
190
185
180 800
175
170
165 750
160
155
150 700
145
140 650
135
130
125 600
120
115 550
110
105
100 500
95
90 450
85
80 400
75
70
65 350
60
55 300
50
45 250
40
35 200
30
25 150
20
15 100
10
50
5

0 0

P M = 10.80 · KESLOC 0.83

FIGURE C.10: Ground Site - Communications (COM) CER Nomogram

188 Software Cost Estimation Metrics Manual for Defense Systems

KESLOC PM
250
245 700
690
240 680
235 670
230 660
225 650
640
220 630
215 620
210 610
205 600
590
200 580
195 570
190 560
185 550
540
180 530
175 520
170 510
165 500
490
160 480
155 470
460
150 450
145 440
140 430
135 420
410
130 400
125 390
380
120 370
115 360
110 350
340
105 330
100 320
95 310

90 300
290
85 280
270
80 260
75 250
240
70 230
65 220
60 210

55
200
190
180
50 170
45 160
40 150
140
35 130
120
30 110
25 100
90
20 80
70
15 60
50
10 40
30
5 20
10
0 0

P M = 6.14 · KESLOC 0.86

FIGURE C.11: Ground Site - Mission Planning (MP) CER Nomogram

 Appendix C - CER Nomograms 189

KESLOC PM
250
245
240 1350
235
230 1300
225
220 1250
215
210 1200
205
1150
200
195
190 1100
185 1050
180
175 1000
170
165 950
160
155 900
150
145 850
140
135 800
130
125 750
120 700
115
110 650
105
100 600
95
90 550
85
80 500
75
450
70
65 400
60
55 350
50
45 300
40 250
35
30 200
25
150
20
15 100
10
50
5
0 0

P M = 8.71 · KESLOC 0.92

FIGURE C.12: Ground Site - Real Time Embedded (RTE) CER Nomogram
190 Software Cost Estimation Metrics Manual for Defense Systems

KESLOC PM
250
245 2450
240 2400
235
230 2350
225 2300
220 2250
215
210 2200
205 2150
200 2100
195 2050
190 2000
185
1950
180
175 1900
170 1850
165 1800
160 1750
155 1700
150 1650
145
140 1600
135 1550
130 1500
125 1450
120 1400
115 1350
110 1300
105 1250
100 1200
95 1150
90 1100
85 1050
80 1000
75 950
70 900
65 850
60 800
55 750
50 700
45 650
40 600
550
35
500
30 450
25 400
20 350
15
300
250
10 200
150
5 100
50
0 0

P M = 31.88 · KESLOC 0.79

FIGURE C.13: Ground Site - Sensor Control and Signal Processing (SCP)
CER Nomogram
 Appendix C - CER Nomograms 191

KESLOC PM
250
245
1050
240
235
230 1000
225
220
215 950
210
205 900
200
195
190 850
185
180 800
175
170
165 750
160
155 700
150
145
140 650
135
130 600
125
120 550
115
110
105 500
100
95 450
90
85
80 400
75
70 350
65
60 300
55
50 250
45
40
35
200
30
150
25
20
15
100
10 50
5
0 0

P M = 7.93 · KESLOC 0.89

FIGURE C.14: Ground Site - System Software (SYS) CER Nomogram

192 Software Cost Estimation Metrics Manual for Defense Systems

C.3.2 Ground Vehicle Operating Environment

 Appendix C - CER Nomograms 193

KESLOC PM
250
245 3300
240 3200
235
230 3100

225 3000
220
2900
215
210 2800
205 2700
200
2600
195
190 2500
185
2400
180
175 2300

170 2200
165
2100
160
155 2000
150 1900
145
140 1800

135 1700
130
1600
125
120 1500
115
1400
110
105 1300
100 1200
95
90 1100
85
80 1000
75 900
70 800
65
60 700
55
600
50
45 500
40
35 400

30 300
25
20 200
15
10 100
5
0 0

P M = 7.81 · KESLOC 1.1

FIGURE C.15: Ground Vehicle - Vehicle Control (VC) CER Nomogram

194 Software Cost Estimation Metrics Manual for Defense Systems

C.3.3 Aerial Vehicle Operating Environment

 Appendix C - CER Nomograms 195

KESLOC PM
250
245
240
235
230
225 3000
220
215
210
205
200
195
2500
190
185
180
175
170
165
160 2000
155
150
145
140
135
130
125 1500
120
115
110
105
100
95
90
85 1000
80
75
70
65
60
55
50 500
45
40
35
30
25
20
15
10
5
0 0

P M = 5.61 · KESLOC 1.16

FIGURE C.16: Aerial Vehicle - Real Time Embedded (RTE) CER Nomo-
gram
196 Software Cost Estimation Metrics Manual for Defense Systems

KESLOC PM
250
245 3200
240
235 3100
230
225 3000
220 2900
215
210 2800
205
200 2700
195
2600
190
185 2500
180
175 2400
170 2300
165
160 2200
155
150 2100
145
140 2000
135 1900
130
125 1800
120 1700
115
110 1600
105
100 1500
95 1400
90
85 1300

80 1200
75
70 1100

65
1000
60
55 900

50 800
45
700
40
35 600

30 500
25
400
20
15 300

10 200

5 100

0 0

P M = 28.26 · KESLOC 0.86

FIGURE C.17: Aerial Vehicle - Sensor Control and Signal Processing (SCP)
CER Nomogram
 Appendix C - CER Nomograms 197

KESLOC PM
250 2650
245 2600
240 2550
235 2500
230 2450
225 2400
2350
220
215 2300
2250
210 2200
205 2150
200 2100
195 2050
190 2000
185 1950
180 1900
175 1850
170 1800
1750
165
1700
160 1650
155 1600
150 1550
145 1500
140 1450
135 1400
130 1350
125 1300
120 1250
115 1200
1150
110 1100
105
1050
100 1000
95 950
90 900
85 850
80 800
75 750
70 700
65 650
60 600
55 550
50 500
45 450
40 400
35 350
30 300
25 250
20 200
15 150
10 100
5 50
0 0

P M = 8.09 · KESLOC 1.05

FIGURE C.18: Aerial Vehicle - Vehicle Control (VC) CER Nomogram

198 Software Cost Estimation Metrics Manual for Defense Systems

KESLOC PM
250
245 2000
240 1950
235
230 1900
225 1850
220 1800
215
210 1750
205 1700
200 1650
195
190 1600
185 1550
180 1500
175
1450
170
165 1400
160 1350
155
1300
150
145 1250
140 1200
135 1150
130
125 1100
120 1050
115 1000
110 950
105
100 900
95 850
90 800
85 750
80
75 700
70 650
65 600
60 550
55 500
50 450
45
40 400
35 350
30 300
25 250
20 200
15 150
10 100
5 50
0 0

P M = 13.38 · KESLOC 0.91

FIGURE C.19: Aerial Vehicle - Vehicle Payload (VP) CER Nomogram

 Appendix C - CER Nomograms 199

C.3.4 Maritime Vessel Operating Environment

200 Software Cost Estimation Metrics Manual for Defense Systems

KESLOC PM
250 1150
245
240 1100
235
230
225 1050
220
215 1000
210
205 950
200
195
190 900
185
180 850
175
170 800
165
160 750
155
150 700
145
140 650
135
130
125 600
120
115 550
110
105 500
100
95 450
90
85
400
80
75
350
70
65
60 300
55
50 250
45
40 200
35
30 150
25
20 100
15
10 50
5
0 0

P M = 5.77 · KESLOC 0.96

FIGURE C.20: Maritime Vessel - Communications (COM) CER Nomogram

 Appendix C - CER Nomograms 201

C.4 CERs by AppType Across All Environments

202 Software Cost Estimation Metrics Manual for Defense Systems

KESLOC PM
250 2150
245
240 2100
235 2050
230 2000
225 1950
220 1900
215 1850
210 1800
205 1750
200 1700
195
1650
190
185 1600
180 1550
175 1500
170 1450
165 1400
160 1350
155
1300
150 1250
145
140 1200
135 1150
130 1100
125 1050
120 1000
115 950
110 900
105
850
100
95 800
90 750
85 700
80 650
75 600
70 550
65
60 500
55 450
50 400
45 350
40 300
35
250
30
25 200
20 150
15 100
10 50
5
0 0

P M = 6.6 · KESLOC 1.05

FIGURE C.21: Command and Control (C&C) CER Nomogram

 Appendix C - CER Nomograms 203

KESLOC PM
250 1100
245
240
235 1050
230
225
220 1000
215
210 950
205
200 900
195
190
185 850
180
175 800
170
165
750
160
155
150 700
145
140 650
135
130
125 600
120
115 550
110
105 500
100
95
450
90
85
80 400
75
70 350
65
60 300
55
50 250
45
40 200
35
30 150
25
20 100
15
10 50
5
0 0

P M = 7.3 · KESLOC 0.91

FIGURE C.22: Communications (COM) CER Nomogram

204 Software Cost Estimation Metrics Manual for Defense Systems

KESLOC PM
250
245
240 700
235
230
225
220 650
215
210
205 600
200
195
190 550
185
180
175
170 500
165
160
155 450
150
145
140
135 400
130
125
120 350
115
110
105 300
100
95
90
85 250
80
75
70 200
65
60
55
150
50
45
40
35 100
30
25
20 50
15
10
5
0 0

P M = 2.64 · KESLOC 1.02

FIGURE C.23: Custom AIS Software (CAS) CER Nomogram

 Appendix C - CER Nomograms 205

KESLOC PM
250 3200
245
240 3100
235
230 3000
225
220 2900
215
2800
210
205 2700
200
195 2600
190
185 2500
180 2400
175
170 2300
165
160 2200
155
2100
150
145 2000
140
135 1900

130 1800
125
120 1700
115
1600
110
105 1500
100
95 1400

90 1300
85
80 1200
75
1100
70
65 1000
60 900
55
50 800

45 700
40
600
35
30 500
25
400
20
300
15
10 200

5 100

0 0

P M = 26.5 · KESLOC 0.87

FIGURE C.24: Sensor Control and Signal Processing (SCP) CER Nomo-
gram
206 Software Cost Estimation Metrics Manual for Defense Systems

KESLOC PM
250
245
240 1100
235
230 1050
225
220 1000
215
210
205 950
200
195 900
190
185 850
180
175
170 800
165
160 750
155
150 700
145
140
135 650
130
125 600
120
115 550
110
105
100 500
95
90 450
85
80 400
75
70 350
65
60 300
55
50 250
45
40
35
200
30
150
25
20
15
100
10 50
5
0 0

P M = 7.43 · KESLOC 0.91

FIGURE C.25: Scientific and Simulation (S&S) CER Nomogram

 Appendix C - CER Nomograms 207

KESLOC PM
250
245
240 900
235
230
225 850
220
215
210 800
205
200
195 750
190
185
180 700
175
170 650
165
160
155 600
150
145
140 550
135
130
125 500
120
115
450
110
105
100 400
95
90
85 350
80
75
70 300
65
60 250
55
50
45 200
40
35 150
30
25
20 100
15
10 50
5
0 0

P M = 6.14 · KESLOC 0.91

FIGURE C.26: Mission Processing (MP) CER Nomogram

208 Software Cost Estimation Metrics Manual for Defense Systems

KESLOC PM
250 1350
245
240 1300
235
230 1250
225
220
215 1200
210
205 1150
200
195 1100
190
185 1050
180
175 1000
170
165 950
160
155 900
150
145 850
140
135 800
130
125 750
120
115 700
110
105 650
100
95 600
90
85 550
80
75 500
70 450
65
60 400
55
50 350
45
40 300
35 250
30
25 200
20 150
15
10 100
5 50

0 0

P M = 13.1 · KESLOC 0.84

FIGURE C.27: Real Time Embedded (RTE) CER Nomogram

 Appendix C - CER Nomograms 209

KESLOC PM
250
245
240 1100
235
230 1050
225
220 1000
215
210 950
205
200 900
195
190
185 850
180
175 800
170
165 750
160
155
700
150
145
650
140
135
130 600
125
120 550
115
110 500
105
100 450
95
90
85 400
80
75 350
70
65 300
60
55 250
50
45
200
40
35
150
30
25
20 100
15
10 50
5
0 0

P M = 5.06 · KESLOC 0.98

FIGURE C.28: System Software (SYS) CER Nomogram

210 Software Cost Estimation Metrics Manual for Defense Systems

KESLOC PM
250 1850
245
240 1800
235 1750
230 1700
225
1650
220
215 1600
210 1550
205
200 1500
195 1450
190 1400
185
1350
180
175 1300
170 1250
165
160 1200
155 1150
150 1100
145
140 1050
135 1000
130 950
125
120 900
115 850
110 800
105
100 750
95 700
90 650
85
80 600
75 550
70 500
65
60 450
55 400
50 350
45
40 300
35 250
30
25 200
20 150
15 100
10
5 50
0 0

P M = 7.42 · KESLOC 1.0

FIGURE C.29: Test, Measurement, and Diagnostic Equipment (TMDE)

CER Nomogram
 Appendix C - CER Nomograms 211

KESLOC PM
250
245
2500
2450
240 2400
235
2350
230 2300
225 2250
220 2200
215 2150
210 2100
205 2050
200 2000
195 1950
190 1900
185 1850
180 1800
175 1750
170 1700
165 1650
160 1600
155 1550
150 1500
145 1450
140 1400
135 1350
130 1300
125 1250
120 1200
115 1150
110 1100
105 1050
100 1000
95 950
90 900
85 850
80 800
75 750
70 700
65 650
60 600
55 550
50 500
45 450
40 400
35 350
30 300
25 250
20 200
15 150
10 100
5 50
0 0

P M = 9.05 · KESLOC 1.02

FIGURE C.30: Vehicle Control (VC) CER Nomogram

212 Software Cost Estimation Metrics Manual for Defense Systems

KESLOC PM
250 1950
245
240 1900
235 1850
230
225 1800
220 1750
215
210 1700
205 1650
200
195 1600
190 1550
185
180 1500
175 1450
170
165 1400
160 1350
155
150 1300
145 1250
140 1200
135
130 1150
125 1100
120
115 1050
110 1000
105 950
100
95 900
90 850
85
800
80
75 750
70 700
65 650
60 600
55
550
50
45 500
40 450
35 400
30 350
25 300
20 250
15 200
10 150

5 100
50
0 0

P M = 22.27 · KESLOC 0.81

FIGURE C.31: Vehicle Payload (VP) CER Nomogram

 Appendix C - CER Nomograms 213

C.5 Cost Model Nomograms

C.5.1 COCOMO
214 Software Cost Estimation Metrics Manual for Defense Systems

KESLOC PM
250
245 1250
240
235 1200
230 1150
225
220 1100
215
210 1050
205
200 1000
195
190 950
185
180 900
175
850
170
165 800
160
155 750
150
145 700
140
135 650
130
125 600
120
115 550
110
105 500
100 450
95
90
85 400
80 350
75
70
65 300
60 250
55
50
45 200
40
35 150
30
25 100
20
15 50
10
5
0 0

P M = 2.94·KESLOC 1.0997

FIGURE C.32: COCOMO Nominal Nomogram

 Appendix C - CER Nomograms 215

KSLOC
1000
900
800
700 PM
600 60000

500 40000
450 30000
400 25000 EAF
20000 10
350
15000
300 8
10000 7
250 8000
6
6000
200 5
4000 4.5
3000 4
150 2500 3.5
2000
3
1500
100 2.5
90 1000
800 2
80
600
70
400 1.5
60
300
50 250
45 200 1
40 150
35 0.8
100 0.7
30 80
0.6
60
25
0.5
40 0.45
20 30 0.4
25 0.35
20
15 0.3
15
0.25
10
8 0.2
10
9 6
8 4 0.15
7 3
6 2.5
2 0.1
5 1.5
4.5
4 1
3.5
3 P M = 2.94 ∗ KSLOC 1.0997 ∗ EAF

2.5

FIGURE C.33: COCOMO with EAF Nomogram

216 Software Cost Estimation Metrics Manual for Defense Systems
Appendix D
Unified CodeCount

D.1 Overview
Unified CodeCount (UCC) is a source code counting and differencing tool
[41]. It allows the user to count, compare, and collect both physical and logical
differentials between two versions of the source code of a software product.
The differencing capabilities allow users to count the number of added/new,
deleted, modified, and unmodified physical and logical source lines of code
(SLOC) of the current version in comparison with the previous version. With
the counting capabilities, users can generate the physical, logical SLOC counts,
and other sizing information, such as comment and keyword counts, of the
target program.
The tool can be compiled using a C/C++ supported compiler. It is run by
providing files of filenames to be counted or providing the directories where
the files reside. It can be downloaded from http://csse.usc.edu/research/
CODECOUNT/.
Sample output is shown below in Figure D.34.

RESULTS SUMMARY
Total Blank Comments Compiler Data Exec. Number File SLOC
Lines Lines Whole Embedded Direct. Decl. Instr. of Files SLOC Type Definition
11833 1140 1620 328 60 496 8517 22 9073 CODE Physical
11833 1140 1620 328 60 496 6434 22 6983 CODE Logical

FIGURE D.34: Unified CodeCount Example Summary Output

The SLOC count in this summary is the total of 6983 Logical SLOC con-
sisting of 496 data declarations and 6434 executable instructions. This is the
software size to be used for estimation, measurement of actuals, and model
calibration.

217
218 Software Cost Estimation Metrics Manual for Defense Systems
Appendix E
Acronyms

4GL Fourth Generation Language

AAF Adaptation Adjustment Factor

AAM Adaptation Adjustment Multiplier

ACAT Acquisition Category

ASP Acquisition Support Plan

AV Aerial Vehicle

AVM Aerial Vehicle Manned

AVU Aerial Vehicle Unmanned

CARD Cost Analysis Requirements Document

CER Cost Estimating Relationship

CM Code Modified Percentage

CMM Capability Maturity Model

COCOMO COnstructive COst MOdel

COCOTS COnstructive Commercial-off-the Shelf

COTS Commercial-off-the Shelf

CSCI Computer Software Configuration Item

CSDR Cost and Software Data Report

DACIMS Defense Automated Cost Information Management System

DAMS Defense Acquisition Management System

DCARC Defense Cost and Resource Center

DM Design Modified Percentage

DoD Department of Defense

219
220 Software Cost Estimation Metrics Manual for Defense Systems

EI External Inputs
EIF External Interfaces
EO External Outputs
EQ External Inquiries
EKSLOC Equivalent Thousands of Source Lines of Code
ESLOC Equivalent Source Lines of Code
FPA Function Point Analysis
FPC Function Point Count
GAO U.S. General Accounting Office
GS Ground Site
GSF Ground Site Fixed
GSM Ground Site Mobile
GUI Graphical User Interface
GV Ground Vehicle
GVM Ground Vehicle Manned
GVU Ground Vehicle Unmanned, e.g., Robot vehicles
HOL Higher Order Language
HWCI Hardware Configuration item
IDPD Incremental Development Productivity Decline
IFPUG International Function Point User’s Group
IIS Intelligence and Information Software
ILF Internal Files
IM Integration Modified Percentage
IRS Interface Requirement Specification
IS Information System
KESLOC Thousands of Equivalent Source Lines of Code
KSLOC Thousands of Source Lines of Code
LCC Life Cycle Cost
 Appendix E - Acronyms 221

MDAP Major Defense Acquisition Program

MP Mission Processing

MSLOC Millions of source lines of code

MV Maritime Vessel

MVM Maritime Vessel Manned

MVU Maritime Vessel Unmanned

NASA National Aeronautics and Space Administration

NDI Non-Development Item

NLM Non-Linear Model

OLS Ordinary Least Squares regression

OO Object Oriented

OpEnv Operating Environment

OSD Office of the Secretary of Defense

OV Ordnance Vehicle

OVU Ordnance Vehicle Unmanned

PC Process Control

PLN Planning software

Pr Productivity

RTE Real-Time Embedded

RUP Rational Unified Process

SCI Scientific Software

SCP Sensor Control and Signal Processing (SCP)

SDD Software Design Document

SDP Software Development Plan

SDR Software Design Review

SEER-SEM System Evaluation and Estimation of Resources Software Es-

timating Model (software cost estimation tool)

SEI Software Engineering Institute

222 Software Cost Estimation Metrics Manual for Defense Systems

SER Schedule Estimating Relationship

SLIM Software LIfecycle Management (software cost estimation tool)

SLOC Source Lines of Code

SRDR Software Resource Data Report

SRR Systems Requirements Review

SRS Software Requirements Specification

SSR Software Specification Review

SSS System Segment Specification

SU Software Understanding
SV Space Vehicle

SVM Space Vehicle Manned

SVU Space Vehicle Unmanned

SYS System Software

TEL Telecommunications software

TOOL Software Tools

TST Test software

TRN Training Software

UCC Universal Code Counter

UNFM Programmer Unfamiliarity

USC University of Southern California

VC Vehicle Control

VP Vehicle Payload
WBS Work Breakdown Structure
Bibliography

[1] C. Abts. Extending the COCOMO II Software Cost Model to Estimate

Effort and Schedule for Software Systems Using Commercial-off-the-Shelf
(COTS) Software Components: The COCOTS Model. PhD thesis, Uni-
versity of Southern California, May 2004.

[2] Air Force Cost Analysis Agency (AFCAA). Cost Risk and Uncertainty
Analysis Handbook. United States Air Force Cost Analysis Agency,
Hanscom AFB, MA, 2007.

[3] D. Anderson. Kanban. Blue Hole Press, Sequim, WA, 2010.

[4] R. Banker and C. Kemerer. Scale economies in new software development.

IEEE Transactions on Software Engineering, October 1989.

[5] B. Boehm. Software Engineering Economics. Prentice Hall, 1981.

[6] B. Boehm. Applying the incremental commitment model to brownfield

system development. In Proceedings of Conference on Systems Engineer-
ing Research(CSER), 2009.

[7] B. Boehm. Future challenges for systems and software cost estimation
and measurement. In Proceedings of the 13th Annual Practical Software
Measurement (PSM) Conference, 2009.

[8] B. Boehm, C. Abts, A.W. Brown, S. Chulani, B. Clark, E. Horowitz,

R. Madachy, D. Reifer, and B. Steece. Software Cost Estimation with
COCOMO II. Prentice Hall, 2000.

[9] B. Boehm and V. Basili. Software defect reduction top 10 list. IEEE
Computer, January 2001.

[10] B. Boehm, J. Lane, S. Koolmanojwong, and R. Turner. The Incremental

Commitment Spiral Model: Principles and Practices for Successful Sys-
tems and Software. Addison Wesley, 2014.

[11] B. Boehm and K. Lee. Empirical results from an experiment on value-

based review (vbr) processes. In Proceedings of International Symposium
on Empirical Software Engineering (ISESE). IEEE Computer Society,
2005.

223
224 Software Cost Estimation Metrics Manual for Defense Systems

[12] G. Booch. private communication, 2009.

[13] M. Broy. Seamless Method- and Model-based Software and Systems En-
gineering, in The Future of Software Engineering. Springer, 2011.

[14] M. Cohn. Agile Estimating and Planning. Prentice Hall PTR, Upper
Saddle River, NJ, 2005.

[15] E. Colbert and B. Boehm. Cost estimation for secure software & systems.
In Proceedings of ISPA / SCEA 2008 Joint International Conference,
June 2008.

[16] A. Elssamadisy and G. Schalliol. Recognizing and responding to bad

smells in extreme programming. In Proceedings of ICSE, 2002. IEEE,
2002.

[17] D. Galorath and M. Evans. Software Sizing, Estimation, and Risk Man-
agement. Taylor & Francis Group, Boca Raton, FL, 2006.

[18] United States Government Accountability Office (GAO. GAO Cost Esti-
mating and Assessment Guide. United States Government Accountability
Office (GAO, Washington, D.C., 2009.

[19] D. Garmus and D. Herron. Function Point Analysis: Measurement Prac-

tices for Successful Software Projects. Addison Wesley, Boston, Mass,
2000.

[20] G. Gill and M. Iansiti. Microsoft corporation: Office business unit. Har-
vard Business School Case Study 691-033, 1994.

[21] W. Goethert, E. Bailey, and M. Busby. Software effort and schedule

measurement: A framework for counting staff-hours and reporting sched-
ule information. Technical Report ESC-TR-92-021, Software Engineering
Institute / Carnegie-Mellon University, Pittsburgh, PA, USA, 1992.

[22] International Function Point Users Group. Function Point Counting

Practices: Manual Release 4.1.1. International Function Point Users
Group, Princeton Junction, NJ, 2000.

[23] R. Hopkins and K. Jenkins. Eating the IT Elephant: Moving from Green-
field Development to Brownfield. IBM Press, 2008.

[24] ISO. ISO JTC 1/SC 27, Evaluation Criteria for IT Security, in Part 1:
Introduction and general model. ISO, Geneva, Switzerland, 1999.

[25] ISO/IEC 12207:1995. Standard for Information Technology – Software

Life Cycle Processes ISO/IEC 12207. International Organization for
Standards (ISO)/International Electronical Commission (IEC), Geneva,
Switzerland.
 Bibliography 225

[26] R. Jensen. An improved macrolevel software development resource esti-

mation model. In Proceedings of 5th ISPA Conference, 1983.
[27] J. Lane. Cost model extensions to support systems engineering cost es-
timation for complex systems and systems of systems. In Proceedings of
Conference on Systems Engineering Research(CSER), 2009.
[28] J. Lane and B. Boehm. Modern tools to support dod software inten-
sive system of systems cost estimation - a dacs state-of-the-art report.
Technical Report 347336, Data and Analysis Center for Software, 2007.
[29] G. Lewis and et al. Smart: Analyzing the reuse potential of legacy com-
ponents on a service-oriented architecture environment. Technical Report
CMU/SEI-2008-TN-008, Software Engineering Institute (SEI)/Carnegie-
Mellon University (CMU), Pittsburgh, PA, USA, 2008.
[30] Q. Li, M. Li, Y. Yang, Q. Wang, T. Tan, B. Boehm, and C. Hu. Bridge
the gap between software test process and business value: A case study.
In Proceedings of International Conference on Software Process (ICSP).
IEEE Computer Society, 2009.
[31] K. Lum, J. Powell, and J. Hihn. Validation of spacecraft software cost
estimation models for flight and ground systems. JPL Report, 2001.
[32] R. Madachy and B. Boehm. Comparative Analysis of COCOMO II,
SEER-SEM and True-S Software Cost Models. Technical Report USC-
CSSE-2008-816, University of Southern California Center for Software
Engineering, Los Angeles, USA, 2008.
[33] R. Madachy, B. Boehm, C. Abts, and S. Chulani. Software development
cost estimation approaches a survey. Technical Report USC-CSE-00-50,
University of Southern California Center for Software Engineering, Los
Angeles, USA, 2000.
[34] Quantitative Software Management. SLIM Estimate for Windows User’s
Guide. Quantitative Software Management, 2003.
[35] R. Moazeni. Incremental Development Productivity Decline. PhD thesis,
University of Southern California, August 2014.
[36] R. Moazeni, D. Link, and B. Boehm. COCOMO II parameters and IDPD:
Bilateral relevances. In Proceedings of the 2014 International Conference
on Software and System Process. ACM, 2014.
[37] V. Nguyen. Improved Size and Effort Estimation Models for Software
Maintenance. PhD thesis, University of Southern California, December
2010.
[38] United States Department of Defense. Instruction 5000.2, Operation of
the Defense Acquisition System. 2008.
226 Software Cost Estimation Metrics Manual for Defense Systems

[39] United States Department of Defense. Work Breakdown Structures for

Defense Materiel Items. (MIL-STD-881C), 2011.
[40] United States. Under Secretary of Defense. Acquisition Technology and
Logistics. Systems Engineering Guide for System of Systems, Version 1.0.
June 2008.
[41] University of Southern California. Unified CodeCount: http://csse.
usc.edu/research/CODECOUNT/.
[42] National Institute of Standards and Technology. Exploratory Data Anal-
ysis Techniques: http://www.itl.nist.gov/div898/handbook/eda/
section3/eda3.htm.
[43] United States. Office of the Secretary of Defense. Defense Cost and Re-
source Center. CSDR Overview and Policy: http://dcarc.cape.osd.
mil/CSDR/CSDROverview.aspx.
[44] United States. Office of the Secretary of Defense. Defense Cost and Re-
source Center. Software resources data reporting: Final developer report
and data dictionary. (DI-MGMT-8174A), 2011.
[45] R. Park. The central equations of the PRICE software cost model. In
Proceedings of the 1988 COCOMO Users Group Meeting, 1988.
[46] L. Putnam and W. Myers. Measures of Excellence. Prentice-Hall, Inc,
Englewood Cliffs, NJ, 1992.
[47] Park R. Software size measurement: A framework for counting source
statements. Technical Report ESC-TR-92-020, Software Engineering In-
stitute (SEI)/Carnegie-Mellon University (CMU), Pittsburgh, PA, USA,
1992.
[48] Park R. A manager’s checklist for validating software cost and schedule
estimates. Technical Report CMU/SEI-95-SR-004, Software Engineering
Institute / Carnegie-Mellon University, Pittsburgh, PA, USA, 1995.
[49] D. Reifer. Let the numbers do the talking. CrossTalk, March 2002.
[50] Cyber Security and Information Systems Information Analysis Center
(CSIAC). Software & Systems Cost and Performance Analysis Toolkit :
https://s2cpat.thecsiac.com/.
[51] R. Selby. Empirically Analyzing Software Reuse in a Production Environ-
ment, In Software Reuse: Emerging Technology, W. Tracz (Ed.). IEEE
Computer Society Press, 1988.
[52] M. Stamatelatos and H. Dezfuli. Probabilistic Risk Assessment Proce-
dures Guide for NASA Managers and Practitioners. NASA/SP-2011-
3421, National Aeronautics & Space Administration (NASA). Office of
Safety and Mission Assurance, Washington, D.C., 2011.
 Bibliography 227

[53] R. Stutzke. Estimating Software-Intensive Systems. Addison-Wesley,

Upper Saddle River, N.J, 2005.

[54] T. Tan, Q. Li, B. Boehm, Y. Yang, M. He, and R. Moazeni. Productivity

trends in incremental and iterative software development. In Proceedings
of ACM-IEEE ESEM, 2009. ACM-IEEE, 2009.
[55] R. Valerdi. The Constructive Systems Engineering Cost Model
(COSYSMO). PhD thesis, University of Southern California, August
2005.
228 Software Cost Estimation Metrics Manual for Defense Systems
Index

AAF, 25, 26, 41–42 COTS, 13, 20, 23, 25, 114, 115, 123,
Activities, 16, 17, 29–30, 134, 144 131, 132
Adaptation Adjustment Factor, see CSDR, 9, 17
AAF Custom AIS Software (CAS), 53, 78,
Adapted software, 20, 23, 25–26, see 201
also Modified software and
Reused software DACIMS, 2, 4, 10
Aerial Vehicle (AV), 47, 56, 71, 184, DAMS, 2, 3
194 Data assessment, 6, 32–43
Application Type, 4, 5, 7, 13, 48–56, Data Dictionary, 11, 16, 34, 38
128 Data normalization, 1, 5, 134, 136,
Auto-generated software, 15, 35, 36, 144
42 Data quality, 33, 36–38
Data segmentation, 56
Blank lines, 22 DCARC, 9, 17
Declarations, 21, 22
Defects, 16
Carryover code, 15 Deleted software, 15
CER, vii, 1–7, 33, 44–109, 135 Design modified (DM), 25, 41–42, 112
COCOMO, 116, 117, 119, 146, 147, DoD, vii, 1–4, 10–11, 17, 126
149, 152, 153, 156–158, 161– Duration, 31, 33, 34, 134–136, 144, see
164, 213 also Schedule
Code counting, 7, 15, 111
Code modified (CM), 25, 41–42, 112 Effort, 1, 4, 6, 15, 29, 33, 35, 36, 38,
Command and Control (C&C), 50, 39, 134–136, 144–150, 153–
201 157, 161
Command and Control (CC), 77 Enterprise Information Systems (EIS),
Comments, 22 54
Communication (COM, 79 Enterprise Service Systems (ESS), 53
Communications (COM), 50, 201 Equivalent size, 4, 13, 20, 23–26, 33,
Compiler directives, 22, 23 35, 37, 39, 40, 42, 112, 117,
Converted software, 20, 22, 25 149, 152, 153, 155, see also
Cost, 1, 4, 5 ESLOC
Cost Estimating Relationships, see Equivalent Source Lines of Code, see
CER ESLOC
Cost estimation process, 3, 7 ESLOC, 24, 36, 112, 113
Cost model, 3, 5, 7, 145–161, 213 Executables, 21, 22

229
230 Software Cost Estimation Metrics Manual for Defense Systems

Experience, 16 Ordnance Vehicle (OV), 47, 56, 75,

External interface requirements, 14 184

Function Points, 26–28 Person-Months, 1, 4, see also Effort,

113
GAO, 7, 125, 141
Personnel experience, 14
Generated software, 20, 22, 25
Personnnel experience, 150
GFS, 20, 23, see also COTS
Phases, 29–30, 132
GOTS, 13, 132
Physical lines, 21–23, 37
Ground Site (GS), 47, 56, 72, 184
Process Control (PC), 51
Ground Vehicle (GV), 47, 73, 184, 192
Product quality, see Quality
Integration required (IM), 25, 41–42, Productivity, 36, 102–108
112 Productivity benchmarks, vii, 1, 5, 6,
33, 102–108, 135
KESLOC, 4 Programming language, 13, 15

Labor categories, 30, 161 Quality, 16, 17, 33

Labor hours, 31, 35, 134
Labor rate, 1, 134, 136 Real Time Embedded (RTE), 50, 81,
Lifecycle phases, see Phases 201
Linear regression, 58, 60, 63–69 Requirements, 14, 17
Logical source statements, 19, 22–23, Requirements stability, see Require-
37–39, 134, see also SLOC ments volatility
Requirements volatility, 14, 112, 121,
Maritime Vessel (MV), 47, 56, 74, 184,
134
199
Response variable, 45, 56, 60, 63–66
MDAP, 9
Reused software, 13, 15, 20, 22, 25, 35,
Metrics definitions, 5, 6, 19–31
36, 42, 112
Microcode and Firmware (M&F), 48
MIL-STD-881C, 7, 54–56
Schedule, 15, 31, 33, 36, see also Du-
Mission Planning (MP), 53, 80
ration, 132, 138
Mission Processing (MP), 201
Scientific and Simulation (S&S), 51,
Modified software, 20, 22, 25, 33, 35,
82, 201
36, 41
SEER-SEM, 116, 146–149, 157, 158,
NCSS, 21–23, 37, 39 161–164
New software, 13, 20, 22, 23, 25, 26, Sensor Control and Signal Processing
41 (SCP), 49, 83, 201
Non-Commented Source Statements, Size, 4–6, 14, 16, 19, 33, 35, 36, 38,
see NCSS 134, 137, 144
Nonexecutables, 22 adapted, 19
Normalization, see Data normaliza- equivalent, 4, 13, 20, 23–26, 33,
tion, 33, 38, 134 35, 37, 39, 40, 42, 112, 117,
149, 152, 153, 155
Operating Environment, 1, 2, 7, 56, SLIM, 146, 148, 150, 152, 155–158,
70, 128 161–164
 Index 231

SLOC, 4, 19–23, 33, 34, 38, 111, 113,

115, 117, 134
Software activities, see Activities
Software cost, see Cost
Software process maturity, 13
Software requirements, see Require-
ments
Software Resources Data Report, see
SRDR
Software reuse, see Reused software
Software size, see Size
Software Tools (TOOL), 52
Software volatility, 25, 26
Source Lines of Code, see SLOC
Space Vehicle (SV), 47, 56
SRDR, vii, 1–7, 9–17, 29, 34, 38, 112,
134
Staffing, 13, 16, 33
System Software (SYS), 51, 84, 201

Test, Measurement, and Diagnostic

Equipment (TMDE), 52, 85,
201
Total equivalent size, 26
Total lines, 21–23, 37–39
Training (TRN), 52
True Planning, 146, 147, 150, 152,
154–158, 161–164

Unified Code Count, 25, 39

Vehicle Control (VC), 49, 86, 201

Vehicle Payload (VP), 49, 87, 201

WBS, 7, 12, 15, 132

Work Breakdown Structure, see WBS