Testing Doesn't Cost It Pays

Testing Doesn’t Cost – It Pays
OT&E/DT&E response to
Acquisition – T&E Relationships
Assessment Report
April 2011
http://www.dote.osd.mil/pub/presentations.html
Reported Root Causes & Mitigation Areas
• Weak linkage amongst Requirements, Program, and Test
Communities
• Issues with Requirements Setting and Management
• Acquisition Strategy – Test Strategy Misalignment and
TEMP Management
• The “Tail End Charlie” Syndrome
• Troubled Programs
Test community agrees with report’s conclusions:

“The Task Team found no significant evidence that the testing community
typically drives unplanned requirements, cost or schedule into programs.”
Mitigations Currently in Work
• Promote early, effective, frequent communication at working level
– IPTs, Working Groups and “Core Teams” with PMs, User Reps, System
Engineers, DT & OT
• Evaluate systems vs. requirements and broader mission context
– Testing to evaluate systems’ mission accomplishment despite Program
Office desire to have their system evaluated in isolation
– DT&E example: F-22 Increment 3.1 Synthetic Aperture Radar Tests – user
input during DT for display and usability of SAR maps
– OT&E example: USS Virginia – did not meet KPPs but was evaluated as
effective
• Discourage stalemates & Elevate issues earlier
– DOT&E policy for Early Review of TEMPs and Test Plans before
coordination cycle begins
– DT&E Early and Continuous Engagement (RFP to IOT&E)
• Plan appropriate scope of testing to identify deficiencies early
– Rational, analytical approach to support test sizing
– Recent examples include SDB II, JASSM, JATAS
Mitigations Planned for Action
• TEMP at Milestone A
– Earlier insight into test resource requirements
– Sets baseline for smoother TEMP approval prior to MS B
• TEMP update flexibility
– Especially important for IT systems
– Current coordination process averages ~6 months
• Realistic expectations at requirements definition
– Testers provide feedback on testability of requirements
– KPP list must consider the “so what?” factor
– Trades for affordability – accepted risk
– Requirements change frequently is a symptom – not a cause – of
program delay
• GAO Report 11-233SP found that programs with decreased, deferred, or
deleted requirements had 40% schedule increase compared to 8% increase
for those programs with no change in requirements
Point of Disagreement: “Giving a Grade”
• Developmental testing characterizes performance

– DT Assessment of Operational Test Readiness (AOTR) is a
“progress report”
– Provided to AT&L, SAE, and DOT&E
• Operational testing WILL provide a grade:

– Sometimes we have to call the baby ugly
– Assures fighting forces and combat developers that the system
can be used in combat
– Required by law
– However, it is an OPEN BOOK exam
Reasons Behind Program Delays
• Case studies of 41 current major programs with

significant delays1
– More than half of the programs had their FRP delayed more than
two years
– All programs had a least one year delay in a major milestone
• Two-thirds of the programs had performance issues in DT

– More than half of those programs had performance issues in OT as
well as poor performance in DT
• Performance problems discovered in testing as opposed

to problems with testing caused majority of delays
1. “Reasons Behind Program Delays,” briefing, Institute for Defense Analyses, 30March2011
Reasons Behind Program Delays
“T&E cost issues in a program are typically the result of under‐estimating the impact of system
complexity; inadequate cost estimating; and/or/ inadequate/immature engineering.”
6 Conducting Test
12 25
Performance in DT
Number of Programs
20
18 Performance in OT
30 15 >3 year delay
Programmatic
2-3 year delay
10
1-2 year delay
Manufacturing
5
89 issues from 41 case studies
23
0
‐ 41 selected case studies showed 89 instances MS C MS III FRP
of issues in five categories resulting in delays ‐ 36 case studies had a major
milestone delay over 1 year; 22
‐ Six of these programs had delays because of of these were more than 3 years
test issues. In no case, were the test issues the
primary causes of overall delay. ‐ 5 other cases delayed fielding
or were canceled
Cost of OT Relative to Program Cost
“…the cost of [testing] is a small portion of the overall program budget; it is a large percent of
the budget in the year(s) in which it occurs.
• Review of 76 recent programs
• Average marginal OT&E cost was 0.65%
• Low Program Acquisition Cost is dominant source of high relative OT&E cost
• Expense of test articles and their expendability is a major driver
30
Number of Programs
25 Few programs (7 out of 76) required

20 more than 1.51% of program
acquisition costs for OT&E
15
10
Cost of OT&E as Percent of Program Acquisition Cost
OT&E is usually 1% ± 0.5% of Program Acquisition Cost

Backups
Listing of programs examined from “Reasons Behind Program Delays,” briefing,
Institute for Defense Analyses, 30March2011 .
Conducting Test
Conducting Test
Problems in OT
Problems in DT
Manufacturing
Programmatic
Performance
Performance
Problems in
Observed
Problem
Program Delay
Joint Strike Fighter FRP delayed 3 years 1 1 1

P-8A Poseidon MS C delayed 18 months 1 1 improper instrumentation during DT
AIM-9X 8.212 OT completion delayed 18 months 1 1
AARGM FRP delayed over 2 years 1 1 1
CIRCM FRP delayed 4 years 1
IDECM Block 3 FRP delayed 5 years 1 1 1
LAIRCM Phase II FRP delayed over 4 years 1 1
SIRFC FRP delayed over a year 1 1 1 1
AOC-WS 10.1 Fielding delayed one quarter 1
MIDS JTRS FRP delayed about a year 1 1 1
Mark XIIA Mode 5 FRPD delayed 3 years 1 1
DoN LAIRCM MS C delayed a year 1 1
MALD IOT&E delayed over 3 years 1 1 1 range availability
B-2 RMP FRP delayed 2 years 1
RMS FRP delayed 9 years 1 1 1
ALMDS FRP delayed 4 years 1
MH-60S Block 2A AMCM FRP delayed over 4 years 1 1
AMNS FRP slipped over 6 years 1
LPD 17 MS III delayed 3 years 1 1 1 targets
SM-6 FRP delayed a year 1 1 telemetry
LCS FOC delayed a year 1 1
Virginia MS III delayed 2 years 1 1 1 1 1 targets
DDG 1000 MS B rescinded 1
CH-47F FRP delayed 3 years 1 1 1
AH-1Z FRP delayed over 4 years 1 1 1
VTUAV IOT&E delayed 3 years 1
Problem Observed
Conducting Test
Conducting Test
Problems in OT
Problems in DT
Manufacturing
Programmatic
Performance
Performance
Problems in
Program Delay
Spider Networked Munition FRP delayed 6 years 1

Precision Guidance Kit MS C delayed 4 years 1
(PGK)
Excalibur Increment Ia-2 FRP delayed over 2 years 1 1
PIM MS C delayed 3 years 1 1
JLTV MS C delayed over 2 years 1 1
E-IBCT 3 of 5 systems cancelled 1 1
JTRS HMS Rifleman Radio MS C, FRP delayed 2 years 1 1
Gray Eagle FRP delayed over 2 years 1 1
Stryker MGS FRP delayed over 3 years 1 1
Net-Centric Enterprise FRP delayed 2 years 1 1 1
Services lack of user base
NPOESS FRP delayed 2 years 1 1
GCCS JOPES 4.2 and Fielding delayed 2 years 1 1
4.2.1
CITS AFNet Increment 1 Fielding delayed 2 years 1 1 1
Patriot PAC-3 FRP delayed 15 years 1 1
MEADS LRIP delayed 9 years 1 1

Testing Doesn't Cost It Pays

Uploaded by

Copyright:

Available Formats

Testing Doesn't Cost It Pays

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Testing Doesn't Cost It Pays

Uploaded by

Copyright:

Available Formats

Testing Doesn’t Cost – It Pays

Test community agrees with report’s conclusions:

• Developmental testing characterizes performance

• Operational testing WILL provide a grade:

• Case studies of 41 current major programs with

• Two-thirds of the programs had performance issues in DT

• Performance problems discovered in testing as opposed

25 Few programs (7 out of 76) required

Cost of OT&E as Percent of Program Acquisition Cost

OT&E is usually 1% ± 0.5% of Program Acquisition Cost

Joint Strike Fighter FRP delayed 3 years 1 1 1

Spider Networked Munition FRP delayed 6 years 1

You might also like