Organizing for a Complex World: The Way Ahead

By
David J. Berteau, Guy Ben-Ari, and Matthew Zlatnik
Copyright 2009
Center for Strategic and International Studies
Washington, DC

For-Comment Draft of April 10, 2009

This draft is not for quoting except as part of the review and comment process.
Permission for use for any other purpose should be obtained from the authors at CSIS.

David J. Berteau [email protected] (202) 775-3136

Guy Ben-Ari [email protected] (202) 775-3206
Matthew Zlatnik [email protected]

Center for Strategic and International Studies

1800 K Street NW
Washington, DC 20006
Abstract

Complex defense and network-centric systems have proven to be difficult to

develop on time and on budget, a consequence of the complexity inherent in both the
systems and the acquisition environment. Complexity in turn results from non-linear,
unpredictable interaction of elements combined in new ways, in order to try to create
unique capabilities. Complex development programs pose governance and management
challenges for a range of systems-integration models, and it is difficult to know in
advance the program-management model most suitable for a given program. This paper
proposes ways to measure or assess success in managing complex programs. It also
addresses ways that the challenge of picking the proper development model can be
partially bypassed, by seeking to cultivate flexibility and resiliency (F&R) within the
organization. Through the ability to understand and adapt to changes in the internal and
external environments, a program-management organization can thrive in a development
environment in which unanticipated events will certainly occur.

Introduction

Complex defense and network-centric systems, such as the Armys Future

Combat System (FCS), the Coast Guards Integrated Deepwater System (IDS) and the
FAAs Next Generation Air Traffic System, have been shown to be far more ambitious
than any previously attempted by the U.S. government. Successfully managing these
complex programs is vital, because they were designed to provide the core of many
critical future national security capabilities. This is true even though the Secretary of
Defense has proposed restructuring the FCS program.

Such systems are difficult to develop and oversee. They incorporate technology
that was not yet created when the systems were still on the drawing board. These
programs are actualized by a team of government managers and industry practitioners,
aided by a vast assemblage of engineering and scientific talent, overseen by political
forces, monitored by auditors at every step, regulated by rules measured in linear feet,
and ultimately evaluated in life-and-death situations. This is tough, difficult work.

For the government, managerial techniques struggle to keep up as complexity

blossoms, often resulting in blown budgets and missed schedules. Our research concludes
that such overruns are rarely the fault of the organizations or the personnel involved.
Rather, they result from the overwhelming difficulty of creating systems comprised of
thousands of elements, addressing dozens or hundreds of requirements, produced by
multiple manufacturers under the direction of one of a handful of prime contractors.
Producing a state-of-the-art weapons system has proved to be demonstrably difficult. The
fact that problems appear regardless of how programs are managed does give us insight,
however, and points toward a potential solution. We conclude that successful
management of complex programs requires the government to make sure that whatever
model it uses, that model can easily recognize and adapt to the challenges complexity
will pose.

2 / 16
 In a recent CSIS book, Organizing for a Complex World: Developing
Tomorrows Defense and Net-Centric Systems, several models were put forward for
rethinking the policy framework in which complex programs are developed. Upon
reviewing these models, one key question emerges: how can program managers and
policymakers choose the model most appropriate for each program? In other words, how
can they measure, compare and assess the relative value of alternatives? And once a
governance model is selected, how do policymakers and managers implement it to cope
with the inevitable unintended consequences and unexpected developments that
accompany complexity?

We posit that policy makers, in their efforts to select the right model to govern
and manage a given defense acquisition program, should focus less on making exactly the
right choice and more on ensuring that whatever option they do select can successfully
identify, survive and respond to changes. In any large defense acquisition program,
changes are inevitable, in user requirements, program scope, budget, outside political
demands, and the operating environment. The impact of these changes is hard to
anticipate, because of the complexity inherent in the program and its environment. And
these impacts can best be handled by a program management model that embodies
flexibility and resilience. We are not recommending a model. Rather, we are
recommending an organizational way of life attributes that a program-management
organization must have, if it is to be able to manage and deliver complex projects
successfully.

This paper describes the challenges that complexity brings to defense acquisition,
highlights the need for a way to compare and assess governance models, and shows how
the challenge of picking the right governance model can be augmented by ensuring that
the model has flexibility and resilience. We define flexibility and resilience as the ability
to recognize, absorb and react to changes in the development environment. We then
suggest ways to achieve flexibility and resilience.

Complexity defined

It is easy to confuse complex with complicated. Programs that seem complex

to many observers are often better labeled complicated. Complicated systems are
characterized by their large scale and by a multitude of moving parts or actors that are
highly dynamic, that constantly interact with and affect one another, and that behave
primarily in a linear fashion. Complicated programs are relatively common and can be
managed to successful delivery by decomposing the program into subprojects and then
using systems engineering techniques to identify and resolve (integrate)
interdependencies across subproject boundaries.

Complex programs, on the other hand, are non-linear and are comprised of
multiple, interrelated elements that interact unpredictably. Even an in-depth familiarity
with each of these elements does not impart an understanding of the system as a whole.
Complex programs are characterized by nonlinear feedback loops and recursiveness.

3 / 16
They are sensitive to small differences in initial conditions, and in their emergent phase
this significantly inhibits the validity of any detailed long-term planning. They are often
implemented in highly pluralist environments where multiple and divergent views exist at
both the technical level and in management. Finally, complex systems cannot be
deconstructed to their constituent elements; doing so would remove the added value that
is provided when the systems integration function is undertaken as part of the entire
system development. It is a fundamental characteristic of complex systems that the
interplay of the various elements brings unique additional capability. Reducing the
complexity of a proposed system could mean foregoing the capability it offers.

Historically, there has always been a tension between the increasing complexity
of new technologies and the policy frameworks that govern and often lag behind their
development. The recent track record indicates that the governments existing
management and integration tools no longer suffice for large-scale, horizontally-
integrated complex programs. Current approaches were developed years ago in an
environment where the government customer was technically astute and worked closely
with one vertically integrated contractor per program. Today, the government customer is
less savvy in matters of technology and less well-staffed in terms of workforce.
Moreover, the contracts for a typical program are executed by a network of firms, often
spanning continents and sharing responsibility for managing cost, schedule and risk.
Companies and even governments may simultaneously be partners and competitors, and
it is a sensitive issue to even share information, much less to integrate it. Furthermore,
dividing a program into more manageable smaller components, then integrating them into
a single platform or system, is by definition not a viable option for complex programs.

Organizing for complexity: a constant struggle

The federal governments ability to bring complex programs to fruition depends

first and foremost on effective governance. In the past, great engineering successes
resulted not only from technical excellence but also from superior project management
and governance structures. Unlike smaller projects, complex programs require that many
external elements, such as the bureaucratic politics of coordinating a large number of
interlinked organizations, be internalized. Integrating external and internal elements is
part of what makes complex programs dynamic, non-linear and risk-intensive. It also
presents significant policy and governance challenges.

With changing technological and commercial environments, program

management models have evolved over the years, from the government-owned arsenal of
the 19th century to the recent Lead System Integrator (LSI) approach. Responsibility for
requirements definition, program management and technical execution has increasingly
shifted away from government and toward the private sector (see Table 1, Program
Responsibility Format Types). This trend resulted from reasonable efforts to reap the
benefits of competition in both innovation and economics.

4 / 16
 Table 1: Program Responsibility Format Types

Arsenal Contract Weapon Outsourcing Lead System

System To Private Integrator
Manager Arsenal

Program Government Government Government Government Industry

Requirements

Technical Direction Government Government Government Industry Industry

Program Management Government Government Industry Industry Industry

Technical Execution Government Industry Industry Industry Industry

External Environment Infrequent wars Some commercial Weapons become Government begins Loss of in-house
Little commercial application of more complicated / to lose in-house tech government tech
application of military tech complex capabilities capabilities leads to
military tech Private sector Coordination of Outsourcing inability to define
private sector sub-systems becomes whats possible
pays better, can be becomes important increasingly
more responsive Large companies acceptable
can better leverage
political support

Source: Harvey Sapolsky (2009) Models for Governing Large Systems Projects, in Guy Ben-Ari and Pierre Chao (eds.) Organizing for a Complex World:
Developing Tomorrows Defense and Net-Centric Systems, Washington, DC: CSIS, p. 26.

5 / 16
 However, the shift of responsibility to the private sector has been accompanied by
a decline in overall government expertise and capability. In fact, during the past two
decades, the capability and capacity of the federal government for systems integration has
been dramatically reduced. At the height of the Cold War, defense systems commands
(such as the Naval Air Systems Command or the Air Force Systems Command)
combined military, civilian, and outside personnel to build and manage large systems.
Assistance with systems-of-systems integration was the purview of research centers and
government labs. But at the end of the 1980s, the DoD began a long period of steady
downsizing of the acquisitions workforce, and the expertise to manage complex
acquisitions began to wither. The impact of the reduced staff numbers during the 1990s
was not immediately apparent, as the pace of defense procurement in the post-Cold War
world was slower and less urgent than before. The need for certification of new systems
or examination of new standards was low. As a result, many of the design engineers and
technicians employed by certifying organizations to develop and evaluate criteria for
construction and design standards retired and were not replaced.1

Recent experience confirms the difficulty of managing complex system

development programs to time and budget. Since 2000, the Department of Defense
(DoD) has significantly increased the number of major defense acquisition programs
(MDAPs) and its overall investment in them, but the track record of delivering on cost
and schedule remains uneven. For example, in a 2009 analysis of select DoD weapon
programs, the Government Accountability Office (GAO) found that for the fiscal year
2008 portfolio of MDAPs, total acquisition costs increased 25 percent and development
costs increased by 42 percent, compared to initial estimates. Both increases are greater
than the corresponding increases for programs in the fiscal year 2000 portfolio. GAO
analysis also found that on average, FY 2008 programs delivered initial capabilities to the
warfighter 22 months behind schedule, a 6-month increase compared to fiscal year 2000
programs. Continued cost growth results in less available funding for other DoD
priorities and programs, while continued failure to deliver weapon systems on time
delays providing critical capabilities to the warfighter.2

In directing programs that have been problematic, managers for the government,
the prime contractors, and the commercial subcontractors shared one common feature:
they underestimated the complexity of requirements, integration of subsystems, and the
interaction of changes in one subsystem with new demands on others.3 That is, while
programs go awry for varied reasons, problematic programs have in common their
inability to address the complexity challenge effectively.

1
Defense Science Board [DSB] Task Force on Integrating Commercial Systems into the DOD, Effectively
and Efficiently, Buying Commercial: Gaining the Cost/Schedule Benefits for Defense Systems
(Washington, DC: Department of Defense, February 2009), p. 30.
2
Government Accountability Office (2009) Defense Acquisitions: Assessments of Selected Weapon
Programs, Washington, DC: Government Accountability Office.
3
David Berteau (2009) Foreword, in Guy Ben-Ari and Pierre Chao (eds.) Organizing for a Complex
World: Developing Tomorrows Defense and Net-Centric Systems, Washington, DC: CSIS, p. ix.

6 / 16
Thinking about a solution: measuring success

As described above, complexity is first and foremost a governance and

management problem. In todays globalized knowledge economy, the speed of change in
technology and society has outpaced the ability of public policy and government
organizations to learn, adapt and respond. Despite this, governance has found ways to
individually and institutionally influence the ability to deliver military systems
successfully.

But as complexity becomes a greater challenge, such solutions are less easy to
find. Despite several efforts to identify innovative governance alternatives, there is no
known or identified method to assess any of them ex ante. Therefore, it has been difficult
for managers to compare potential solutions and to assess whether a given policy or
governance framework will have the desired effect.

Ultimately, the program-management challenge and the value brought by a

good systems-of-systems integrator lies in helping DoD make tradeoff decisions. How
can we measure this? The ability to make tradeoffs requires broad access to knowledge,
not only on technology but also on military need and relative priority, across all
potentially applicable systems and subsystems and all components and specialties.
Although this is a demanding challenge, it is true that access to knowledge can, in fact, be
measured. The number of systems and subsystems and components and specialties are
known (or at least knowable), and whether they are known can be documented and
measured with considerable precision. As such, by measuring the degree of access to
relevant information, it is possible to compare different project-management models
against one another.

This ability to measure suggests one possible approach. Stable teams of talented
scientists and engineers can be assessed in each of their access-to-knowledge categories
(systems, subsystems, components, technologies). Those measures can be both relative
(i.e., comparing DoD labs, R&D centers, and private contractors) and absolute (Do we
have enough? Is everything covered?). While this approach is input-oriented and assumes
that better access to relevant information will lead to better outcomes, it allows for the
comparing and relative ranking of competing organizations or management structures,
and it also allows comparison over time. Eventually, a baseline standard can emerge.

Based on the premise that the clash of ideas, and the evaluation of tradeoffs
among those ideas, really does lead to better solutions, the measures above may also
support an assessment of who can do a better job of systems-of-systems integration. By
measuring who has better access to knowledge, we can identify who can better foster that
clash of ideas and the corresponding tradeoffs. Such a process could move the choice of
program management structure from one of emotion and philosophy to one of analysis
and metrics. This approach offers promise for such a process.

7 / 16
Organizational flexibility and resilience: keys to a solution

Some direction can also be found by considering what must happen for a program
to meet schedule and budget objectives:

Warfighter needs must be gathered and assembled into a complete,

comprehensible specification or a Request for Proposal (RFP).

Potential bidders must be able first to understand that RFP and then to submit
bids that allow fair compensation and that share risk reasonably.

Government analysts need to compare bids, applying their expertise and

experience to identify and reject unrealistic assumptions.

End users must have input to tradeoffs across capability, schedule and budget.

Once a contract has been awarded and execution begins, design changes must be
integrated appropriately. Realistic assessments about progress and potential must be
made, and ways found to manage newly encountered tradeoffs. Unanticipated events will
happen, and success will hinge on how well the chosen program management and
governance frameworks react to the unexpected. The measurement of access to
knowledge outlined in the previous section does not help us assess an organizations
ability to respond to change; we must seek that elsewhere.

By looking at successful private sector examples, we can see some elements of a

potentially successful approach. One typical private sector approach to addressing
complexity is to improve the ability of an organization to understand and respond to
changes in its environment, by becoming a learning organization. Such organizations
can learn and adapt to changes in the environment and ultimately bring about their own
continuing transformation.4 Successful innovators in dynamic industries for example,
IBM, 3M, Goldman Sachs, and Google have institutional and organizational structures
that enable them to adapt quickly to changing commercial conditions. They can tolerate
false starts and the accompanying waste, having honed organizational characteristics that
allow them to change course quickly. They devote resources to learning whats
happening, pursue what works and abandon what does not, spend less time planning for
everything in advance, and not try to execute the plan regardless of what is learned along
the way. These attributes amount to making the organization more capable of handling
unexpected situations, by accepting that they will occur and trying to make the
organization more resilient.

Complexity entails unpredictable, rapid changes. It can be addressed by

increasing the systems flexibility and resilience (F&R), so it can successfully absorb and
react to changes, problems and opportunities. We define F&R as the ability to recognize,
4
Donald Schn (1973) Beyond the Stable State: Public and Private Learning in a Changing Society,
Harmondsworth: Penguin, p. 28.

8 / 16
survive and respond to changes. In practical terms, this means the organization must
recognize, understand and react to internal and external developments. The organization
must be acutely aware, from the lowest to the highest levels, of changes in the external
environment (user needs, the operating environment, relevant doctrine, etc.) and the
impact of those changes internally and on the program in question. An awareness of the
internal environment the details of the production and design cycle, technological or
engineering developments that might threaten budget or schedule commitments, etc. is
also necessary. Managers and employees at all levels must be empowered to
communicate their conclusions about perceived changes and the impact of those changes
and to take appropriate action to react to the changes.

Flexible and resilient programs must have management and leadership including
political overseers that is willing to tolerate a certain amount of failure, a certain
number of false starts, and spending that sometimes appears to be less than completely
efficient. Program management and the accompanying contracting process need to focus
on accountability, sometimes at the perceived sacrifice of efficiency. We contend that
coping with complexity puts a premium on flexibility, and some sacrifice of apparent
efficiency is necessary in order to get greater benefits of on-time and on-budget delivery.

Can the government create such organizations?

Successfully instituting flexibility and resilience will require significant changes

in the culture of acquisition and program-management organizations. Training will be
needed so employees understand the big picture and can make their piece of the small
picture work better. Management needs to cultivate some level of tolerance for error and
be willing to grant the necessary autonomy and authority for decision-making at lower
levels. Systems across the organization must be designed to allow widespread sharing of
information, perhaps even including sensitive data on profitability. The organization must
also gather external information and disseminate it widely, to allow the ranking and
prioritization of system attributes that allow tradeoffs to be made more easily. An effort
should be made to reduce bureaucratic barriers to efficiency, such as extremely tight
budget controls or overly stringent documentation requirements. Most importantly,
incentives should be aligned throughout the organization to encourage and reward desired
behaviors. A model might be the way the military sometimes pushes authority and
responsibility down through the hierarchy, with commanders describing their intent and
junior officers and troops having some latitude in how they attempt to achieve it, within
guidelines.

The attractiveness of F&R attributes lies in their ability to be applied regardless of

what overall management approach is chosen. Focusing on F&R in existing organizations
is a way to side-step the discussion over choosing exactly the right project management
model that would rely more on the private sector, an FFRDC, or a government laboratory
(see Box 1, Program Management Models). Traditionally, a management model would be
chosen by analyzing the project, selecting a management model, then hoping that the
choice was correct. Instead, F&R offers an approach that should work regardless of what

9 / 16
challenges are encountered, because F&R is itself is a model of adapting to complexity,
of embracing it and being ready for the pitfalls and opportunities it offers.

Box 1: Program Management Models

FFRDC
Federally funded research and development centers (FFRDCs) which sprang up in the
early years of the Cold War are specially chartered non-profit institutions that receive
long-term government contracts to conduct research. FFRDCs cannot compete for
production contracts, and the long timeframes of their contracts typically result in low
employee turnover and long institutional memory. Prominent FFRDCs in the defense
sector include MITRE, the Aerospace Corporation, RAND, the Center for Naval
Analyses and the Institute for Defense Analyses.

UARC
University affiliated research centers (UARCs) developed during World War II to
explore advanced technologies sponsored by the Department of Defense, primarily the
U.S. Navy. There are five officially designated UARCs, including the Johns Hopkins
University Applied Physics Laboratory and the Applied Research Center at the
Pennsylvania State University. Other organizations including the Institute for Soldier
Nanotechnology (ISN) at MIT also conduct UARC-like work.

Government Lab
As part of the militarys systems commands, government labs provide assistance with
systems integration. Government labs are staffed by civil service personnel, allowing for
organizational longevity and good customer understanding. Although government labs
provide quick fixes to the warfighter, this often hinders their ability to focus on long-
term research projects. Government labs are sometimes criticized for a lack of
organizational independence and an inability to provide the total package for systems
integration. Examples include the Air Force Research Lab (AFRL) and the U.S. Naval
Research Lab.

LSI
The Lead Systems Integrator (LSI) model involves a contractor or team of contractors
hired to execute large, complex, defense procurement programs. This model gives the
contractor a variety of roles: requirements generation, technology development; source
selection; construction or modification work; procurement of systems or components
from, and management of, supplier firms; testing; validation; and administration.5
Prominent programs managed by LSIs include the Coast Guards Deepwater and the
Armys Future Combat Systems (FCS).

5
Valerie Bailey Grasso (2009) Defense Acquisition: Use of Lead System Integrators (LSIs) Background,
Oversight, Issues, and Options for Congress, Washington, DC: Congressional Research Service.
http://www.fas.org/sgp/crs/natsec/RS22631.pdf

10 / 16
 Each of the three organization types possesses F&R to some degree (see Table 2,
Flexibility and Resilience under Various Governance Models). Lets look at each in turn.

First, FFRDCs and UARCs have proven that they can be flexible in managing
technical teams in dynamic environments and sustaining them over time, even when
those teams have been challenged with different types of projects that demand a wide
array of skills. UARCs and FFRDCs sustain the institutional stovepipe functions needed
to house technical knowledge and expertise. They provide the matrixed integration to
bring that array of technical knowledge and expertise to bear in a systems-of-systems
architecture approach. As a result, they can more easily provide a broader reach across
technical areas, integrating multiple disciplines under a single pursuit. This flexibility to
reach across disciplines may become more significant in the future, as overall defense
missions are clarified in new technology areas. Further, UARCs and FFRDCs have the
flexibility to attract and retain top talent.

Several elements make FFRDCs and UARCs particularly resilient: their

independence, the absence of even the appearance of conflict of interest, the protection of
proprietary information, and the provision of equal access to all potential interested and
qualified parties (public and private). FFRDCs and UARCs have a lower rate of
employee turnover, contributing to the institutions' historical memory and ability to
promise steady configuration-control procedures. However, FFRDCs, with dedicated
budget line items, while less driven to take on customers regardless of how their work fits
into institutional priorities, may become sluggish or too responsive to the expected
answer phenomena because of the line item funding. (This is less true of UARCs, with no
dedicated funding line.)

Second, government labs and engineering centers exhibit F&R to a lesser degree.
One reason for this is their link to a systems command sponsor, a relationship that
sometimes exhibits tension. The lab or center may feel that the systems command ignores
their priorities and feeds its own larger goals, while the systems command may feel that
the lab or center does the same in reverse, that is ignores systems command goals to keep
doing what the lab and its leadership consider more worthwhile. In other words, the
systems commands may see labs as less responsive and flexible. This is particularly
important if the systems commands rely on their researchers to help them be smart
buyers.

Furthermore, systems commands tend to believe that scientists should support

their immediate needs for advice on particular acquisition programs and for quick fixes to
get equipment working for upcoming deployments, even if the solutions are temporary,
non-systematic, and non-repeatable. Both of these pressures detract from in-house lab
scientists' ability to pursue long-term research projects.

On the other hand, DoD labs and centers during the past decade or more have
actively moved into new partnerships with the private sector. Driven primarily by a need

11 / 16
to find additional (non-government) business to finance their workforce and facilities,
labs have broken into new areas that are not consistent with their system commands
priorities or with the labs prior core competencies. This provides them with the potential
for flexibility.

Furthermore, because they are part of the military itself but are staffed mostly by
long-term civil servants, government labs possess resilience due to organizational
longevity and customer understanding. However, DoD labs are less successful than
FFRDCs or UARCs in attracting and retaining top talent because of the constraints of the
federal civilian employee rules and limitations.

Third, private companies have, in recent years, led many of the cutting-edge
military systems integration efforts, whether individually or in partnerships. Some efforts
have been more successful than others, but prime contractors clearly have a base of
program management experience to build on. Given that the defense business cycle is
affected by the annual political cycle of Congressional appropriations, industry has had to
develop the key attribute of flexibility if only to incorporate such considerations into their
business. Similarly, flexibility was developed in order to manage relationships with both
industry partners and sub-contractors. The ability to attract, retain and manage top talent
from relevant technical disciplines also supports this flexibility.

Private companies also possess resilience to a great extent. This stems from a high
level of customer understanding. The ability to grasp military jargon and to track various
military ideas and doctrinal initiatives is developed intensively, both through internal
training and by hiring retired military officers. Furthermore, the need to stay
commercially viable provides great incentive to find ways to adjust to changing
conditions.

12 / 16
 Table 2: Flexibility and Resilience under Various Governance Models

FFRDC/UARC Government lab or center Private contractor

Resilience Independence (incl. ability to Long-term relationships with High level of customer
verify performance) and lack of customers understanding
conflict of interest Organizational longevity
Ability to retain talent
Work on long-term contracts
Institutional memory

Flexibility Technical expertise across wide Range of collaborative efforts Ability to manage relationships
range of topics with academia and industry with customer as well as with
Ability to attract talent partners / suppliers
Ability to attract talent
Strong (financial) incentive to
adapt to changing conditions

13 / 16
Practical matters

Flexibility and resilience are clearly elements of successful management of

complexity, and they are available to the government through each of its current models.
To be effective, though, F&R needs to be increased at each of the three phases in
program development:

Requirements determination: Currently, system requirements, once formalized,

are difficult to change. The process of reaching a decision through the Joint Requirements
Oversight Council in DoD can take two years or longer, making any system manager
(even the Secretary of Defense) reluctant to raise questions that could cause that process
to be restarted. Yet, requirements should permit users and developers to be smarter today
than they were yesterday. System design goals should be adjusted accordingly. Flexibility
in requirements is necessary to promote competition and a better alignment of contracts
and resources.

Pre-award (contract preparation): The process of converting requirements into

an RFP, running the bidding and making an award must be improved. This pre-award
process is the governments way of converting requirements into a solicitation document,
then seeking bids from potential contractors. The process includes the scope of work that
will be performed by the winning bidders and the criteria for evaluating their bids. It also
includes the governments evaluation of those bids and selection of the winner or
winners. In recent years, the results of this pre-award process have been less successful
than in the past, as measured by the number of successful protests lodged by losing
bidders with GAO. In many cases, protests are upheld because of procedural flaws by the
government in solicitation, evaluation, and award of contracts. In some cases, good
decisions have been undermined by governance failures as simple as inadequate
documentation. Process failures need to be reversed as a minimum condition of success
in organizing for complex systems. Tolerance for adapting to changing conditions while
maintaining compliance with regulations requires both flexibility and resilience and will
lead to better pre-award efforts.

Post-award (contract management): The process of managing contracts

following award needs to be improved. For complex systems, this post-award process is
hard enough even with clear requirements and a pristine pre-award process, because the
tasks under contract are challenging and difficult to achieve. Yet the quality and quantity
of post-award personnel, the contract administration organizations, has been dramatically
reduced since 1990, and the process of restoring them has yet to begin. This is an area
where private sector best practices in F&R are most applicable, as government
organizations adapt to changes in technology, threats and responses.

Summary

What may be needed, therefore, is a way to tie the underlying DoD skill base to
systems-of-systems integration by connecting it to access to knowledge, both current and

14 / 16
emerging. By doing so, we may be able to address both our concerns how do we
organize for better management of complex systems, and how do we measure success.
Incorporating flexibility and resilience into the management structure will also contribute
to success.

Areas for future research

How to measure flexibility / resiliency?

o Measures of input
o Measures of output
Consider for example SEIs Capabilities Maturities Models (which evolved from
software process measures). If the CMM approach can be adapted to cover broader
systems engineering work, as is being tried today, we may evolve measures that can
encompass the entire systems-of-systems integration. Can we assess components of
a systems-of-systems integration approach to derive subsidiary measures that are
useful within the management, budgetary, or programmatic time frames?

What components could we measure? One such component could be subsystem

familiarity. Knowledge of subsystems is essential to successful systems-of-systems
integration. Surrogate sub-measures for this knowledge could be:
o the processes for training and educating engineers, scientists, and program
managers;
o the hiring of personnel from subsystem contractors;
o rotating personnel into other phases of component design and production, etc.
Somehow, these surrogate measures each seem to miss the mark, but perhaps these
together with others could address the overall issue.

How to institute flexibility / resiliency? Theres a lot of business / organizational

management literature on doing this at the organizational level, but very little on
how to do it at the policy / governance level.

Are there applicable lessons from how the private sector approaches management /
governance of complexity?

Author Biographies

David J. Berteau is senior adviser and director of the CSIS Defense-Industrial

Initiatives Group. A former director of Syracuse Universitys National Security Studies
Program, Mr. Berteau is an adjunct professor at Georgetown University, a member of
the Defense Acquisition University Board of Visitors, and a director of the Procurement
Round Table. He is a fellow of the National Academy of Public Administration. Prior
to joining CSIS, he was director of national defense and homeland security for Clark &
Weinstock, a senior vice president at Science Applications International Corporation
(SAIC), and principal deputy assistant secretary of defense for production and logistics.

15 / 16
Mr. Berteau holds a B.A. from Tulane University and a masters degree from the LBJ
School of Public Affairs at the University of Texas.

Guy Ben-Ari is a fellow with the Defense-Industrial Initiatives Group at CSIS, where
he specializes in defense technology and defense industrial policies. Before joining
CSIS, he was a research associate at the George Washington Universitys Center for
International Science and Technology Policy as well as a consultant focusing on
innovation policy and evaluation for the European Commission and the World Bank.
He is coauthor (with Gordon Adams) of Transforming European Militaries: Coalition
Operations and the Technology Gap (Routledge, 2006) and of various book chapters
and articles. He holds a masters degree in science, technology, and public policy from
the George Washington University and a bachelors degree in political science and
history from Tel Aviv University.

Matthew Zlatnik is a consultant with the Defense-Industrial Initiatives Group at CSIS,

focusing on how technological, industrial, and budgetary issues affect defense policy.
He previously spent 10 years in investment banking. Mr. Zlatnik graduated from
Carleton College, holds an M.B.A. in finance from the Wharton School, and is studying
for a masters degree in international relations at the Johns Hopkins School of
Advanced International Studies (SAIS).

16 / 16