NORTH ATLANTIC TREATY RESEARCH AND TECHNOLOGY

ORGANISATION ORGANISATION

AC/323(IST-026)TP/190 www.rto.nato.int

RTO TECHNICAL REPORT TR-IST-026

Evolutionary Software
Development
(Développement évolutionnaire
de logiciels)

Final Report of the Task Group IST-026/RTG-008.

Published August 2008

Distribution and Availability on Back Cover

NORTH ATLANTIC TREATY RESEARCH AND TECHNOLOGY
ORGANISATION ORGANISATION

AC/323(IST-026)TP/190 www.rto.nato.int

RTO TECHNICAL REPORT TR-IST-026

Evolutionary Software
Development
(Développement évolutionnaire
de logiciels)

Final Report of the Task Group IST-026/RTG-008.

 The Research and Technology
Organisation (RTO) of NATO
RTO is the single focus in NATO for Defence Research and Technology activities. Its mission is to conduct and promote
co-operative research and information exchange. The objective is to support the development and effective use of
national defence research and technology and to meet the military needs of the Alliance, to maintain a technological
lead, and to provide advice to NATO and national decision makers. The RTO performs its mission with the support of an
extensive network of national experts. It also ensures effective co-ordination with other NATO bodies involved in R&T
activities.
RTO reports both to the Military Committee of NATO and to the Conference of National Armament Directors.
It comprises a Research and Technology Board (RTB) as the highest level of national representation and the Research
and Technology Agency (RTA), a dedicated staff with its headquarters in Neuilly, near Paris, France. In order to
facilitate contacts with the military users and other NATO activities, a small part of the RTA staff is located in NATO
Headquarters in Brussels. The Brussels staff also co-ordinates RTO’s co-operation with nations in Middle and Eastern
Europe, to which RTO attaches particular importance especially as working together in the field of research is one of the
more promising areas of co-operation.
The total spectrum of R&T activities is covered by the following 7 bodies:
• AVT Applied Vehicle Technology Panel
• HFM Human Factors and Medicine Panel
• IST Information Systems Technology Panel
• NMSG NATO Modelling and Simulation Group
• SAS System Analysis and Studies Panel
• SCI Systems Concepts and Integration Panel
• SET Sensors and Electronics Technology Panel
These bodies are made up of national representatives as well as generally recognised ‘world class’ scientists. They also
provide a communication link to military users and other NATO bodies. RTO’s scientific and technological work is
carried out by Technical Teams, created for specific activities and with a specific duration. Such Technical Teams can
organise workshops, symposia, field trials, lecture series and training courses. An important function of these Technical
Teams is to ensure the continuity of the expert networks.
RTO builds upon earlier co-operation in defence research and technology as set-up under the Advisory Group for
Aerospace Research and Development (AGARD) and the Defence Research Group (DRG). AGARD and the DRG share
common roots in that they were both established at the initiative of Dr Theodore von Kármán, a leading aerospace
scientist, who early on recognised the importance of scientific support for the Allied Armed Forces. RTO is capitalising
on these common roots in order to provide the Alliance and the NATO nations with a strong scientific and technological
basis that will guarantee a solid base for the future.

The content of this publication has been reproduced

directly from material supplied by RTO or the authors.

Published August 2008

Copyright © RTO/NATO 2008

All Rights Reserved

ISBN 978-92-837-0042-5

Single copies of this publication or of a part of it may be made for individual use only. The approval of the RTA
Information Management Systems Branch is required for more than one copy to be made or an extract included in
another publication. Requests to do so should be sent to the address on the back cover.

ii RTO-TR-IST-026
 Table of Contents

Page

Executive Summary and Synthèse ES-1

Chapter 1 – Introduction, Motivation, Hypothesis 1-1

1.1 Introduction 1-1
1.2 Motivation 1-1
1.3 Hypothesis 1-2
1.4 References 1-3

Chapter 2 – Key Concepts 2-1

Chapter 3 – Supporting Architectures 3-1

3.1 The Role of Architectures 3-1
3.2 Model Driven Architecture Development 3-2
3.3 Adaptable Architectures 3-2
3.4 Dynamic Architectures 3-2
3.5 Product-Line Architectures 3-3
3.6 Commercial-Off-The-Shelf Software and Architecture: Friend or Foe 3-3
3.7 References 3-5

Chapter 4 – State of the Art / Practice 4-1

4.1 State of the Art 4-1
4.1.1 Other Workshops and Conferences 4-1
4.1.2 Tom Gilb 4-1
4.1.3 Evolutionary Procurement 4-1
4.2 Concrete Examples Taken from Symposium IST-034 RSY-010 4-2
4.3 Related Concepts 4-3
4.3.1 Prototyping 4-3
4.3.2 User-Centered Design 4-3
4.3.3 Agile Software Development 4-3
4.3.4 Achieving Specific Non-Functional Requirement Targets 4-4
4.4 References 4-4

Chapter 5 – Issues and Concerns (FAQ) 5-1

5.1 Questions and Answers 5-1
5.1.1 Why do Developers and Customers Use the Evolutionary Process? (Benefits) 5-1
5.1.1.1 Partial Functionality is Available Early 5-1

RTO-TR-IST-026 iii
 5.1.1.2 Partial Functionality Can be Achieved Before Complete Funding 5-1
5.1.1.3 Learn-From-Use Feedback Corrects Requirements 5-1
5.1.2 Why Don’t Developers and Customers Use the Evolutionary Process? 5-1
5.1.2.1 Lack of Familiarity with ESD 5-1
5.1.2.2 Concern that ESD is Not Permitted 5-2
5.1.2.3 Lack of Tool Support for ESD 5-2
5.1.3 How Can ESD be Managed? 5-2
5.1.4 How Can ESD be Taken into Account in Current Processes? 5-2
5.1.5 Cost Estimation of ESD 5-2
5.1.6 QA (Quality Assurance) and ESD 5-3
5.1.7 ESD and Business Decisions 5-3
5.1.8 Relationship of ESD to Open Source Movement 5-3
5.1.9 Relationship of ESD to the Agile Movement 5-3
5.2 Concerns 5-3
5.2.1 Evolution is Risky 5-3
5.2.2 Evolutionary Process Facilitates Decreasing Risk 5-4

Chapter 6 – Unresolved Challenges (Future Research, Possible Improvements) 6-1

6.1 Software Architecture 6-1
6.1.1 Dynamic and Adaptive Architectures 6-1
6.1.2 Product Lines 6-1
6.1.3 Service Oriented Architectures 6-2
6.2 Implications for Procurement 6-2
6.3 Cost Estimation 6-3
6.4 Project Management, Including Product and Project Metrics 6-3
6.5 Tool Support, Including Configuration Management 6-3
6.6 How to Economize in Testing 6-3
6.7 Documentation 6-4
6.8 Aids to Retrain Users 6-4
6.9 Scaffolding 6-4
6.10 COTS-Based Systems 6-4
6.11 Interoperability 6-5
6.12 Simultaneous Field Support for Multiple Releases 6-5
6.13 Automated Upgrading 6-6

Chapter 7 – Recommendations/Conclusions 7-1

7.1 ESD Should be More Widely Adopted for Long-Lived Systems 7-1
7.2 The TG-008 Web Site Should be Continued as an Active Vehicle for Information Interchange 7-1

Annex A – Definitions of Key Concepts A-1

Annex B – Links to Other Research Groups and Workshops Addressing B-1

Software Evolution

Annex C – Review of History of Task Group C-1

iv RTO-TR-IST-026
Annex D – Members of Task Group 026/RTG-008 D-1

Annex E – Iterative and Incremental Development: A Brief History E-1

RTO-TR-IST-026 v
vi RTO-TR-IST-026
 Evolutionary Software Development
(RTO-TR-IST-026)

Executive Summary
This is the final report of the task group IST-026/RTG-008.

For many years, military software development in many countries mandated a sequential predictive
development process, often characterized by US DoD MIL-STD-2167A and referred to as the waterfall
model. This is despite the fact that ever since the 50’s, some successful military software projects have
instead employed iterative development processes, and in each decade leaders of software thought have
advocated them. The situation persists even though 2167A itself was revoked in 1994 because it led to
many project failures: cost and delivery date overruns, projects abandoned before completion and products
which even if delivered were never fielded because they were too far from what the customer actually
needed. The intrinsic flaw in the waterfall process was that it did not acknowledge inherent
incompleteness of project requirements and uncertainties in available implementation technology at the
time of project initiation. Moreover, it did not acknowledge the evolution in system role, available
technology, economics or user expectations that naturally occurs during an extended development period,
never mind during a long operational system life. Software is not subject to the driver of hardware
procurement policy whereby, once delivered and deployed, changes to individual units are so expensive
that (except perhaps for minor maintenance and a possible “mid-life kicker” upgrade) the requirements
that the product meets must remain stable throughout an extended operational life. Many DCI and PCC
call for software systems that can be fielded early and can be adapted rapidly as needs evolve.

The commercial world, needing to respond quickly to business opportunities, as well as to respond to
pressures from competitive product offerings, had moved long ago to a periodic release cycle over the
whole product life. This process, is called evolutionary software development (ESD), typically involves
several releases being in planning, perhaps even development, concurrently. Successive releases can
accommodate changes in requirements. Features not complete in time slip to the next release rather than
holding up the release ship date. The Agile Manifesto, 2001, has been particularly influential.

We found that ESD is appropriate for military software development and there are success stories of its
use. However, we found that there are unresolved challenges in how best to manage ESD, to do cost
estimation for ESD, to perform quality assurance for ESD and to manage risk with ESD. Tool support for
ESD, appropriate scaffolding to facilitate ESD, methods of reducing costs associated with repeated
integration, repeated testing, repeated retraining, all are open topics for improvements. We conclude that
ESD is effective, and should be the normal process used for software that is expected to evolve.

RTO-TR-IST-026 ES - 1
 Développement évolutionnaire de logiciels
(RTO-TR-IST-026)

Synthèse
Ceci est le rapport final du groupe de travail IST-026/RTG-008.

Pendant longtemps, dans de nombreux pays, le développement de logiciels militaires a requis un processus
de développement prédictif séquentiel, souvent caractérisé par le DoD MIL-STD-2167A américain et
désigné sous le nom de modèle en cascade. Et cela en dépit du fait que, depuis les années 50, certains
projets réussis de logiciels militaires ont utilisé à la place des processus de développement itératif, et qu’à
chaque décennie, les leaders de la pensée logicielle ont prôné leur cause. Cette situation perdure, bien que
le 2167A lui-même ait été retiré en 1994 car il menait à l’échec de nombreux projets : dépassements des
coûts et des dates de livraison, projets abandonnés avant leur achèvement, ou produits qui, bien que livrés,
n’étaient jamais utilisés car trop éloignés des besoins réels du client. Le défaut intrinsèque du processus en
cascade était de ne pas prendre en compte l’inachèvement inhérent aux exigences des projets et les
incertitudes de la technologie d’implémentation disponible au moment du lancement du projet. En outre,
il ne reconnaissait pas l’évolution du rôle du système, de la technologie disponible, de l’économie ou des
attentes de l’utilisateur, qui se produisait naturellement au cours d’une période de développement
prolongée ou de la longue durée de vie d’un système opérationnel. Les logiciels ne sont pas soumis à la
politique d’acquisition des pilotes de matériel selon laquelle, une fois ces derniers livrés et déployés,
les modifications apportées aux unités individuelles sont si onéreuses que (à l’exception peut-être d’une
maintenance mineure ou d’une éventuelle mise à niveau à la moitié de sa vie) les exigences auxquelles
répond le produit doivent rester stables tout au long d’une durée de vie opérationnelle prolongée. Nombre
d’Initiatives sur les capacités de défense (DCI) et de Cellules de coordination du partenariat (PCC)
demandent des systèmes logiciels pouvant être déployés tôt et s’adapter rapidement en fonction des
besoins.

Le monde commercial, devant saisir rapidement les opportunités professionnelles tout en faisant face à la
pression des offres de produits concurrents, a depuis longtemps évolué vers un cycle de versions
périodiques tout au long de la durée de vie du produit. Ce processus, appelé développement évolutionnaire
de logiciels (ESD), implique généralement que plusieurs versions soient en projet, peut-être même en
cours de développement, simultanément. Les versions successives peuvent répondre à l’évolution des
exigences. Les fonctions qui n’ont pu être terminées à temps passent sur la version suivante, plutôt que de
retarder la date d’expédition de la version. Le Manifeste Agile de 2001 a eu beaucoup d’influence.

Nous sommes parvenus à la conclusion que l’ESD est adapté au développement de logiciels militaires.
Il existe des cas où il a déjà été employé avec succès. Toutefois, nous avons découvert qu’il restait encore
des défis à relever : déterminer la meilleure manière de gérer l’ESD, réaliser une estimation de son coût,
mettre en place une assurance qualité et gérer les risques. Un support d’outils pour l’ESD, une architecture
appropriée pour sa facilitation, des méthodes de réduction des coûts associées à une intégration répétée,
des tests fréquents, des recyclages réguliers, il s’agit là de sujets ouverts à l’amélioration. En conclusion,
l’ESD est efficace et devrait être le processus normal utilisé pour les logiciels que l’on s’attend à voir
évoluer.

ES - 2 RTO-TR-IST-026
 Chapter 1 – INTRODUCTION, MOTIVATION, HYPOTHESIS

1.1 INTRODUCTION
Military software procurement has traditionally followed a phased development process called the waterfall
model. A project moves sequentially through stages of concept, requirements elicitation, specification
definition, preliminary design, detail design, unit implementation, system integration, acceptance testing,
and deployment. Reviews between the phases provide convenient assessment points for project review, as
well as a convenient mechanism for gating progress payments to contractors.

Despite this, iterative [1] and incremental development processes have a long history going back to the
1950’s, of successful application to software systems, many military, and “in each decade has been advocated
by prominent leaders of software engineering thought” [2].

An excellent survey of this history, published in IEEE Computer June 2003” by C. Larman and V. Basili,
is attached to this report by permission as Annex E. It is also worth noting that, if read carefully, the 1970
article [3] by W. Royce generally regarded as the origin of the waterfall model can be seen as actually
advocating iterative development. Today iterative and incremental development is promoted as an essential
aspect of Agile Programming, a rapidly growing trend in software development process [4].

MIL-STD-2167A of the US DoD was widely regarded as prescribing a development process based on the
waterfall model, and the failures ascribed to this model lead to that standard being replaced in December
1994. Nevertheless, more than a decade later, waterfall development of military software is still prevalent in
many nations. In the meantime, commercial practice has heavily shifted to a software development process
called Evolutionary Software Development.

1.2 MOTIVATION
Many DCI (Defence Capability Initiatives) [5] and PCC (Prague Capabilities Commitment) [6] describe
systems which require software that can be fielded early in the lifecycle of the systems, and can be readily
adapted as those systems evolve. The traditional development process has a poor record in meeting these
objectives, so a different development process is called for.

The procurement agencies of many nations follow a deliberate policy that alternate suppliers should be
eligible for follow-on programs. This carries over from hardware, where the functionality and performance of
a product is largely fixed at initial delivery, and on-going maintenance is considered a low level activity of
repairs, preventative precautions, and minor enhancements. The assumption is that these activities require
different resources than initial development, and so should be open to alternate suppliers. Again, following
hardware practice, a major redevelopment effort may be undertaken after a system has been field for a few
years, a so-called “mid-life kicker”. The assumption is that this redevelopment might benefit from fresh ideas,
so should be open to alternate suppliers.

Experience with software systems, however, is that software support often needs to be quite different than
this. Changing requirements, the availability of new technology, and rising user expectations result in software
support involving frequent and substantial modifications to fielded software systems. Ramping up a new team
each time to learn the system in sufficient depth of understanding to be able to make these changes effectively
is time-consuming and expensive. For example, when a series of weapons has been developed over time,

RTO-TR-IST-026 1-1
INTRODUCTION, MOTIVATION, HYPOTHESIS

each offering exactly the same functionality, this policy has been responsible for development of software
over the entire series at excessive cost, not just in fiscal terms. The intermediate position, that the developer
should be responsible for a planned sequence of releases, has not even been considered.

The Tomahawk missile is but one of many examples of a system where repurposing over the system lifetime
has resulted in decades of almost continuous software development.

1.3 HYPOTHESIS
In contrast to the conventional delivery of a product in a single “big bang”, the Evolutionary Software
Development process delivers a product over a s planned sequence or releases based on a “learn from experience
in use” cycle. This philosophy arises in response to the following hypotheses:
1) Early fielding of partial functionality is generally better than no deployment until full functionality is
confirmed. Funding or staffing uncertainty can delay full functionality indefinitely, and a working
system with some functionality is usually better than none. It follows that incremental delivery may
happen over several releases.
2) Requirements are never completely defined (large and complex systems, customers have trouble
specifying needs, new technology triggers new requirements, ...). Since incremental delivery only
attempts to support partial functionality in each release, new requirements may be accommodated in
subsequent releases.
3) Incremental delivery by itself is not sufficient, because deployed functionality may have to be
withdrawn if requirements are dropped, or if conflicting new requirements take precedence.
Experience with partial functionality is often a source of changes in requirements.
4) Obsolescence of hardware, software, laws, and regulations causes requirements to be withdrawn
over the product lifecycle, perhaps even during initial development.
5) Environment of the system is changing all the time (technology, politics, procedures, mission,
tactics, etc.). A system that is not matched to the current environment may no longer be useful.
6) Civil and military systems are sometimes perceived to have very different requirements (specifically
the military point of view regarding evolutionary system development). In fact this perceived
difference is often better characterized as a difference in point of view between commercial versus
governmental (long-term systems). The difference often derives from the procurement process,
i.e. open tendering, gated progress payments, etc.
7) The customers are often not themselves the end-users. Indeed, for many systems the actual end-users
are unknown or at least unavailable during procurement and initial development. This forces
feedback options that might not be chosen when known and identifiable users are available.
8) Implementation issues are not necessarily well understood at the time that development is initiated,
with the consequence: that some exploratory developments are needed, recognizing that unsuccessful
directions that should be abandoned must be expected.
9) Time and finance constraints often do not allow you to develop the system you ideally would like to
have, nevertheless you would like to get a part of the system that can later be extended rather than
something you will have to throw away. This has significant consequences for the procurement
process and for the development process, especially for project management.

1-2 RTO-TR-IST-026
 INTRODUCTION, MOTIVATION, HYPOTHESIS

10) End-users are typically able to criticize/invalidate models and prototypes, but not to specify
requirements precisely in advance, nor to validate before a fully working system is available.
11) Multiple releases of a system imply continuous integration and testing. This has the consequence: of
accentuating process improvement.

Note that because Evolutionary Software Development addresses changes over the full life cycle of a system,
it is not quite the same as either the spiral model or the agile process, which both typically use iteration during
the initial development process but then may revert to a conventional maintenance and enhancement lifecycle.

1.4 REFERENCES
[1] Luckey, P.H., Pittman, R.M. and LeVan, A.Q. (1992). “Iterative Development Process with Proposed
Applications”, Technical Report, IBM Owego, New York.

[2] Larmen, C. and Basili, V.R. “Iterative and incremental development: a brief history” IEEE Computer
June 2003, pp. 2-11.

[3] Royce, W.W. (1970). “Managing the development of large software systems: Concepts and Techniques”,
Proc. WESCON, IEEE Computer Society Press, Los Alamitos, CA. Reprinted at the ICSE’87, Monterey,
California, USA. March 30 – April 2, 1987.

[4] The Agile Manifesto, Snowbird, UT, February 11 – 13, 2001 http://agilemanifesto.org/

[5] NATO Factsheet 9 August 2000 http://www.nato.int/docu/facts/2000/nato-dci.htm

[6] A reader’s Guide to the Prague Summit and NATO’s Transformation.

RTO-TR-IST-026 1-3
INTRODUCTION, MOTIVATION, HYPOTHESIS

1-4 RTO-TR-IST-026
 Chapter 2 – KEY CONCEPTS

This report is a report on Evolutionary Software Development1. This phrase can be interpreted in two ways,
which both exist equally widespread in literature:
1) The Evolutionary Development of software, and
2) The development of Evolutionary Software.

The first interpretation considers evolutionary software development to be a Software Development Process,
and a special instantiation of Iterative Development. Iterative development has been around successfully for
a long time [2], but has been overshadowed for a long period of time by the Waterfall Model, which has a
purely sequential character, although the original description of the waterfall model did include iteration [3].

In this report, Evolutionary Software Development is interpreted in the second way and is defined as the:
Development of (Systems and) Software, able to evolve with little effort to meet changing user needs,
to interact with changing environments, and to benefit from emerging new technology.

The crucial difference between these two interpretations is that the first interpretation regards only software
development, resulting in a delivery of a final product for installation, albeit with intermediate partial releases.
The second interpretation regards the entire Software Lifecycle and treats software as a living entity, whose
life does not end when the “final” development product is released, but rather starts with the initial release and
continues as long as the software is in use. And during that lifetime, the software will grow and evolve in
response to changes in requirements, in technology, in hardware, and in other parts of the environment of the
software product. Evolutionary Software Development according to the second interpretation distinguishes
itself from other development models by already planning for these changes during the development of the
initial product.

To phrase it differently:
In ESD there is no such thing as a “final” product, other than the state the product is in when it is taken
out of operation at the end of its lifetime.

Although the evolutionary development approach has been embraced as a good thing by most of the software
engineering research community, warnings have been issued [4]:
The difficulty [with the evolutionary development model] is to distinguish it from code-and-fix models,
whose spaghetti code and lack of planning were the initial motivation for the waterfall model. It is also
based on the often-unrealistic assumption that the operational system will be flexible enough to
accommodate unplanned evolution paths. This assumption is unjustified in three primary circumstances:
1) Circumstances in which several independently evolved applications must be closely integrated,
2) “Information-sclerosis” cases, in which temporary work-arounds for software deficiencies
increasingly solidify into unchangeable requirements on evolution, and
3) Bridging situations, in which the new software is incrementally replacing a large existing system.
If the existing system is poorly modularised, it is difficult to provide a good sequence of bridges
between the old software and the expanding increments of the new software.

1
Key concepts are given in Bold when introduced for the first time in this section. Definitions of these terms are given in Annex A.

RTO-TR-IST-026 2-1
KEY CONCEPTS

Under such conditions, evolutionary development projects have come to grief by pursuing stages in the
wrong order: evolving a lot of hard-to-change code before addressing long-range architectural and
usage considerations.

These warnings have not been ignored and large interest has grown in the area of Software Architectures to
accommodate for software evolution over the lifetime of a product. A software architecture is the fundamental
organisation of software embodied in its components, their relationships to each other and to the environment
and the principles guiding its design and evolution. An overview of the different attempts on architectures for
evolution and the evolution of architectures themselves is given in Chapter 3.

[1] http://www.cs.dal.ca/ESD.

[2] Larman, C. and Basili, V.R. Iterative and Incremental Development: A Brief History, cover feature of
IEEE Computer, June 2003.

[3] Royce, W. Managing the Development of Large Software Systems, Proceedings Westcon, IEEE CS
Press, 1970, pp. 328-339.

[4] Boehm, B. A Spiral Model of Software Development and Enhancement, IEEE Computer, May 1988,
pp. 61-72.

2-2 RTO-TR-IST-026
 Chapter 3 – SUPPORTING ARCHITECTURES

In this section, software architectures that support evolution are described. First of all, the role software
architectures play in today’s software development is discussed. Then, a number of different types of
architectures addressing the management of change (evolution) are discussed. Finally, a section is devoted to the
implications on architecture and evolution of using Commercial-Off-The-Shelf (COTS) components in software
product development.

3.1 THE ROLE OF ARCHITECTURES

In [1], an overview of software architecture is given, including “three fundamental reasons why software
architecture is important, and architecture-based development is worthwhile:
1) Mutual Communication. Software architecture represents a common high-level abstraction of the
system that most, if not all, of the system’s stakeholders can use as a basis for creating mutual
understanding, forming consensus, and communicating with each other.
2) Early Design Decisions. Software architecture represents the embodiment of the earliest set of design
decisions about a system, and these early bindings carry weight far out of proportion to their
individual gravity with respect to the system’s remaining development, its service in deployment,
and its maintenance life.
3) Transferable Abstraction of a System. Software architecture embodies a relatively small,
intellectually graspable model for how the system is structured and how its components work
together; this model is transferable across systems; in particular, it can be applied to other systems
exhibiting similar requirements, and can promote large scale reuse.”

For discussion on support of architectures for evolutionary software development, the second reason is the
most relevant one. More specifically, the early design decisions and the resulting architecture provide a basis
for reasoning about, and management of, change. Deciding when changes are essential, determining which
change paths have least risk, assessing the consequences of proposed changes, and arbitrating sequences and
priorities for requested changes all require broad insight into relationships, dependencies, performance,
and behavioural aspects of software components. Reasoning at an architecture level can provide the insight
necessary to make decisions and plans related to change.

Furthermore, a system architecture provides the structure of the system in components and relations between
components. Changes to the system may either affect only a single component (local change), or affect
multiple components, or, in the worst case, affect the underlying structure between components.
An architectural change that affects the way in which components interact with each other will probably
require changes all over the system. Therefore, the architecture plays a crucial part in anticipating change by
structuring the system in components in such a way that the most likely changes will be local to a single
component.

The discussion above shows the importance of having a good architecture in place early in the development
process. This has been widely recognised in the software engineering community and has led to the
introduction of numerous “architecture first” development methods, usually referred to as architecture-based
development methods. Architecture-based development starts with understanding the domain requirements
[1]. Since a primary advantage of architectures is the insight they lend across whole families of systems, some
extra effort should be spent on studying the requirements, not only for the current system, but also for the

RTO-TR-IST-026 3-1
SUPPORTING ARCHITECTURES

whole family of systems the current system is or will be a part of. These families can be a set of related
systems, all fielded simultaneously, but with small differences, or a single system that exists in many versions
over time, each differing from each other by small changes. The domain analysis should therefore investigate
the requirements as well as anticipate, enumerate and record changes, variations, and similarities.

In the remainder of this section, architectural principles and example architectures that specifically aim at
reducing the impact of change will be discussed.

3.2 MODEL DRIVEN ARCHITECTURE DEVELOPMENT

A few years ago, the Object Management Group, known for the specification of the Common Object Request
Broker Architecture CORBA, introduced the concept of Model Driven Architecture (MDA) [2].
MDA attempts to separate the logic of a system from the platform it is actually implemented on. MDA is
organised around a Platform-Independent Model (PIM), which is a specification of the system in terms of
domain concepts. These domain concepts are to a degree independent of different platforms of similar type
(e.g., CORBA, .NET, and J2EE). The PIM can then be compiled towards any of those platforms by
transforming the PIM to a Platform Specific Model (PSM). The PSM specifies how the system uses a
particular type of platform.

This approach particularly addresses the evolution of the platform (hardware and operating system) and tries
to minimise the impact of such changes on the overall software. Instead of having to re-design and re-
implement the entire software, only the PSM and the transformation from the PIM to the new PSM have to be
developed.

3.3 ADAPTABLE ARCHITECTURES

Adaptable is defined in many different ways, but the general consensus seems to be that a system is called
adaptable if it can easily be changed. In order to ensure that the software system finally developed exhibits
adaptability, this non-functional requirement should be considered during the development of the architecture.

In [3], numerous adaptation techniques are mentioned, categorised into the following: architecture-based
techniques, component-based techniques, code-based techniques, genetic algorithm techniques, dynamic
adaptation techniques, and adaptation methodologies.

These categories differ from each other in the phase of the software lifecycle (including operation and
maintenance) in which the adaptation itself takes place. But they all agree on the fact that the structure to
allow adaptation has to be designed, or at least considered, during the initial development of the software
architecture.

A comprehensive adaptation technique in the architecture-based category is described in [4]. In this technique,
an adaptable system has two managers – one for adaptation and one for evolution. In an iterative process, the
adaptation manager takes high-level decisions, which are implemented by the evolution manager.

3.4 DYNAMIC ARCHITECTURES

Dynamic architectures [4,5] allow a system to adopt its behaviour at run-time, either by selecting alternatives
within the system (constrained dynamism), or accepting new modules at run-time (unconstrained dynamism).
The latter is often referred to as plug-and-play.

3-2 RTO-TR-IST-026
 SUPPORTING ARCHITECTURES

In case of constrained dynamism, all possible changes must be known a priori. By means of techniques such
as parameterised instantiation of elements, conditional reconnection, and modification event handling,
the architecture may be changed at run-time, but only in ways that have been designed into the architecture
(hence the term “constrained” dynamism). The advantage of this approach is that system integrity is preserved
because all run-time options may be tested before operation.

“Unconstrained” architectural dynamism allows “any” change in principle, but the validity of these changes
must be ensured at run-time. Changes may include addition of components, removal of components,
replacement of components, or re-configuration. Frequently used design patterns in these types of
architectures are the observer pattern and the mediator pattern [6].

3.5 PRODUCT-LINE ARCHITECTURES

Development of software product lines [7,8] relies heavily on the use of variability to manage the differences
between similar products (often referred to as a product family) by delaying design decisions to later stages in
the development and usage of the constructed software. There are different variability realisation techniques
depending on the binding time (e.g., architecture derivation, compilation, linking, and run-time) and the
involved software entities (e.g., frameworks, components, classes, lines-of-code).

A taxonomy of variability realisation techniques is given in [9]. This taxonomy includes the following
mechanisms:
• Inheritance: different or extended implementations of methods;
• Extension points: additional behaviour or functionality of components;
• Parameterisation: selection of component behaviour at build-time, or run-time;
• Configuration: selection or de-selection of components as-a-whole at build-time, or run-time; and
• Generation: components are generated from a higher-level language in which change may be
expressed more easily.

An attempt to use the Model Driven Architecture from Section 3.2 as an approach towards variability
management is described in [10].

The research into product-line architectures and the management of change is ongoing and new directions still
arise. But, in general, the trend in variability seems to be towards two directions:
1) Addressing variability more and more in software instead of hardware, and
2) Preparing for variation earlier in the development process to postpone the selection of a single variant
as long as possible into run-time to allow the user more flexibility.

3.6 COMMERCIAL-OFF-THE-SHELF SOFTWARE AND ARCHITECTURE:

FRIEND OR FOE
Today, many operational systems and systems under construction contain a large number of Commercial-Off-
The-Shelf (COTS) software components. Reasons for using COTS are mainly [11]:
• Cost savings (development and maintenance costs spread over many customers),

RTO-TR-IST-026 3-3
SUPPORTING ARCHITECTURES

• Better products (more feedback),

• Lower training costs, and
• Faster integration of new technologies and new standards.

However, the use of COTS is not without risk. In the same report [11] as well as in [12], a long list of risks
that come with COTS is given:
• Loss of Control: COTS tools are marketed by a vendor according to the schedule of the vendor,
and may be discontinued at any time by the vendor. The vendor may even go out of business. License
agreements may change over time, and costs for maintenance and upgrades may increase
unanticipated.
• Lack of Understanding: Developers do not have access to source code, complete and correct
behavioural specifications may not be available, and analysis and testing must be done in a black box
manner. COTS products may not meet non-functional requirements such as availability, integrity,
and security. This deficiency of information increases the possibility of introducing design errors.
• Frequent Updates: Commercial products are continuously being upgraded and customers are often
required to upgrade in order to fix bugs, or in order to keep receiving maintenance and support.
Replacing a component may be time-consuming because of regression testing and development of
new workarounds, changes in integration software, or necessary updates of hardware.
• COTS components may not operate well together, because of lacking conformance to standard
interfaces and exchange formats.
• Troubleshooting: When the system fails, the component that causes the failure must be determined.
As mentioned before, COTS are usually black boxes and determining which component causes a
failure may therefore be difficult. Furthermore, the exact circumstances under which a component
fails must be determined to provide to the support organisation and the priority given to this problem
by the vendor may be low.
• COTS Products Come with their own Architectures: These may not match the intended
architecture of the system as a whole. Therefore, the COTS product may dictate some aspects of the
system architecture and limit alternatives, or the system architecture may limit the choice of COTS
products you can consider.

Because of these potential risks, several researchers have proposed extra development activities and
architectural solutions to mitigate these risks [11,13]:
• Experience with (combinations of) COTS products early to gain better understanding;
• Evaluate alternative architectures to identify weaknesses or deficiencies in COTS products;
• Architecture Migration: Design an architecture in such a way that a transition to an architecture
without a certain COTS product, or with a different (version of the same) COTS product, is easy; and
• Evaluate the proposed architecture against the characteristics of evolution to identify the ability of
the architecture to change. In [14], such characteristics, called Evolutionary Characteristics Of
Architecture (ECOA), are described.

Most of these solutions try to embed the COTS in the architecture in such a way that changes (upgrading a
COTS component, or replacing a COTS component by another) can be made more easily. These solutions

3-4 RTO-TR-IST-026
 SUPPORTING ARCHITECTURES

may also be used for integrating in-house developed components. Because of the emphasis of these solutions
on managing change, they are also useful to be considered for use in architectures for evolving systems, even
if they do not use COTS components.

A discussion of COTS cannot be closed without referring to the Open Source movement. In recent years,
open source software has become popular as a basis for developing products.

Although the use of open source software mitigates the first two risks mentioned at the beginning of this
section, the other risks still apply. The proposed solutions in this paragraph therefore also apply to product
development based on open source components.

3.7 REFERENCES
[1] Clemens, P.C. and Northrop, L.M. Software Architecture: An Executive Overview, Technical Report of
the Software Engineering Institute, CMU/SEI-96-TR-003, February 1996.

[2] MDA Guide v1.0, OMG, 2003 (http://www.omg.org).

[3] Subramanian, N. and Chung, L. Software Architecture Adaptability: An NFR Approach, Proceedings of
the International Workshop on Principles of Software Evolution (IWPSE 2001), ACM Press, Vienna,
September, 2001, ACM Press, pp. 52-61.

[4] Oreizy, P., et al., An Architecture-Based Approach to Self-Adaptive Software, IEEE Intelligent Systems,
May/June 1999, pp. 54-62.

[5] Oreizy, P. Dynamic Software Architecture Resources Web pages, http://www.ics.uci.edu/~peymano/

dynamci-arch/.

[6] Gamma, E., Helm, R., Johnson, R. and Vlissides, J. Design Patterns, Addison-Wesley, 1995.

[7] Lai, C.T.R. and Weiss, D.M. Software Product-Line Engineering: A Family Based Software
Development Process, Addison-Wesley, 1999.

[8] Clements, P. and Northrop, L. Software Product Lines: Practices and Patterns, Addison-Wesley, 2001.

[9] Svahnberg, M., van Gurp, J. and Bosch, J. A Taxonomy of variability realization techniques, technical
paper ISSN: 1103-1581, Blekinge Institute of Technology, Sweden 2002.

[10] Deelstra, S., Sinnema, M., van Gurp, J. and Bosch, J. Model Driven Architecture as Approach to
Manage Variability in Software Product Families.

[11] Vigder, M. An Architecture for COTS Based Software Systems, NRC Report No. 41603, National
Research Council of Canada, 1998.

[12] Davis, R. The Role of Architecture in Managing COTS Based High Integrity Systems, Presentation from
the Ground Systems Architecture Workshop, March 2003, http://sunset.usc.edu/gsaw/gsaw2003/s8b/
davis.pdf.

RTO-TR-IST-026 3-5
SUPPORTING ARCHITECTURES

[13] Carnegie Mellon Software Engineering Institute Architecture Lessons: http://www.sei.cmu.edu/cbs/

architecture/lessons.htm.

[14] Davis, L., Payton, J. and Gamble, R. Toward Identifying the Impact of COTS Evolution on Integrated
Systems, Paper of the COTS Workshop on Continuing Collaborations for Successful COTS
Development, Held in conjunction with ICSE 2000, Limerick, Ireland, June 4 – 5, 2000,
http://wwwsel.iit.nrc.ca/projects/cots/icse2000wkshp/.

3-6 RTO-TR-IST-026
 Chapter 4 – STATE OF THE ART / PRACTICE

4.1 STATE OF THE ART

Symposium IST-034 RSY-010 Technology for Evolutionary Software Development was held in Bonn,
23 – 24 September 2002. Key issues from the papers in this symposium were:
• The concept of Evolutionary Software Development is not well understood, perhaps not even agreed
upon. Many papers in the Symposium described practices that do not meet the definition used here.
• Experience reporting is not systematic or consistent, making comparison difficult.
• There is a lack of training in Evolutionary Software Development.
• There is no NATO standard for Evolutionary Software Development.

4.1.1 Other Workshops and Conferences

There have been many other workshops and conferences where the subject of software evolution and how to
cope with it has been discussed. Two of the most important workshops held annually since 1998 and 2002
respectively have been IWPSE (International Workshop on Principles of Software Evolution) and USE
(International Workshop Unanticipated Software Evolution). Although papers at these workshops do not
always refer to Evolutionary Software Development, the examples and theories that they discuss do lead to an
increased understanding of software evolution, and the problems that Evolutionary Software Development is
trying to address. Mention of this topic would be incomplete without reference to Lehman and Belady’s Laws
of Software Evolution that were originally put forward in the 1970’s, and updated in Project FEAST in the
1990’s. These laws, especially the second law, are still highly controversial.

4.1.2 Tom Gilb

One of the original advocates of Evolutionary Software Development is the internationally known consultant
from Norway, Tom Gilb. He served as the technical rapporteur for Symposium IST-034 RSY-010, he put on a
pro bono course on evolutionary development for the IST Panel, and a CD-ROM of material collected by him
on evolutionary development was distributed to every member of the IST Panel. His website www.gilb.com is
an important source of tutorial and reference material, as well as examples.

4.1.3 Evolutionary Procurement

Without the acquiescence, indeed more than that, the support and encouragement by procurement practices,
Evolutionary Software Development would only be a theory. These practices have been called Evolutionary
Acquisition or Evolutionary Procurement. Evolutionary procurement implies Evolutionary Software
Development. There are many obvious challenges for evolutionary procurement compared to traditional
procurement processes. Competitive bidding to select a supplier or choose options requires different processes
when the requirements are known only in broad outline and the ability to respond to changes must be an
essential criterion. Contracting a project in separate phases each subject to competitive bidding is an option,
but delays in approval and risk of loss of continuity of staff are countervailing arguments. How to gate
progress payments in the absence of conventional milestone reviews requires new metrics of progress. Testing
and evaluation no longer is an isolated end-of-project, perhaps even post-delivery event but must become an

RTO-TR-IST-026 4-1
STATE OF THE ART / PRACTICE

integral activity of each release cycle. Early delivery of partial functionality is only of value if the user
organization actually exploits it by at least subjecting it to operational testing and preferably fielding that
increment. To avoid the increment remaining shelfware implies a significant commitment by the user
organization and even individual users. An excellent discussion of these and other issues is contained in
the report DSTO-TR-0481 http://www.dsto.defence.gov.au/publications/2095/ of the Australian Defence
Organization, published in 1997 by Derek E. Henderson and Andrew P. Gabb [1].

Evolutionary procurement is the preferred practice for procurement of software intensive systems by NC3A
http://www.afcea.org/signal/articles/anmviewer.asp?a=720&z=7. To promote the practice and to advance the
technology, NC3A sponsored the conference Evolutionary Procurement of Information Systems (EPIS ‘90) in
1990, followed by EPIS 2000 in 2001. Several noteworthy NATO systems, including NATO ICC (Integrated
Command and Control), Cronos (eventually to become ACCIS, Automated Command and Control
Information System) and ADAMS (Allied Deployment and Movement System), were acquired by
evolutionary procurement, The US DoD has also endorsed Evolutionary Acquisition http://www.dtic.mil/whs/
directives/corres/html/50002.htm. Progressive Acquisition is a further refinement of Evolutionary Acquisition
endorsed by WEAG (Western European Armaments Group) TA (Technical Area) 13 http://sesam.tranet.
fmv.se/dokumentation/rapporter/rapporterna/FD2-1.doc.

4.2 CONCRETE EXAMPLES TAKEN FROM SYMPOSIUM IST-034 RSY-010

Two of the papers from Symposium IST-034 RSY-010 discussed the application of Evolutionary Software
development to concrete examples of systems. These examples help understanding of the role that evolution
can play in software development and throughout the lifetime of software.

François B. J. de Laender, in Toward an Evolutionary Strategy of Developing a Software Product Line in the
Field of Airport Support Systems, describes the effort NLR is making to bring together all of its civil airport
support systems into a software product line. A software product line is a set of software products that share a
managed collection of resources. The shared resources, called ASAP (Airport Scenario Analysis Platform),
are currently under development. Existing products must evolve to use these shared resources instead of,
as currently, each implementing things in its own unique way. New products will use the shared resources ab
initio. However it must be expected that the shared resources will themselves evolve, both in that the design
and implementation of a particular resource will change over time, and in that the set of resources to be shared
may be augmented by additional resources or may be reduced by resources being dropped.

Capt Roberto Ing. Ambra and Ing. Fabio Ruta, in The Evolutionary Software Development Process used in the
Upgraded AMX Human Machine Interface Design, describe the process by which the Human Computer
Interface in the cockpit of the AMX aircraft was evolved to accommodate new weapons and navigational
systems. This is particularly interesting because a predictive elicitation of requirements by itself is insufficient
for Human-Computer Interfaces. The requirements must b validated by experimental evaluation with pilots
actually working with proposed new interfaces. In all, thirteen prototypes were needed to explore the different
aspects of the interface that were changed. Concurrent engineering was critical to developing the new
interface on an acceptable schedule. The project faced both management and contractual challenges in using
this unfamiliar process.

4-2 RTO-TR-IST-026
 STATE OF THE ART / PRACTICE

4.3 RELATED CONCEPTS

4.3.1 Prototyping
Prototyping is commonly used in engineering design as a way of answering specific design questions,
by producing a working system model that can be tried and studied. These questions may be, as in the
example above, about requirements. They may instead be questions about implementation alternatives.
They may also be operational questions about the system once deployed. Sometimes the questions can be
answered economically by mock-ups or other technology unrelated to the production system. On the other
hand, sometimes the economic way to answer the questions is by creating a modification of an existing
production system. Often, once the questions have been answered, the prototype is a throw-away. On the other
hand, sometimes the prototype needs to be preserved, in case similar questions arise in the future. In many
situations, the investment in creating the prototype is substantial, and it makes sense to consider evolving the
prototype into the production system.

Thus prototyping is not necessarily associated with evolutionary software development, but it can be.

4.3.2 User-Centered Design

From the Wikipedia (free on-line encyclopedia):
“In broad terms, user-centered design (UCD) is a design philosophy and a process in which the needs,
wants, and limitations of the end user of an interface or document are given extensive attention at each
stage of the design process. User-centered design can be characterized as a multi-stage problem solving
process that not only requires designers to analyze and foresee how users are likely to use an interface,
but to test the validity of their assumptions with regards to user behaviour in real world tests with actual
users. Such testing is necessary as it is often very difficult for the designers of an interface to understand
intuitively what a first-time user of their design experiences, and what each user’s learning curve may
look like.
The chief difference from other interface design philosophies is that user-centered design tries to bend
and structure the functioning of a user interface around how people can, want or need to work, rather
than the opposite way around.”

Thus user-centered design is typically iterative, and addresses both functionality and the way that functionality
is experienced by the end-user through the human-computer interface. It typically evolves as each issue is
resolved. However, it can differ from evolutionary software development in that it focuses on the design stage
of a product, and as such may not address the entire life of that product once deployed.

4.3.3 Agile Software Development

Agile software development is an iterative approach to software development that attempts to minimize risk
by using very short duration timeboxes for each iteration, typically one to four weeks. Close interaction with
the customer during or between iteration is key.

The Agile Manifesto cites:

We emphasize:
Individuals and interactions over processes and tools
Working software over comprehensive documentation

RTO-TR-IST-026 4-3
STATE OF THE ART / PRACTICE

Customer collaboration over contract negotiation

Responding to change over following a plan

Each of the various different agile methodologies incorporates additional practices into its methodology,
but the philosophy is a common thread through all. Experience to date has been that any agile methodology
can be very effective for software development, especially fir small projects.

Agile methodologies only focus on initial development, and not on the whole life cycle of the software
product, and so differ from evolutionary software development.

4.3.4 Achieving Specific Non-Functional Requirement Targets

A key difference between functional and non-functional requirements is that whereas functional requirements
can often be proved to have been met by appropriate design, for typical non-functional requirements (such as
performance, availability, scalability, or interoperability) with specific targets, design can only be shown to
have intended to meet the criteria. For such non-functional requirements, the real satisfaction of criteria must
be demonstrated after the system is in operation. This often leads to iteration, where in system test the targets
are found not to be met, and tuning, adjustments, and even re-implementation of critical components must be
repeatedly tried until the targets are achieved. During operation of the fielded system, changing operational
conditions may well upset the balance so carefully reached, so that if the specific target is of ongoing
importance, the system may periodically require retuning. A concrete example of this situation was a post-
mission sonar reprocessing system, which had a non-functional requirement that in order to keep up with
presented load, it to operate at three times real time. That capability could only be ascertained once the system
was deployed, and over time, changes in sonar processing algorithms and changes in behaviour of sonar
targets required retuning of system parameters.

Once again, this does result in iterative development, but it is not the planned and scheduled succession of
releases that constitute evolutionary software development.

4.4 REFERENCES
[1] Henderson, D.E. and Gabb, A.P. (1997). “Using Evolutionary Acquisition for the Procurement of
Complex Systems”, Australian Defence Organization, Report DSTO-TR-0481 http://www.dsto.defence.
gov.au/publications/2095/.

4-4 RTO-TR-IST-026
 Chapter 5 – ISSUES AND CONCERNS (FAQ)

5.1 QUESTIONS AND ANSWERS

5.1.1 Why do Developers and Customers Use the Evolutionary Process? (Benefits)

5.1.1.1 Partial Functionality is Available Early

The most common attraction to customers of ESD is that early releases of the software are fully deployable,
albeit with only a subset of the ultimate functionality. Partial functionality is usually more useful than none at
all. The customers can thus get on with at least part of their work, often months earlier than they would be
able to if they had to wait for completion of the full functionality. One typical use of partial functionality is
training, not just for the operators of the system, but also for others whose activities are affected by the
system. Another important use of a system with partial functionality is where it will be used with other
existent systems in a system of systems. Use of that system of systems is often inhibited until the existent
systems have at least something to interoperate with that can take on the role that the system under
construction is intended to fulfill. Developers are attracted to ESD when it leads to customer satisfaction.

5.1.1.2 Partial Functionality Can be Achieved Before Complete Funding

Both in the commercial world and in the military world, funding for the complete system is often not available
immediately when the need, and the scope, for the complete system are identified. If meaningful increments of
partial functionality can be identified that deliver significant value, and can be funded sequentially, then the
system can be built over time as funds become available. The successful deployment of the previous
increments often assists in raising the funding to develop successive increments.

5.1.1.3 Learn-From-Use Feedback Corrects Requirements

A different attraction of ESD arises from the recognition that initial requirements documents are often flawed,
and consequently that specifications derived from them are therefore also flawed. This is particularly true for
novel applications, where due to lack of experience with the application, detailed requirements follow from
analysis of hypothetical situations that may not turn out to be what happens in practice. Users typically are
much better at recognizing what is wrong with a system that they can work with than they are at predicting
whether a system described in a design document will satisfactorily meet their needs. Consequently,
producing a system with partial functionality and then trying to use it in a production environment can be the
most effective way to refine requirements documents to identify the system that should have been built,
particularly when the system is then revised in keeping with this increased understanding. Both the customers
and the developers are happier when the customers are satisfied. In the long run, even the cost of rework to
accommodate improved understanding of requirements is more than compensated for by the better acceptance
of a system that meets the right requirements, and the greater effectiveness in its use.

5.1.2 Why Don’t Developers and Customers Use the Evolutionary Process?

5.1.2.1 Lack of Familiarity with ESD

The most common explanation for ESD not being used is that customers and developers are often unfamiliar
with the process. This situation is made worse by the fact that popular software engineering and project

RTO-TR-IST-026 5-1
ISSUES AND CONCERNS (FAQ)

management textbooks and courses still rarely mention iterative development processes, never mind drawing
the distinctions among them. A particular concern is that because the process is apparently novel, it may
impose risks in the management of the development activity. Questions such as how to measure progress do
not have self-evident answers.

5.1.2.2 Concern that ESD is Not Permitted

Many customers believe that the procurement policies of their organization do not permit iterative
development. Many developers of software under contract believe that the contract forbids iterative
development processes. While this may be true, frequently it is not. MIL-STD-498 was explicitly introduced
to encourage iterative development; ISO/IEC-12207 was carefully worded not to prohibit iterative
development. Yet both standards were widely interpreted as requiring waterfall development, as MIL-STD-
2167A had been. However some developers had even found it possible to do iterative development under
2167A.

5.1.2.3 Lack of Tool Support for ESD

Modern software development relies intensively on interactive development environments (IDEs),
configuration management systems and other tools. Project management has relied for decades on Gantt
charts, PERT charts, CPM, Earned Value Analysis and similar tools. These tools are often thought to be
incompatible with ESD, and tools explicitly to support ESD are rare.

5.1.3 How Can ESD be Managed?

Between customers and developers, prioritizing features to incorporate in the next few releases rarely appears
difficult. Concurrent engineering practices, such as workspace based configuration management and nightly
build ensure that development proceeds in a way such that there is always a version of the software ready to
ship: fixed release dates or funding constraints on releases can always be met by slipping functionality still in
progress to subsequent releases. Including documentation. installation aids and other ancillary artifacts under
configuration management ensures they match the software in the release. Unrolling the development iteration
for the next few releases so that the work items in each appear explicitly in the Work Breakdown structure
(WBS) means standard project management tools can be used in almost conventional ways.

5.1.4 How Can ESD be Taken into Account in Current Processes?

The obvious answer for most processes is to consider interim releases before the current final delivery,
or subsequent releases upgrading the product after the current planned final delivery. The former might
provide increased value within the current funding envelope; the latter might provide a way of generating
value after the completion of the current funded activity. If the current process is already an iterative one,
such as Agile methodology, then taking EDS into account might involve carrying occasional iterations
through to operational use, not merely demonstration.

5.1.5 Cost Estimation of ESD

Formalized cost estimation methods, such as COCOMO or function points, estimate cost using predictors that
are functions of sizing parameters of the project with coefficients that are statistically determined form
experience with similar projects. The same kinds of predictors are equally appropriate for ESD, although
similar projects being different, the coefficients and the accuracy of prediction can be expected to be different.

5-2 RTO-TR-IST-026
 ISSUES AND CONCERNS (FAQ)

The more serious challenge is that estimation of per release cost corresponds to upgrade cost estimation,
which traditional cost estimation does not do well. It is unclear what are appropriate factors to consider,
or what constitute similar projects. In practice this means that cost estimates will likely be based on judgment
and experience, as they usually are.

5.1.6 QA (Quality Assurance) and ESD

Regression testing is essential with ESD, as it is with any process that leads to many releases of a product.
The efficacy of ensuring changes in this release did not impact things that should not have been affected is
less critical than avoiding the embarrassment of reintroducing bugs already solved. To the extent that the
customer perception of quality includes responsiveness of the supplier to newly recognized needs, software
with an ESD life cycle can be seen as higher quality than software upgraded through a more cumbersome
process.

5.1.7 ESD and Business Decisions

ESD permits early fielding of partial functionality. This facilitates revenue generation before the product is
complete, creates a presence in the market before the product is complete, and can even be crucial in attracting
investor as well as customer support in a speculative product. The machinery for adapting to changing
requirements provides a vehicle for responding to pressure from competitors with somewhat different feature
sets.

5.1.8 Relationship of ESD to Open Source Movement

ESD is as relevant to the Open Source movement as it is to more conventional contract software development.
In fact it may even be more relevant, because an Open Source project must attract and maintain a volunteer
group of developers, and the success of early releases can be a large factor in sustaining and growing
commitment.

5.1.9 Relationship of ESD to the Agile Movement

ESD and Agile development are both iterative processes, and sometimes might be synonymous. The first
difference is that Agile development iterations may only involve demonstrating the product to the customer,
whereas ESD iterations involve deploying the product in production. The second difference, not unrelated to
the first, is that whereas Agile iterations occur every week or two, ESD iterations might occur only every
several months. Finally, Agile development may only involve iterations until the product is delivered, whereas
ESD iterations continue until the product is retired.

5.2 CONCERNS

5.2.1 Evolution is Risky

The whole point of evolution is that it facilitates change as the driving factors change over time. What that
means, of course, is that at project initiation you can’t predict the form of the product many cycles out.
Some people see that as a concern – some people see it as a strength. It depends how wisely the evolution is
guided.

RTO-TR-IST-026 5-3
ISSUES AND CONCERNS (FAQ)

5.2.2 Evolutionary Process Facilitates Decreasing Risk

One principle that can be used for steering evolution across individual releases or across multiple releases is
that the changes implemented should reduce risk. This evaluates the evolution in terms of how well the risks
were understood and appreciated, and how wisely the judgments were made and implemented.

5-4 RTO-TR-IST-026
 Chapter 6 – UNRESOLVED CHALLENGES (FUTURE
RESEARCH, POSSIBLE IMPROVEMENTS)

6.1 SOFTWARE ARCHITECTURE

Software architecture, the structure of a software-intensive system, plays a huge role in how amenable the
system is to evolutionary development. This software structure comprises the software elements,
the externally visible properties of those elements, and the relationships among them. Every software system
has an architecture, whether or not it was explicitly designed. Software architecture has been recognized as a
discipline for about 15 years and many aspects have been studied.

Nevertheless, little work has been done yet on understanding what architectural aspects assist or hinder
evolution, and hence how to choose an architecture for a system which is known to need to support evolution.
Indeed, little work has yet been done in understanding how to use specific architectural features to facilitate
evolutionary development. Future research is needed to resolve this. Some particular such features are the
following:

6.1.1 Dynamic and Adaptive Architectures

Dynamic architectures are architectures that change over time. Not only are changes of which components are
connected to which supported, but the nature of the components can change, and so can the nature of the
connectors. The study of a dynamic architecture includes the details of how a transition takes place, as will as
an understanding of when different transitions are permitted. Evolution of a software architecture can often be
modeled with a dynamic architecture.

Adaptive architectures are a different way of addressing a somewhat similar situation. Rather than making the
changes in the architecture explicit, an adaptive architecture hides the changes at an architectural level by
defining components and connectors in such a way that changes affect the nature or usage of each component
and connector, but not the connections that are established between components. The individual components
and connectors are said to “adapt” to evolving circumstances, and the way that adaptation can take place is
studied. As an example, rather than defining point-to-point connectors between components, an adaptive
architecture might connect all the components with a bus, and then examine the traffic on that bus changes
with demand. Evolution of a software architecture can therefore also sometimes be modeled with an adaptive
architecture.

Examples of dynamic architectures and of adaptive architectures have appeared in the literature, illustrating
how they can be used. There is a need for more study of real systems undergoing typical evolution, framing
these systems as dynamic or adaptive architectures to see how these formulations can assist in analysis,
understanding and design of systems that evolve.

6.1.2 Product Lines

One of the hot topics in software architecture in recent years has been the study of product lines. A product
line is defined as a set of products that share a managed set of common resources. The study of product lines
includes not just how to best exploit the commonalities, but how to manage the common resources. Often a
product line is thought of as a set of products that exist at the same time, but it can also be useful to think of
versions of a product that exist over time as forming a product line, with the shared resources being the

RTO-TR-IST-026 6-1
UNRESOLVED CHALLENGES (FUTURE
RESEARCH, POSSIBLE IMPROVEMENTS)

software components, documentation, training material etc. that are re-used from one version of the product to
the next. Although this perspective has been noted, there is as yet no literature illustrating how this
observation should be used to manage the re-use activities that make up a labor-intensive part of the
evolutionary software development process.

Research into thinking of product evolution as a product line would be particularly natural in the case of
products that are highly configurable, that is, that already exist in many versions simultaneously. A material
management system that needs to be configured differently for each base or ship on which it is used,
for instance, might be a naturally subject of study for how the configuration mechanism could be used to
accommodate upgrades to respond to changes in operational procedures.

6.1.3 Service Oriented Architectures

One very popular software architecture style stimulated by the worldwide web has been Service Oriented
Architectures. Such architectures decouple the usual association between a component requiring some service
and the component providing that service by delaying the binding until run-time, and by doing the binding
with any server on any computer that can provide a suitable service. This fits well with the commercial
realities of the web, where there are many suppliers competing to provide essentially equivalent services,
and where the choice of which service to use depends on price, convenience, responsiveness, relationship
sustainability, and numerous other business factors. Similar factors can occasionally affect relationships for
support of military services. Coalition operations, for instance, involve interoperability issues whereby on
different occasions different partner nations provide essential services, and the integration of complex systems
of systems from independently developed and independently operated subsystems is a critical necessity.
What is more germane, however, is that even within systems fielded by a single nation, the effect of evolution
is that components must be replaced or upgraded, and that changing requirements and expectations require
connections amongst components to change. Service oriented architectures provide a standard way to think
about this, as well as standard mechanisms to accomplish it.

Experience with service-oriented architectures on the web is growing day by day. Research is needed,
however, to see how to exploit this experience to see how to use it to build better military systems that can be
more responsive to change.

6.2 IMPLICATIONS FOR PROCUREMENT

The issues for future research discussed above are essentially technical issues addressing how to carry out the
process of evolutionary software development. However, as pointed out in earlier sections, for military
systems evolutionary software development can only be practiced when permitted by the procurement
process. In many nations, the procurement process appears to actively discourage taking into consideration the
entire life cycle of a system, or even just a large multi-year portion of it. Often this is an unintended
consequence of other policies, policies that in themselves seem reasonable and even desirable. Limited budget
horizon, competitive bidding for successive contract phases, no rolling contracts, no bridging funding between
contracts among others are all procurement policies that have some justification, but make it difficult for a
contractor to sustain a team to work on the product over many releases, let alone to plan several releases
ahead.

Despite this, project officers working with procurement agencies in many nations have found ways to conduct
projects with evolutionary software development. More experience with novel approaches to procurement and

6-2 RTO-TR-IST-026
 UNRESOLVED CHALLENGES (FUTURE
RESEARCH, POSSIBLE IMPROVEMENTS)

contracting needs to be shared, both to promote successes with the process, and to suggest approaches that
others might try.

6.3 COST ESTIMATION

Cost estimation is necessary for developers bidding on RFPs, for project proponents attempting to secure
budgets, for project managers monitoring burn rates, and others. Unfortunately, despite attempts over the
years to automate cost estimation via tools such as COCOMO or function points, it remains true that the most
reliable cost estimation is still the judgment of experienced managers. This is particularly true for projects that
are experimental, novel, and where expectations are evolving rapidly – exactly the kind of projects for which
evolutionary software development is best suited. Despite the disappointing progress in cost estimation in the
past, renewed efforts to develop tools for cost estimation are needed. Iterative processes would seem to be
particularly difficult to estimate, because the number of iterations is often unknown. However, when project
pricing is on a fixed price basis, instead of time and materials or cost plus, project goals are continually
adjusted to stay within time and money budgets, which makes the overall project cost much more predictable.

6.4 PROJECT MANAGEMENT, INCLUDING PRODUCT AND PROJECT

METRICS
Standard project management tools, from CPM, PERT Earned Value Analysis and Gantt Charts through to
common software metrics and their analysis, are not designed to work with iterative processes such as
evolutionary software development. Unorthodox usage such as unrolling the next few iterations, can provide
workarounds to overcome some of the shortcomings, but other deficiencies, such as support for moving work
breakdown structure activities between releases are still poorly served. Some of the Agile methodologies have
introduced new tools, such as burndown charts, to support their needs in project management. There is great
opportunity for creativity in this area.

6.5 TOOL SUPPORT, INCLUDING CONFIGURATION MANAGEMENT

Practical software development and support today depends heavily on tools, from Interactive Development
Environments (IDE) and Configuration Management Systems to Program Understanding and Reverse
Engineering tools. As evolutionary software development continues into later releases, it is increasingly
important that new work integrates well with what is already there, and as corporate and individual memory
fades, this is only possible with tool support. A well structured repository of source code and other
development artifacts can simplify this, but it is not unusual for evolution to proceed in unanticipated
direction, necessitating refactoring of repository and artifacts. Tools to support this have typically been
produced from a re-engineering perspective, but the evolutionary software development need is somewhat
different, as in that case several releases are often under development concurrently. Advances in tools would
be very beneficial.

6.6 HOW TO ECONOMIZE IN TESTING

The traditional approach to testing new releases of software is to apply two different kinds of tests: functional
tests to confirm that the new functionality is working as intended, and regression tests to confirm that things
that were not supposed to change did not. This is fine as far as it goes. However, there are many aspects of the

RTO-TR-IST-026 6-3
UNRESOLVED CHALLENGES (FUTURE
RESEARCH, POSSIBLE IMPROVEMENTS)

software system that it does not test, from security to performance to robustness. Furthermore, for the testing
it does do, is that testing done effectively in terms of time, cost and effort? Is that testing being done
effectively in terms of sensitivity to what it is intending to detect? Current industry practice leaves great
opportunities for improvement, but even theoretical approaches have many unexplored gaps.

6.7 DOCUMENTATION
Classic paper documentation, especially MIL-STD-2167A style documentation, is notorious for being
incorrect and obsolete – and cumbersome and expensive to update. In a world that is actively intending to
support evolution, it is essential to do better. Web-based documentation, subject to configuration management
and exploiting hyperlinks and multimedia, increasingly is being used as a vehicle to facilitate better
documentation. Increased reliance on search engines has reduced the enormous overhead of restructuring
documentation as usage and expectations change. Keeping documentation current and relevant to all
stakeholders is still a Herculean task, and one for which there are few exemplars, much less guidelines and
tools to assist documenters.

6.8 AIDS TO RETRAIN USERS

Related to the last point about documentation is that there are really two purposes for documentation: for
training novices and for reference by old-timers. The challenge for old-timers with a rapidly evolving system
is that what they know and don’t think they need to look up may no longer be correct. The challenge of how to
retrain users without insulting them or wasting their time is not insignificant. Today most people learn to use
software systems not from manuals but from experimenting and on-line help. Correspondingly, they re-learn
changed software systems best not from update documents, but from on-line wizards that watch what they do,
and draw their attention to actions indicating that they have yet to assimilate some procedural revision.
Excellent examples of this kind of re-education exist, but there is as yet little assistance for someone
responsible for producing such wizards.

6.9 SCAFFOLDING
The delivered software system is only part of the software built in constructing that deliverable. Additional
software, specific to that product, has had to be built in order to generate parts of it, prepare data for it,
configure it, migrate existing repositories to be compatible with it, compare alternatives, install it and carry
out many other tasks that affect, but are not part of, the deliverable. This supporting software is called
scaffolding. Some of these tasks are not simply one-time efforts, but recur at least for every release of the
software. With luck, the same scaffolding can be re-used each time it is needed. It is not uncommon, on the
other hand, to find that as the product itself evolves, there must be corresponding evolution in some of the
scaffolding. Today that secondary evolution is normally manual. It would be preferable by far if we knew how
to generate both the change in the system and the change in the scaffolding automatically from the same
change action.

6.10 COTS-BASED SYSTEMS

Today very few systems are built from a blank piece of paper: most systems employ pre-existing components.
These pre-existing components form a platform on which the system is built, a platform whose characteristics
can permeate the resulting product. When the components in this platform are obtained from a commercial off

6-4 RTO-TR-IST-026
 UNRESOLVED CHALLENGES (FUTURE
RESEARCH, POSSIBLE IMPROVEMENTS)

the shelf (COTS) source, an important characteristic is that the COTS component has a life of its own.
The military customer is rarely a significant part of the COTS components supplier’s marketplace.
The military customer must thus accept fixes and enhancements if and when the commercial marketplace
motivates the supplier to make them. The military customer must also accommodate new features and
revisions entirely unrelated to what is needed for the component in his system. Postponing adoption of the
updated component is only sometimes possible, and then not always a wise choice. The ultimate such revision
is when the supplier discontinues support for the component, and the system can only continue to operate if
the component is replaced in the platform by a different nearly equivalent one.

This constant, uncontrolled revision of the platform is familiar to all computer users through the impact of
major components such as operating systems, compilers, databases, networking protocols, email systems,
etc. Unfortunately, it occurs with all third party components whatever scale, whether commercial or Open
Source. Portability at one time was often thought of as being an issue of adapting to different hardware
vendors. Today this constantly changing software platform is a much bigger issue. The “write once run
anywhere” slogan of Java, purportedly achieved by standardization, does not deal with this problem, and to
build long-lived systems, the domain of evolutionary software development, will require new technology,
or perhaps a return to some of the older technologies for coping with portability challenges that have recently
fallen out of fashion.

6.11 INTEROPERABILITY
Military systems today can no longer be traditional stovepipes – to be useful they need to interoperate with
many other systems. This is particularly a challenge because regardless of any evolution that the system itself
may need to accommodate its changing requirements, it needs to change to accommodate the continuing
churn of change in the systems it interoperates with. Some examples of military systems cite that as hundreds
of systems, each changing in its own way on its own time schedule. Just tracking the changes is a horrendous
task. Experience even trying just to synchronize the updates has shown the hopelessness of such a
coordination approach.

Fortunately, we have before us the experience with the commercial Internet, which while not perfect has
shown that it can be viable to interoperate between systems operated by independent uncoordinated
organizations. The secret is a service-oriented architecture based on flexible delayed binding protocols such as
SOAP. Communicating objects self-identify and negotiate interface requirements via XML descriptions.
If a communicating partner changes, the interface must be re-negotiated. With numerous success stories
evident in e-commerce on the web, and Microsoft .Net demonstrating that such an approach works in
corporations, we need more examples of Net-centric military systems taking advantage of this approach.

6.12 SIMULTANEOUS FIELD SUPPORT FOR MULTIPLE RELEASES

One of the critical consequences of the evolutionary development process is that several releases of software
are fielded in a short time. A consequence of this is that unless all users of the system can be forced to upgrade
simultaneously, at any time there will be multiple releases of the software operating in the field. There are
many valid reasons for individual sites to postpone upgrades, so a plethora of versions of the software running
at any time is the normal situation. At the very least this means that communications between instances of the
software, or communication between interoperating systems and instances of this software, need to be tagged
as to what version of the software is involved, and inconsistencies need to be negotiated. The complications
go well beyond this initial step. From exception reporting to user interactions to configuration, any thing that

RTO-TR-IST-026 6-5
UNRESOLVED CHALLENGES (FUTURE
RESEARCH, POSSIBLE IMPROVEMENTS)

can be different between versions needs to be handled. Masking all differences is an illusory goal – after all,
the software evolved exactly in order to realize those differences. We need examples of how this can be done
and still leave operations to be manageable.

6.13 AUTOMATED UPGRADING

In recent years one approach some suppliers have taken to minimizing the disparity between systems found in
the field is to automatically update systems that are permanently on-line. Microsoft and Apple are among the
suppliers who do this for their operating systems, but many other vendors do it for any other products they
supply. Applying updates automatically can reduce the investment in deploying updates for the supplier,
and can reduce the cost of performing upgrades for the system operator, but it is fraught with risks and
discouraged by many users, Ways need to be found to reduce the risks and increase the enticement of this
potentially valuable practice.

6-6 RTO-TR-IST-026
 Chapter 7 – RECOMMENDATIONS/CONCLUSIONS

Task Group IST-026/RTG-008 has reached two conclusions to summarize its work.

7.1 ESD SHOULD BE MORE WIDELY ADOPTED FOR LONG-LIVED SYSTEMS

Evolutionary Software Development is a successful technique to solve a very real need shared by many
military systems, and its widespread adoption should be encouraged.

7.2 THE TG-008 WEB SITE SHOULD BE CONTINUED AS AN ACTIVE

VEHICLE FOR INFORMATION INTERCHANGE
Many stakeholders in the development of military systems are currently unfamiliar with the Evolutionary
Development process. This often includes the technical development staff themselves. Moreover, there is and
will continue to be progress made in improving the practice. Consequently, it would be a valuable
contribution to have a single web site that would be regarded as a source of up-to-date information about
Evolutionary software Development. For this reason, the task group web site will be restructured for improved
information interchange on the topic, and maintained indefinitely.

RTO-TR-IST-026 7-1
RECOMMENDATIONS/CONCLUSIONS

7-2 RTO-TR-IST-026
 Annex A – DEFINITIONS OF KEY CONCEPTS

In this Annex, definitions of key concepts used in Chapter 2 are given.

Software Development Process – The process by which user needs are translated into a software product.
The process involves translating user needs into software requirements, transforming the software
requirements into design, implementing the design in code, testing the code, and sometimes, installing and
checking out the software for operational use.

Software Lifecycle Process – The process governing the entire lifecycle of a software product,
from development of an initial release through maintenance and subsequent releases to out-of-service.
Includes Software Development Process.

Sequential Development – A general term for software development processes in which requirements
definition, design, implementation, testing, and installation phases are executed in a sequential order, usually
with an official review and approval of the products of one phase before moving into the next, and with little
or no overlap between phases, and little or no iteration over phase boundaries. Contrast with: Iterative
Development.

Iterative Development – A general term for software development processes in which the requirements
definition, design, implementation, and testing occur in an overlapping, iterative (rather than sequential)
manner, with user feedback at the end of each iteration, providing input for the next. Contrast with:
Sequential Development.

Incremental Development – A software development process in which the requirements definition, design,
implementation, and testing occur in an overlapping, iterative (rather than sequential) manner, resulting in
incremental completion of the overall software product. Requirements are more or less fixed at the start,
each increment delivers more and more parts of the final system. Example of: Iterative Development,
Contrast with: Waterfall Model.

Incremental Delivery – A software lifecycle process in which not only is the software development is
conducted by incremental development, but moreover as each increment is completed it is immediately
delivered to the customer and fielded. This makes for early delivery of partial functionality. However,
it assumes that functionality once deployed is not subsequently withdrawn.

Evolutionary Software Development – The development of (Systems and) Software, able to evolve with
little effort to meet changing user needs, to interact with changing environments, and to benefit from emerging
new technology.

Waterfall Model – A model of a software development process in which the constituent activities, typically
a concept phase, requirements phase, design phase, implementation phase, test phase, and installation and
checkout phase, are performed in that order, possibly with overlap but with little or no iteration. Example of:
Sequential Development, Contrast with: Incremental development; Rapid prototyping; Spiral model.

Spiral Model – A model of a software development process in which the constituent activities, typically
requirements analysis, preliminary and detailed design, coding, integration, and testing, are performed in
iterations until the software is complete. The system is only delivered at the end of the whole process.
Example of: Iterative Development, Contrast with: Waterfall Model.

RTO-TR-IST-026 A-1
ANNEX A – DEFINITIONS OF KEY CONCEPTS

Architecture [IEEE1471 2000] – The fundamental organisation of a system embodied in its components,
their relationships to each other and to the environment and the principles guiding its design and evolution.

Software Architecture (see Architecture) – The fundamental organisation of software embodied in its
components, their relationships to each other and to the environment and the principles guiding its design and
evolution.

A-2 RTO-TR-IST-026
 Annex B – LINKS TO OTHER RESEARCH GROUPS AND
WORKSHOPS ADDRESSING SOFTWARE EVOLUTION

1) http://swerl.tudelft.nl/, web site of The Software Evolution Research Laboratory of the Technical University
of Delft in The Netherlands. This research group aims at finding the principles, concepts, tools and methods
that are needed to keep software flexible and adjustable to ever changing business and technological
requirements.
“A software system that is actually used is a living, dynamic entity, which is subjected to continuous
change.”

2) http://www.rug.nl/informatica/onderzoek/programmas/softwareEngineering/, the Software Engineering

and Architecture (SEARCH) group at the University of Groningen. The SEARCH group was founded in
September 2000 when Jan Bosch was appointed as a Software Engineering professor at the University of
Groningen. Before that he had been leading the RISE (research in software engineering) group at the
University of Karlskrona/Ronneby (http://www.ipd.hk-r.se/). The focus of the SEARCH group is on
software architecture assessment and design, software product lines, software architecture and usability
and software reuse.

3) http://www.rcost.unisannio.it/iwpse2005/, International Workshop on Principles of Software Evolution

(IWPSE). Topics of this workshop include theory of evolution, architecture for evolution, evolution of
architecture, software process for evolution, evolution of software process, methodology for evolutionary
design and development, and many more. Workshops have been held at:
2005 Lisbon, Portugal
2004 Kyoto, Japan
2003 Helsinki, Finland
2002 Orlando, Florida, USA
2001 Vienna, Austria
2000 Kanazawa, Japan
1999 Fukuoka, Japan
1998 Kyoto, Japan

4) http://esecfse05.di.fct.unl.pt/, European Software Engineering Conference (ESEC, biannual) in

co-operation with the International Symposium on Foundations of Software Engineering (FSE). During
interleaved years, the FSE is held in USA together with the ACM SIGSOFT (Special Interest Group on
Software Engineering, http://www.acm.org/sigsoft/) conference. These general software engineering-
conferences also include tutorials, workshops and paper sessions on evolution and architectures.
Past conferences:
2005 Lisbon, Portugal, ESEC/FSE-’05
2004 Newport Beach, California, USA, ACM SIGSOFT/FSE-12
2003 Helsinki, Finland, ESEC/FSE-11
2002 Charleston, South Carolina, USA, ACM SIGSOFT/FSE-10
2001 Vienna, Austria, ESEC/FSE-9
2000 San Diego, California, USA, ACM SIGSOFT/FSE-8

RTO-TR-IST-026 B-1
ANNEX B – LINKS TO OTHER RESEARCH GROUPS
AND WORKSHOPS ADDRESSING SOFTWARE EVOLUTION

B-2 RTO-TR-IST-026
 Annex C – REVIEW OF HISTORY OF TASK GROUP

The idea for an Exploratory Team on Evolutionary Software Development was first proposed in the second
IST Panel meeting in 1998. Following initial preliminary discussions over the Internet and in a short meeting
by interested parties, a presentation was given to the Panel during its Fall 1999 meeting which was widely
accepted. A sufficient number of Panel members supported the formation of a Task Group on the subject for
more detailed and substantial work.

The Exploratory Team that preceded the Task Group met in Brussels in 2000.

The Task Group itself held meetings in:

1) Istanbul, Turkey, October 2000
2) Paris, France, March 2001
3) Quebec City, Canada, May 2001
4) Warsaw, Poland, October 2001
5) Estoril, Portugal, May 2002
6) Bonn, Germany, September 2002 (Symposium IST-034/RSY-10)
7) Toulouse, France, January 2003
8) Chester, UK, April 2003
9) Prague, Czech Republic, October 2003

RTO-TR-IST-026 C-1
ANNEX C – REVIEW OF HISTORY OF TASK GROUP

C-2 RTO-TR-IST-026
 Annex D – MEMBERS OF TASK GROUP 026/RTG-008

TASK GROUP CHAIRMAN

CANADA
Dr. W. Morven GENTLEMAN
Global Information Networking Institute
Dalhousie University
6050 University Avenue
Halifax, Nova Scotia B3H 1W5
Email: [email protected]
Tel. +1 (902) 494-2652
Fax. +1 (902) 492-1517

TASK GROUP MEMBERS

CZECH REPUBLIC
Dr. Milan SNAJDER
Military Technical Institute of Electronics
Pod Vodovodem 2
15801 PRAGUE 5
Email: [email protected]
Tel. +420 (2) 20.20.77.31
Fax. +420 (2) 57.21.60.54

FRANCE
Dr. Michel LEMOINE
ONERA/DPRS/SAE
2, avenue Edouard Belin
B.P. 4025
31055 Toulouse Cedex 4
Email: [email protected]
Tel. +33 (0) 5.62.25.26.45
Fax. +33 (0) 5.62.25.25.93

HUNGARY
Mr. Attila SAPO
Computer and Automation
Research Institute of the Hungarian Academy of Sciences
PO Box 63.H 1518 Budapest
Email: [email protected]
Tel. +36 (1) 466.56.44 Ext. 417
Fax. +36 (1) 466.75.03

RTO-TR-IST-026 D-1
ANNEX D – MEMBERS OF TASK GROUP 026/RTG-008

NETHERLANDS
Mr. Yves Van de VIJVER
National Aerospace Laboratory
Anthony Fokkerweg 2
PO Box 90502
1006 BM Amsterdam

POLAND
Capt. Ryszard RUGALA
R&D Marine Technology Centre
Dickmana 62
81-109 Gdynia
Email: [email protected]
Tel. +48 (58) 666.53.40

Capt. Dr. Grzegorz BLIZNIUK

Military University of Technology
Kaliskiego 2
00-908 Warsaw
Email: [email protected]
Tel. +48 (22) 685.71.28
Fax. +48 (22) 685.78.58

PORTUGAL
Mr. Helder Antonio De CAMPOS DORES
Army Software Center
Lisbon
Email: [email protected]

Mr. Antonio RITO SILVA

INESC – Software Engineering Group
Technical University of Lisbon
Lisbon
Email: [email protected]

SLOVAK REPUBLIC
Ir. Luboslav LACKO
Military Technology Institute
ul. Kpt. Nalepku
03101 Lipovsky Mikulas
Email: [email protected]
Tel. +421 (849) 552.56.37
Fax. +421 (849) 552.56.37

D-2 RTO-TR-IST-026
 ANNEX D – MEMBERS OF TASK GROUP 026/RTG-008

TURKEY
Dr. Fuat INCE
ISIK University
Büyükdere Cad
80670 Maslak
Istanbul
Email: [email protected]
Tel. +90 (212) 286.29.60 Ext. 22.51
Fax. +90 (212) 285. 28.75

The active participants, who attended almost every meeting and regularly conducted business electronically,
were Gentleman, Snajder, Lemoine, Van de Vijver, Rugala and Lacko.

RTO-TR-IST-026 D-3
ANNEX D – MEMBERS OF TASK GROUP 026/RTG-008

D-4 RTO-TR-IST-026
 COVER FEATURE

Iterative and Incremental

Development:
A Brief History
Although many view iterative and incremental development as a modern
practice, its application dates as far back as the mid-1950s. Prominent
software-engineering thought leaders from each succeeding decade
supported IID practices, and many large projects used them successfully.

Craig

A
s agile methods become more popular, opment” merely for rework, in modern agile meth-
Larman some view iterative, evolutionary, and ods the term implies not just revisiting work, but
Valtech incremental software development—a also evolutionary advancement—a usage that dates
cornerstone of these methods—as the from at least 1968.
Victor R. “modern” replacement of the waterfall
Basili model, but its practiced and published roots go back PRE-1970
University of decades. Of course, many software-engineering stu- IID grew from the 1930s work of Walter
Maryland dents are aware of this, yet surprisingly, some com- Shewhart,1 a quality expert at Bell Labs who pro-
mercial and government organizations still are not. posed a series of short “plan-do-study-act” (PDSA)
This description of projects and individual con- cycles for quality improvement. Starting in the
tributions provides compelling evidence of iterative 1940s, quality guru W. Edwards Deming began
and incremental development’s (IID’s) long exis- vigorously promoting PDSA, which he later
tence. Many examples come from the 1970s and described in 1982 in Out of the Crisis.2 Tom Gilb3
1980s—the most active but least known part of and Richard Zultner4 also explored PDSA applica-
IID’s history. We are mindful that the idea of IID tion to software development in later works.
came independently from countless unnamed pro- The X-15 hypersonic jet was a milestone 1950s
jects and the contributions of thousands and that project applying IID,5 and the practice was consid-
this list is merely representative. We do not mean ered a major contribution to the X-15’s success.
this article to diminish the unsung importance of Although the X-15 was not a software project, it is
other IID contributors. noteworthy because some personnel—and hence,
We chose a chronology of IID projects and IID experience—seeded NASA’s early 1960s Project
approaches rather than a deep comparative analy- Mercury, which did apply IID in software. In addi-
sis. The methods varied in such aspects as iteration tion, some Project Mercury personnel seeded the
length and the use of time boxing. Some attempted IBM Federal Systems Division (FSD), another early
significant up-front specification work followed by IID proponent.
incremental time-boxed development, while others Project Mercury ran with very short (half-day)
were more classically evolutionary and feedback iterations that were time boxed. The development
driven. Despite their differences, however, all the team conducted a technical review of all changes,
approaches had a common theme—to avoid a sin- and, interestingly, applied the Extreme Pro-
gle-pass sequential, document-driven, gated-step gramming practice of test-first development, plan-
approach. ning and writing tests before each micro-increment.
Finally, a note about our terminology: Although They also practiced top-down development with
some prefer to reserve the phrase “iterative devel- stubs.

2 Computer Published by the IEEE Computer Society 0018-9162/03/$17.00 © 2003 IEEE

 The recollections of Gerald M. Weinberg, who Another 1960s reference comes from
worked on the project, provide a window into some Robert Glass:8 “We were doing
practices during this period. In a personal commu-
nication, he wrote: It is the opinion of the author that incremen-
incremental
tal development is worthwhile, [it] leads to development as
We were doing incremental development as early a more thorough system shakedown, avoids early as 1957, in
as 1957, in Los Angeles, under the direction of implementer and management discourage- Los Angeles, under
Bernie Dimsdale [at IBM’s Service Bureau ment.
the direction of
Corporation]. He was a colleague of John von
Neumann, so perhaps he learned it there, or THE SEVENTIES Bernie Dimsdale
assumed it as totally natural. I do remember Herb In his well-known 1970 article, “Manag- [at IBM’s
Jacobs (primarily, though we all participated) ing the Development of Large Software Service Bureau
developing a large simulation for Motorola, where Systems,” Winston Royce shared his opin- Corporation].”
the technique used was, as far as I can tell, indis- ions on what would become known as the
tinguishable from XP. waterfall model, expressed within the con-
When much of the same team was reassembled straints of government contracting at that
in Washington, DC in 1958 to develop Project time.9 Many—incorrectly—view Royce’s paper as
Mercury, we had our own machine and the new the paragon of single-pass waterfall. In reality, he
Share Operating System, whose symbolic modifi- recommended an approach somewhat different
cation and assembly allowed us to build the system than what has devolved into today’s waterfall con-
incrementally, which we did, with great success. cept, with its strict sequence of requirements analy-
Project Mercury was the seed bed out of which sis, design, and development phases. Indeed,
grew the IBM Federal Systems Division. Thus, that Royce’s recommendation was to do it twice:
division started with a history and tradition of
incremental development. If the computer program in question is being
All of us, as far as I can remember, thought developed for the first time, arrange matters so
waterfalling of a huge project was rather stupid, that the version finally delivered to the customer
or at least ignorant of the realities… I think what for operational deployment is actually the second
the waterfall description did for us was make version insofar as critical design/operations areas
us realize that we were doing something else, are concerned.
something unnamed except for “software devel-
opment.” Royce further suggested that a 30-month project
might have a 10-month pilot model and justified
The earliest reference we found that specifically its necessity when the project contains novel ele-
focused on describing and recommending iterative ments and unknown factors (hardly a unique case).
development was a 1968 report from Brian Randell Thus, we see hints of iterative development, feed-
and F.W. Zurcher at the IBM T.J. Watson Research back, and adaptation in Royce’s article. This iter-
Center.6 M.M. Lehman later described Randell and ative feedback-based step has been lost in most
Zurcher’s work and again promoted iterative devel- descriptions of this model, although it is clearly not
opment in his September 1969 internal report to classic IID.
IBM management on development recommenda- What did Royce think about the waterfall ver-
tions:7 sus IID when he learned of the latter approach? In
a personal communication, Walker Royce, his son
The basic approach recognizes the futility of sep- and a contributor to popular IID methods in the
arating design, evaluation, and documentation 1990s, said this of his father and the paper:
processes in software-system design. The design
process is structured by an expanding model He was always a proponent of iterative, incre-
seeded by a formal definition of the system, which mental, evolutionary development. His paper
provides a first, executable, functional model. It is described the waterfall as the simplest description,
tested and further expanded through a sequence but that it would not work for all but the most
of models, that develop an increasing amount of straightforward projects. The rest of his paper
function and an increasing amount of detail as to describes [iterative practices] within the context
how that function is to be executed. Ultimately, of the 60s/70s government-contracting models (a
the model becomes the system. serious set of constraints).

June 2003 3
 This was an ironic insight, given the influ- or FSD would face a $100,000 per day late penalty.
ence this paper had as part of the bulwark The team organized the project into four time-
The first major
promoting a strict sequential life cycle for boxed iterations of about six months each. There
documented large, complex projects. was still a significant up-front specification effort,
IBM FSD IID The next earliest reference comes from and the iteration was longer than normally recom-
application was Harlan Mills, a 1970s software-engineering mended today. Although some feedback-driven evo-
the life-critical thought leader who worked at the IBM FSD. lution occurred in the requirements, O’Neill noted
In his well-known “Top-Down Programming that the IID approach was also a way to manage the
command and
in Large Systems,” Mills promoted iterative complexity and risks of large-scale development.11
control system for development. In addition to his advice to Also in 1972, an IBM FSD competitor, TRW,
the first US Trident begin developing from top-level control struc- applied IID in a major project—the $100 million
submarine. tures downward, perhaps less appreciated TRW/Army Site Defense software project for bal-
was the related life-cycle advice Mills gave for listic missile defense. The project began in February
building the system via iterated expansions:10 1972, and the TRW team developed the system in
five iterations. Iteration 1 tracked a single object,
… it is possible to generate a sequence of interme- and by iteration 5, a few years later, the system was
diate systems of code and functional subspecifica- complete. The iterations were not strictly time
tions so that at every step, each [intermediate] boxed, and there was significant up-front specifi-
system can be verified to be correct… cation work, but the team refined each iteration in
response to the preceding iteration’s feedback.12
Clearly, Mills suggested iterative refinement for As with IBM FSD, TRW (where Royce worked)
the development phase, but he did not mention was an early adopter of IID practices. Indeed, Barry
avoiding a large up-front specification step, did not Boehm, the originator of the IID spiral model in the
specify iteration length, and did not emphasize mid-1980s, was chief scientist at TRW.
feedback and adaptation-driven development from Another mid-1970s extremely large application
each iteration. He did, however, raise these points of IID at FSD was the development of the Light
later in the decade. Given his employment at the Airborne Multipurpose System, part of the US
IBM FSD, we suspect Mills’s exposure to the more Navy’s helicopter-to-ship weapon system. A four-
classic IID projects run there in the early 1970s year 200-person-year effort involving millions of
influenced his thought, but we could not confirm lines of code, LAMPS was incrementally delivered
this with colleagues. in 45 time-boxed iterations (one month per itera-
Early practice of more modern IID (feedback-dri- tion). This is the earliest example we found of a
ven refinement with customer involvement and project that used an iteration length in the range of
clearly delineated iterations) came under the lead- one to six weeks, the length that current popular
ership of Mike Dyer, Bob McHenry, and Don IID methods recommend. The project was quite
O’Neill and many others during their tenure at IBM successful: As Mills wrote, “Every one of those
FSD. The division’s story is fascinating because of deliveries was on time and under budget.”13
the extent and success of its IID use on large, life- In 1975, Vic Basili and Joe Turner published a
critical US Department of Defense (DoD) space and paper about iterative enhancement that clearly
avionics systems during this time. described classic IID:14
The first major documented IBM FSD applica-
tion of IID that we know of was in 1972. This was The basic idea behind iterative enhancement is to
no toy application, but a high-visibility life-critical develop a software system incrementally, allowing
system of more than 1 million lines of code—the the developer to take advantage of what was being
command and control system for the first US learned during the development of earlier, incre-
Trident submarine. O’Neill was project manager, mental, deliverable versions of the system.
and the project included Dyer and McHenry. Learning comes from both the development and
O’Neill conceived and planned the use of IID use of the system, where possible. Key steps in the
(which FSD later called “integration engineering”) process were to start with a simple implementa-
on this project; it was a key success factor, and he tion of a subset of the software requirements and
was awarded an IBM Outstanding Contribution iteratively enhance the evolving sequence of ver-
Award for the work. (Note that IBM leadership vis- sions until the full system is implemented. At each
ibly approved of IID methods.) iteration, design modifications are made along
The system had to be delivered by a certain date with adding new functional capabilities.

4 Computer
 The paper detailed successful IID application to an open loop, rather than a closed loop with
the development of extendable compilers for a fam- user feedback between iterations. The danger Tom Gilb
ily of application-specific programming languages in the sequence [waterfall approach] is that the
introduced the
on a variety of hardware architectures. The project project moves from being grand to being
team developed the base system in 17 iterations grandiose, and exceeds our human intellectual terms “evolution”
over 20 months. They analyzed each iteration from capabilities for management and control. and “evolutionary”
both the user’s and developer’s points of view and to the process
used the feedback to modify both the language And perhaps reflecting several years of see- lexicon.
requirements and design changes in future itera- ing IID in action at FSD, Mills asked, “...why
tions. Finally, they tracked measures, such as cou- do enterprises tolerate the frustrations and
pling and cohesion, over the multiple iterations. difficulties of such [waterfall] development?”
In 1976, Tom Gilb published Software Metrics In 1977, FSD incorporated the Trident IID
(coining the term), in which he discussed his IID approach, which included integrating all software
practice—evolutionary project management—and components at the end of each iteration into its
introduced the terms “evolution” and “evolution- software-engineering practices—an approach
ary” to the process lexicon. This is the earliest book McHenry dubbed “integration engineering.” Some
we could find that had a clear IID discussion and Trident team members and Mills were key advisers
promotion, especially of evolutionary delivery:3 in this incorporation effort.16 Integration engi-
neering spread to the 2,500 FSD software engi-
“Evolution” is a technique for producing the neers, and the idea of IID as an alternative to the
appearance of stability. A complex system will be waterfall stimulated substantial interest within
most successful if it is implemented in small steps IBM’s commercial divisions and senior customer
and if each step has a clear measure of successful ranks and among its competitors.
achievement as well as a “retreat” possibility to a Although unknown to most software profes-
previous successful step upon failure. You have the sionals, another early and striking example of a
opportunity of receiving some feedback from the major IID success is the very heart of NASA’s space
real world before throwing in all resources shuttle software—the primary avionics software
intended for a system, and you can correct possi- system, which FSD built from 1977 to 1980. The
ble design errors… team applied IID in a series of 17 iterations over 31
months, averaging around eight weeks per itera-
The book marked the arrival of a long-standing tion.17 Their motivation for avoiding the waterfall
and passionate voice for evolutionary and iterative life cycle was that the shuttle program’s require-
development. Gilb is one of the earliest and most ments changed during the software development
active IID practitioners and promoters. He began process. Ironically (in hindsight), the authors sound
the practice in the early 1960s and went on to almost apologetic about having to forego the
establish several IID milestones. His material was “ideal” waterfall model for an IID approach:
probably the first with a clear flavor of agile, light,
and adaptive iteration with quick results, similar Due to the size, complexity, and evolutionary
to that of newer IID methods. [changing requirements] nature of the program, it
By 1976, Mills had strengthened his IID mes- was recognized early that the ideal software devel-
sage:15 opment life cycle [the waterfall model] could not
be strictly applied...However, an implementation
Software development should be done incremen- approach (based on small incremental releases)
tally, in stages with continuous user participation was devised for STS-1 which met the objectives by
and replanning and with design-to-cost program- applying the ideal cycle to small elements of the
ming within each stage. overall software package on an iterative basis.

Using a three-year inventory system project as a The shuttle project also exhibited classic IID prac-
backdrop, he challenged the idea and value of up- tices: time-boxed iterations in the eight-week range,
front requirements or design specification: feedback-driven refinement of specifications, and
so on.
...there are dangers, too, particularly in the con- The first IID discussion in the popular press that
duct of these [waterfall] stages in sequence, and we could find was in 1978, when Tom Gilb began
not in iteration-i.e., that development is done in publishing a column in the UK’s Computer Weekly.

June 2003 5
 The column regularly promoted IID, as well a synonym for waterfall during this period suggests
as evolutionary project management and its unquestioned dominance. Contrast this to its
The IID practice
delivery. In his 6 April 1978 column, Gilb qualified use in the 1990s, “sequential life cycle”
of evolutionary wrote, or “iterative life cycle.”)
prototyping was In 1982, William Swartout and Robert Balzer
commonly used Management does not require firm estimates argued that specification and design have a neces-
in 1980s efforts of completion, time, and money for the entire sary interplay, and they promoted an iterative and
project. Each [small iterative] step must meet evolutionary approach to requirements engineer-
to create artificial one of the following criteria (priority order): ing and development.23 The same year also pro-
intelligence either (a) give planned return on investment vided the earliest reference to a very large appli-
systems. payback, or, if impossible, then (b) give cation successfully built using evolutionary proto-
breakeven (no loss); or, at least, (c) some posi- typing, an IID approach that does not usually
tive user benefit measurably; or, at least (d) include time-boxed iterations. The $100 million
some user environment feedback and learning. military command and control project was based
on IBM’s Customer Information Control System
Another discussion of incremental development, technology.24
although published in 1984, refers to a System In 1983, Grady Booch published Software
Development Corp. project to build an air defense Engineering with Ada,25 in which he described an
system, which began in 1977 and finished in 1980. iterative process for growing an object-oriented sys-
The project combined significant up-front specifi- tem. The book was influential primarily in the DoD
cations with incremental development and builds. development community, but more for the object-
Ostensibly, the project was meant to fit within DoD oriented design method than for its iterative advice.
single-pass waterfall standards, with testing and However, Booch’s later 1990s books that covered
integration in the last phase. Carolyn Wong com- IID found a large general audience, and many first
ments on the unrealism of this approach and the considered or tried iterative development through
team’s need to use incremental development:18 their influence.
The early 1980s was an active period for the
The [waterfall] model was adopted because soft- (attempted) creation of artificial intelligence systems,
ware development was guided by DoD stan- expert systems, and so on, especially using Lisp
dards…In reality, software development is a machines. A common approach in this community
complex, continuous, iterative, and repetitive was the IID practice of evolutionary prototyping.26
process. The [waterfall model] does not reflect this In another mid-1980s questioning of the sequen-
complexity. tial life cycle, Gilb wrote “Evolutionary Delivery
versus the ‘Waterfall Model.’” In this paper, Gilb
THE EIGHTIES promoted a more aggressive strategy than other IID
In 1980 Weinberg wrote about IID in “Adaptive discussions of the time, recommending frequent
Programming: The New Religion,” published in (such as every few weeks) delivery of useful results
Australasian Computerworld. Summarizing the to stakeholders.27
article, he said, “The fundamental idea was to build A 1985 landmark in IID publications was
in small increments, with feedback cycles involv- Barry Boehm’s “A Spiral Model of Software
ing the customer for each.” A year later, Tom Gilb Development and Enhancement,” (although the
wrote in more detail about evolutionary develop- more frequent citation date is 1986).28 The spiral
ment.19 model was arguably not the first case in which a
In the same year, Daniel McCracken and Michael team prioritized development cycles by risk: Gilb
Jackson promoted IID and argued against the “stul- and IBM FSD had previously applied or advocated
tifying waterfall” in a chapter within a software variations of this idea, for example. However, the
engineering and design text edited by William spiral model did formalize and make prominent
Cotterman. The chapter’s title, “A Minority the risk-driven-iterations concept and the need to
Dissenting Position,” underscored the subordinate use a discrete step of risk assessment in each iter-
position of IID to the waterfall model at the time.20 ation.
Their arguments continued in “Life-Cycle Concept In 1986, Frederick Brooks, a prominent soft-
Considered Harmful,”21 a 1982 twist on Edsger ware-engineering thought leader of the 1970s and
Dijkstra’s late 1960s classic “Go To Statement 1980s, published the classic “No Silver Bullet”
Considered Harmful.”22 (The use of “life cycle” as extolling the advantages of IID:29

6 Computer
 Nothing in the past decade has so radically attention to high risks and the core archi-
changed my own practice, or its effectiveness [as tecture in the early iterations. The Cleanroom
incremental development]. Bill Curtis and colleagues published a par-
ticularly agile-relevant paper during this method
Commenting on adopting a waterfall process, decade,32 reporting results on research into incorporated
Brooks wrote the processes that influenced 19 large pro- evolutionary
jects. The authors identified that the pre- development
Much of present-day software acquisition proce- scriptive waterfall model attempted to satisfy
with more formal
dure rests upon the assumption that one can spec- management accountability goals, but they
ify a satisfactory system in advance, get bids for did not describe how projects successfully methods of
its construction, have it built, and install it. I think ran. The paper also noted that successful specification
this assumption is fundamentally wrong, and that development emphasizes a cyclic learning and proof.
many software acquisition problems spring from process with high attention to people’s skills,
that fallacy. common vision, and communication issues,
rather than viewing the effort as a sequential
Perhaps summing up a decade of IID-promoting “manufacturing process.” As the authors state,
messages to military standards bodies and other
organizations, Brooks made his point very clear in The conclusion that stands out most clearly from
his keynote speech at the 1995 International our field study observations is that the process of
Conference on Software Engineering: “The water- developing large software systems must be treated,
fall model is wrong!” at least in part, as a learning and communication
In 1986, David Parnas and Paul Clements pub- process.
lished “A Rational Design Process: How and Why
to Fake It.”30 In it, they stated that, although they In 1987, as part of the IBM FSD Software
believe in the ideal of the waterfall model (thorough, Engineering Practices program, Mills, Dyer, and
correct, and clear specifications before develop- Rick Linger continued the evolution of IID with
ment), it is impractical. They listed many reasons, the Cleanroom method, which incorporated evo-
including (paraphrased) lutionary development with more formal methods
of specification and proof, reflecting Mills’s strong
• A system’s users seldom know exactly what mathematical influences.33
they want and cannot articulate all they know. By the late 1980s, the DoD was experiencing sig-
• Even if we could state all requirements, there nificant failure in acquiring software based on the
are many details that we can only discover strict, document-driven, single-pass waterfall
once we are well into implementation. model that DoD-Std-2167 required. A 1999 review
• Even if we knew all these details, as humans, of failure rates in a sample of earlier DoD projects
we can master only so much complexity. drew grave conclusions: “Of a total $37 billion for
• Even if we could master all this complexity, the sample set, 75% of the projects failed or were
external forces lead to changes in require- never used, and only 2% were used without exten-
ments, some of which may invalidate earlier sive modification.”34 Consequently, at the end of
decisions. 1987, the DoD changed the waterfall-based stan-
dards to allow IID, on the basis of recommenda-
and commented that for all these reasons, “the pic- tions in an October 1987 report from the Defense
ture of the software designer deriving his design in Science Board Task Force on Military Software,
a rational, error-free way from a statement of chaired by Brooks. The report recommended
requirements is quite unrealistic.” replacing the waterfall, a failing approach on many
In 1987, TRW launched a four-year project to large DoD projects, with iterative development:
build the Command Center Processing and Display
System Replacement (CCPDS-R), a command and DoD-Std-2167 likewise needs a radical overhaul
control system, using IID methods. Walker Royce to reflect modern best practice. Draft 2167A is a
described the effort in 60 pages of detail.31 The step, but it does not go nearly far enough. As
team time-boxed six iterations, averaging around drafted, it continues to reinforce exactly the doc-
six months each. The approach was consistent ument-driven, specify-then-build approach that
with what would later become the Rational lies at the heart of so many DoD software prob-
Unified Process (to which Royce contributed): lems….

June 2003 7
 In the decade since the waterfall model was the IID material from Software Metrics. Gilb
Tom Gilb’s developed, our discipline has come to recog- described the Evo method, distinguished by frequent
nize that [development] requires iteration evolutionary delivery and an emphasis on defining
Principles of
between the designers and users. quantified measurable goals and then measuring the
Software actual results from each time-boxed short iteration.
Engineering Finally, in a section titled “Professional
Management was Humility and Evolutionary Development” 1990 TO THE PRESENT
the first book with (humility to accept that the 2167’s goals— By the 1990s, especially the latter half, public
get the specifications accurate without incre- awareness of IID in software development was sig-
substantial chapters mental implementation and feedback—was nificantly accelerating. Hundreds of books and
dedicated to not possible), the report stated: papers were promoting IID as their main or sec-
IID discussion ondary theme. Dozens more IID methods sprang
and promotion. Experience with confidently specifying and forth, which shared an increasing trend to time-
painfully building mammoths has shown it to boxed iterations of one to six weeks.
be simplest, safest, and even fastest to develop In the 1970s and 1980s, some IID projects still
a complex software system by building a min- incorporated a preliminary major specification
imal version, putting it into actual use, and then stage, although their teams developed them in iter-
adding functions [and other qualities] according ations with minor feedback. In the 1990s, in con-
to the priorities that emerge from actual use. trast, methods tended to avoid this model,
Evolutionary development is best technically, preferring less early specification work and a
and it saves time and money. stronger evolutionary analysis approach.
The DoD was still experiencing many failures with
Both DoD overseers and contractors often view “waterfall-mentality” projects. To correct this and
the updated DoD-Std-2167A, released in February to reemphasize the need to replace the waterfall
1988, as the epitome of a waterfall specification. model with IID, the Defense Science Board Task
Yet, its authors actually wanted it to be an amend- Force on Acquiring Defense Software Commercially,
ment (hence the A) for life-cycle neutrality that chaired by Paul Kaminski, issued a report in June
allowed IID alternatives to the waterfall: 1994 that stated simply, “DoD must manage pro-
grams using iterative development. Apply evolu-
This standard is not intended to specify or dis- tionary development with rapid deployment of
courage the use of any particular software devel- initial functional capability.”
opment method. The contractor is responsible for Consequently, in December 1994, Mil-Std-498
selecting software development methods (for replaced 2167A. An article by Maj. George
example, rapid prototyping) that best support the Newberry summarizing the changes included a sec-
achievement of contract requirements. tion titled “Removing the Waterfall Bias,” in which
he described the goal of encouraging evolutionary
Despite this intent, many (justifiably) interpreted acquisition and IID:36
the new standard as containing an implied prefer-
ence for the waterfall model because of its contin- Mil-Std-498 describes software development in
ued document-driven milestone approach. one or more incremental builds. Each build imple-
Ironically, in a conversation nearly a decade later, ments a specified subset of the planned capabili-
the principal creator of DoD-Std-2167 expressed ties. The process steps are repeated for each build,
regret for creating the strict waterfall-based stan- and within each build, steps may be overlapping
dard. He said that at the time he knew of the sin- and iterative.
gle-pass document-driven waterfall model, and
others he questioned advised it was excellent, as Mil-Std-498 itself clearly states the core IID prac-
did the literature he examined, but he had not heard tices of evolving requirements and design incre-
of iterative development. In hindsight, he said he mentally with implementation:
would have made a strong recommendation for IID
rather than the waterfall model. If a system is developed in multiple builds, its
In 1988, Gilb published Principles of Software requirements may not be fully defined until the
Engineering Management, the first book with sub- final build…. If a system is designed in multiple
stantial chapters dedicated to IID discussion and builds, its design may not be fully defined until the
promotion.35 In it he reiterated and expanded on final build.

8 Computer
 Meanwhile, in the commercial realm, Jeff DeLuca created an overall iterative process
Sutherland and Ken Schwaber at Easel Corp. had description, Feature-Driven Development XP garnered
started to apply what would become known as the (FDD), that also incorporated ideas from
significant public
Scrum method, which employed time-boxed 30- Coad.43
day iterations. The method took inspiration from In 1998, the Standish Group issued its attention because
a Japanese IID approach used for nonsoftware widely cited “CHAOS: Charting the Seas of of its emphasis on
products at Honda, Canon, and Fujitsu in the Information Technology,” a report that ana- communication,
1980s; from Shashimi (“slices” or iterations); and lyzed 23,000 projects to determine failure simplicity, and
from a version of Scrum described in 1986.37 A factors. The top reasons for project failure,
1999 article described their later refinements to according to the report, were associated with
testing, and its
Scrum.38 waterfall practices. It also concluded that IID sustainable
In January 1994, a group of 16 rapid application practices tended to ameliorate the failures. developer-oriented
development (RAD) practitioners met in the UK to One of the report’s key conclusions was to practices.
discuss the definition of a standard iterative process adopt IID:
to support RAD development. The group drew
inspiration from James Martin’s RAD teachings. Research also indicates that smaller time
Martin, in turn, had taken his inspiration from the frames, with delivery of software components
time-boxing work at Dupont, led by Scott Shultz early and often, will increase the success rate.
in the mid-1980s. The RAD group’s process defin- Shorter time frames result in an iterative process
ition would eventually become the Dynamic of design, prototype, develop, test, and deploy
Systems Development Method (DSDM), an IID small elements.
method that predictably had more early advocates
in Europe and has since spread.39 In 2000, DoD replaced Mil-Std-498 with
In the early 1990s, a consortium of companies another software acquisition standard, DoD
began a project to build a new-generation 5000.2, which again recommended adopting evo-
Canadian Automated Air Traffic Control System lutionary acquisition and the use of IID:
(CAATS) using a risk-driven IID method. The pro-
ject, under the process leadership of Philippe There are two approaches, evolutionary and sin-
Kruchten, used a series of six-month iterations, rel- gle step [waterfall], to full capability. An evolu-
atively long by today’s standards. The project was tionary approach is preferred. … [In this]
a success, despite its prior near-failure applying a approach, the ultimate capability delivered to the
waterfall approach.40 user is divided into two or more blocks, with
In the mid-1990s, many contributors within increasing increments of capability...software
Rational Corp. (including Kruchten and Walker development shall follow an iterative spiral devel-
Royce) and its clients created the Rational Unified opment process in which continually expanding
Process, now a popular IID method. A 1995 mile- software versions are based on learning from ear-
stone was the public promotion of the daily build lier development.
and smoke test, a widely influential IID practice
institutionalized by Microsoft that featured a one- In 2001, Alan MacCormack reported a study of
day micro-iteration.41 key success factors in recent projects; first among
In 1996, Kent Beck joined the Chrysler C3 pay- these was adopting an IID life cycle:44
roll project. It was in this context that the full set of
XP practices matured, with some collaboration by Now there is proof that the evolutionary approach
Ron Jeffries and inspiration from earlier 1980s to software development results in a speedier
work at Tektronix with Ward Cunningham. XP process and higher-quality products. […] The iter-
went on to garner significant public attention ative process is best captured in the evolutionary
because of its emphasis on communication, sim- delivery model proposed by Tom Gilb.
plicity, and testing, its sustainable developer-
oriented practices, and its interesting name.42 In February 2001, a group of 17 process
In 1997, a project to build a large logistics system experts—representing DSDM, XP, Scrum, FDD,
in Singapore, which had been running as a water- and others—interested in promoting modern, sim-
fall project, was facing failure. With the collabora- ple IID methods and principles met in Utah to dis-
tion of Peter Coad and Jeff De Luca, the team cuss common ground. From this meeting came the
resurrected it and ran it as a successful IID project. Agile Alliance (www.agilealliance.org) and the now

June 2003 9
 popular catch phrase “agile methods,” all of which
apply IID. And in 2002, Alistair Cockburn, one of References
the participants, published the first book under the 1. W. Shewhart, Statistical Method from the Viewpoint
new appellation.45 of Quality Control, Dover, 1986 (reprint from 1939).
2. W.E. Deming, Out of the Crisis, SPC Press, 1982;
reprinted in paperback by MIT Press, 2003.
n a typical quip, H.L. Mencken said, “For every

I
3. T. Gilb, Software Metrics, Little, Brown, and Co.,
complex problem, there is a solution that is sim- 1976 (out of print).
ple, neat, and wrong.” In the history of science, 4. R. Zultner, “The Deming Approach to Quality Soft-
it is the norm that simplistic but inferior ideas first ware Engineering,” Quality Progress, vol. 21, no. 11,
hold the dominant position, even without sup- 1988, pp. 58-64.
porting results. Medicine’s four humors and related 5. W.H. Dana, The X-15 Lessons Learned, tech. report,
astrological diagnosis and prescription dominated NASA Dryden Research Facility, 1993.
Europe for more than a millennium, for example. 6. B. Randell and F.W. Zurcher, “Iterative Multi-Level
Software development is a very young field, and Modeling: A Methodology for Computer System
it is thus no surprise that the simplified single-pass Design,” Proc. IFIP, IEEE CS Press, 1968, pp. 867-
and document-driven waterfall model of “require- 871.
ments, design, implementation” held sway during 7. M.M. Lehman, “The Programming Process,” inter-
the first attempts to create the ideal development nal IBM report, 1969; reprinted in Program Evolu-
process. Other reasons for the waterfall idea’s early tion—Processes of Software Change, Academic Press,
adoption or continued promotion include: 1985.
8. R. Glass, “Elementary Level Discussion of Com-
• It’s simple to explain and recall. “Do the piler/Interpreter Writing,” ACM Computing Surveys,
requirements, then design, and then imple- Mar. 1969, pp. 64-68.
ment.” IID is more complex to understand and 9. W. Royce, “Managing the Development of Large
describe. Even Winston Royce’s original two- Software Systems,” Proc. Westcon, IEEE CS Press,
iteration waterfall immediately devolved into 1970, pp. 328-339.
a single sequential step as other adopters used 10. H. Mills, “Debugging Techniques in Large Systems,”
it and writers described it. Software Productivity, Dorset House, 1988.
• It gives the illusion of an orderly, accountable, 11. D. O’Neill, “Integration Engineering Perspective,” J.
and measurable process, with simple docu- Systems and Software, no. 3, 1983, pp. 77-83.
ment-driven milestones (such as “requirements 12. R.D. Williams, “Managing the Development of Reli-
complete”). able Software,” Proc. Int’l Conf. Reliable Software,
• It was promoted in many software engineering, ACM Press, 1975, pp. 3-8.
requirements engineering, and management 13. H. Mills, “Principles of Software Engineering,” IBM
texts, courses, and consulting organizations. Systems J., vol. 19, no. 4, 1980, pp. 289-295.
It was labeled appropriate or ideal, seemingly 14. V. Basili and J. Turner, “Iterative Enhancement: A
unaware of this history or of the statistically Practical Technique for Software Development,”
significant research evidence in favor of IID. IEEE Trans. Software Eng., Dec. 1975, pp. 390-
396.
This brief history shows that IID concepts have 15. H. Mills, “Software Development,” IEEE Trans.
been and are a recommended practice by promi- Software Eng., Dec. 1976, pp. 265-273.
nent software-engineering thought leaders of each 16. D. O’Neill, “The Management of Software Engi-
decade, associated with many successful large pro- neering,” IBM Systems J., vol. 19, no. 4, 1980, pp.
jects, and recommended by standards boards. 421-431.
Yet, even though the value of IID is well known 17. W. Madden and K. Rone, “Design, Development,
among literate, experienced software engineers, Integration: Space Shuttle Flight Software System,”
some commercial organizations, consulting compa- Comm. ACM, Sept. 1984, pp. 914-925.
nies, and standards bodies still promote a document- 18. C. Wong, “A Successful Software Development,”
driven single-pass sequential life cycle as the ideal. IEEE Trans. Software Eng., no. 3, 1984, pp. 714-
We conclude with this recommendation: In the inter- 727.
est of promoting greater project success and saving 19. T. Gilb, “Evolutionary Development,” ACM Soft-
taxpayer or investor dollars, let’s continue efforts to ware Eng. Notes, Apr. 1981, p. 17.
educate and promote the use of IID methods. ■ 20. W.W. Cotterman et al., eds., Systems Analysis and

10 Computer
 Design: A Foundation for the 1980’s, North-Hol- ment,” Pattern Languages of Program Design, vol.
land, 1981. 4, 1999, pp. 637-651.
21. D. McCracken and M. Jackson, “Life-Cycle Concept 39. J. Stapleton, DSDM: Dynamic Systems Development
Considered Harmful,” ACM Software Eng. Notes, Method, Addison-Wesley, 1997.
Apr. 1982, pp. 29-32. 40. P. Kruchten, “Rational Development Process,”
22. E. Dijkstra, “Go To Statement Considered Harmful,” Crosstalk: J. Defense Software Eng., July 1996,
Comm. ACM, Mar. 1968, pp. 147-148. www.stsc.hill.af.mil/crosstalk/frames.asp?uri=1996/07/
23. W. Swartout and R. Balzer, “On the Inevitable Inter- rational.asp.
twining of Specification and Implementation,” 41. J. McCarthy, Dynamics of Software Development,
Comm. ACM, July 1982, pp. 438-440. Microsoft Press, 1995.
24. D. Tamanaha, “An Integrated Rapid Prototyping 42. K. Beck, Extreme Programming Explained: Embrace
Methodology for Command and Control Systems: Change, Addison-Wesley, 1999.
Experience and Insight,” ACM Software Eng. Notes, 43. P. Coad et al., “Feature-Driven Development,” in
Dec. 1982, pp. 387-396. Java Modeling in Color with UML, Prentice Hall,
25. G. Booch, Software Engineering with Ada, Benjamin- 1999.
Cummings, 1983. 44. A. MacCormack, “Product-Development Practices
26. R. Budde et al., eds., Approaches to Prototyping, That Work,” MIT Sloan Management Rev., vol. 42,
Springer Verlag, 1984. no. 2, 2001, pp. 75-84.
27. T. Gilb, “Evolutionary Delivery versus the ‘Waterfall 45. A. Cockburn, Agile Software Development, Addi-
Model’,” ACM Software Requirements Eng. Notes, son-Wesley, 2002.
July 1985.
28. B. Boehm, “A Spiral Model of Software Development
and Enhancement,” Proc. Int’l Workshop Software
Process and Software Environments, ACM Press,
1985; also in ACM Software Eng. Notes, Aug. 1986,
pp. 22-42.
29. F. Brooks, “No Silver Bullet: Essence and Accidents
of Software Engineering,” Proc. IFIP, IEEE CS Press,
1987, pp. 1069-1076; reprinted in Computer, Apr.
1987, pp. 10-19.
30. D. Parnas and P. Clements, “A Rational Design
Process: How and Why to Fake It,” IEEE Trans. Craig Larman is chief scientist for Valtech, an
Software Eng., Feb. 1986, pp. 251-257. international consulting company, and he speaks
31. W. Royce, Software Project Management, Addison- and consults worldwide. He is also the author of
Wesley, 1998. Agile and Iterative Development: A Manager’s
32. W. Curtis et al., “On Building Software Process Mod- Guide (Addison-Wesley, 2003), which examines
els under the Lamppost,” Proc. Int’l Conf. Software both historical and other forms of evidence demon-
Eng., IEEE CS Press, 1987, pp. 96-103. strating the advantages of iterative methods. Lar-
33. H. Mills et al., “Cleanroom Software Engineering,” man is a member of the ACM and the IEEE.
IEEE Software, Sept. 1987, pp. 19-25. Contact him at [email protected].
34. S. Jarzombek, Proc. Joint Aerospace Weapons Sys-
tems Support, Sensors and Simulation Symp., Gov’t
Printing Office Press, 1999.
35. T. Gilb, Principles of Software Engineering Manage-
ment, Addison Wesley Longman, 1989.
36. G.A. Newberry, “Changes from DOD-STD-2167A Victor R. Basili is a professor of computer science
to MIL-STD-498,” Crosstalk: J. Defense Software at the University of Maryland and executive direc-
Eng., Apr. 1995, www.stsc.hill.af.mil/crosstalk/ tor of the Fraunhofer Center-Maryland, where he
frames.asp?uri=1995/04/Changes.asp. works on measuring, evaluating, and improving
37. H. Takeuchi and I. Nonaka, “The New New Product the software development process and product. He
Development Game,” Harvard Business Rev., Jan. is an IEEE and ACM fellow and co-editor-in-chief
1986, pp. 137-146. of Kluwer’s Empirical Software Engineering: An
38. M. Beedle et al., “SCRUM: An Extension Pattern International Journal. Contact him at basili@cs.
Language for Hyperproductive Software Develop- umd.edu.

June 2003 11
 REPORT DOCUMENTATION PAGE
1. Recipient’s Reference 2. Originator’s References 3. Further Reference 4. Security Classification
of Document
RTO-TR-IST-026 ISBN UNCLASSIFIED/
AC/323(IST-026)TP/190 978-92-837-0042-5 UNLIMITED
5. Originator
Research and Technology Organisation
North Atlantic Treaty Organisation
BP 25, F-92201 Neuilly-sur-Seine Cedex, France
6. Title
Evolutionary Software Development
7. Presented at/Sponsored by

Final Report of the Task Group IST-026/RTG-008.

8. Author(s)/Editor(s) 9. Date

Multiple August 2008

10. Author’s/Editor’s Address 11. Pages

Multiple 62
12. Distribution Statement
There are no restrictions on the distribution of this document.
Information about the availability of this and other RTO
unclassified publications is given on the back cover.
13. Keywords/Descriptors
Acquisition Methodology
Armed forces procurement Military applications
Criteria Requirements
Decision making Software development
ESD (Evolutionary Software Development) Software engineering
Evaluation Specifications
Management Standards
14. Abstract

This task group investigated iterative processes for software development, especially those (called
Evolutionary Software Development) that span many cycles of software implementation, release,
fielding of the product, learning from the field experience, then updating the requirements for
subsequent releases. This goes beyond the Spiral Model or Agile Methods when they are only used
prior to initial delivery, and also beyond incremental delivery. The methodology followed was to
review the literature, examine case studies, sponsor a public symposium (IST-034/RSY-010) to
collect external input, and then within the task group resolve the best way to present our findings.
The group itself did not have the resources to undertake any original research. The principal
findings were that iterative processes have been used successfully in military software projects since
the 1950’s and continue to be viable and exhibit advantages over strictly sequential processes such
as the Waterfall Model or the V-Model. Nevertheless, there remain outstanding research questions
to be resolved with potential to improve the process.

RTO-TR-IST-026
RTO-TR-IST-026
 NORTH ATLANTIC TREATY ORGANISATION RESEARCH AND TECHNOLOGY ORGANISATION

BP 25 DIFFUSION DES PUBLICATIONS

F-92201 NEUILLY-SUR-SEINE CEDEX • FRANCE
Télécopie 0(1)55.61.22.99 • E-mail [email protected] RTO NON CLASSIFIEES
Les publications de l’AGARD et de la RTO peuvent parfois être obtenues auprès des centres nationaux de distribution indiqués ci-dessous. Si vous
souhaitez recevoir toutes les publications de la RTO, ou simplement celles qui concernent certains Panels, vous pouvez demander d’être inclus soit à
titre personnel, soit au nom de votre organisation, sur la liste d’envoi.
Les publications de la RTO et de l’AGARD sont également en vente auprès des agences de vente indiquées ci-dessous.
Les demandes de documents RTO ou AGARD doivent comporter la dénomination « RTO » ou « AGARD » selon le cas, suivi du numéro de série.
Des informations analogues, telles que le titre est la date de publication sont souhaitables.
Si vous souhaitez recevoir une notification électronique de la disponibilité des rapports de la RTO au fur et à mesure de leur publication, vous pouvez
consulter notre site Web (www.rto.nato.int) et vous abonner à ce service.

CENTRES DE DIFFUSION NATIONAUX

ALLEMAGNE HONGRIE PORTUGAL
Streitkräfteamt / Abteilung III Department for Scientific Analysis Estado Maior da Força Aérea
Fachinformationszentrum der Bundeswehr (FIZBw) Institute of Military Technology SDFA – Centro de Documentação
Gorch-Fock-Straße 7, D-53229 Bonn Ministry of Defence Alfragide
P O Box 26 P-2720 Amadora
BELGIQUE H-1525 Budapest
Royal High Institute for Defence – KHID/IRSD/RHID REPUBLIQUE TCHEQUE
Management of Scientific & Technological Research ISLANDE LOM PRAHA s. p.
for Defence, National RTO Coordinator Director of Aviation o. z. VTÚLaPVO
Royal Military Academy – Campus Renaissance c/o Flugrad Mladoboleslavská 944
Renaissancelaan 30, 1000 Bruxelles Reykjavik PO Box 18
197 21 Praha 9
CANADA ITALIE
DSIGRD2 – Bibliothécaire des ressources du savoir General Secretariat of Defence and ROUMANIE
R et D pour la défense Canada National Armaments Directorate Romanian National Distribution
Ministère de la Défense nationale 5th Department – Technological Centre
305, rue Rideau, 9e étage Research Armaments Department
Ottawa, Ontario K1A 0K2 Via XX Settembre 123 9-11, Drumul Taberei Street
00187 Roma Sector 6
DANEMARK 061353, Bucharest
Danish Acquisition and Logistics Organization (DALO) LUXEMBOURG
Lautrupbjerg 1-5, 2750 Ballerup Voir Belgique ROYAUME-UNI
Dstl Knowledge Services
ESPAGNE NORVEGE Information Centre
SDG TECEN / DGAM Norwegian Defence Research Building 247
C/ Arturo Soria 289 Establishment Dstl Porton Down
Madrid 28033 Attn: Biblioteket Salisbury
P.O. Box 25 Wiltshire SP4 0JQ
ETATS-UNIS NO-2007 Kjeller
NASA Center for AeroSpace Information (CASI) SLOVENIE
7115 Standard Drive PAYS-BAS Ministry of Defence
Hanover, MD 21076-1320 Royal Netherlands Military Central Registry for EU and
Academy Library NATO
FRANCE P.O. Box 90.002 Vojkova 55
O.N.E.R.A. (ISP) 4800 PA Breda 1000 Ljubljana
29, Avenue de la Division Leclerc
BP 72, 92322 Châtillon Cedex POLOGNE TURQUIE
Centralny Ośrodek Naukowej Milli Savunma Bakanlığı (MSB)
GRECE (Correspondant) Informacji Wojskowej ARGE ve Teknoloji Dairesi
Defence Industry & Research General Al. Jerozolimskie 97 Başkanlığı
Directorate, Research Directorate 00-909 Warszawa 06650 Bakanliklar
Fakinos Base Camp, S.T.G. 1020 Ankara
Holargos, Athens

AGENCES DE VENTE
NASA Center for AeroSpace The British Library Document Canada Institute for Scientific and
Information (CASI) Supply Centre Technical Information (CISTI)
7115 Standard Drive Boston Spa, Wetherby National Research Council Acquisitions
Hanover, MD 21076-1320 West Yorkshire LS23 7BQ Montreal Road, Building M-55
ETATS-UNIS ROYAUME-UNI Ottawa K1A 0S2, CANADA
Les demandes de documents RTO ou AGARD doivent comporter la dénomination « RTO » ou « AGARD » selon le cas, suivie du numéro de série
(par exemple AGARD-AG-315). Des informations analogues, telles que le titre et la date de publication sont souhaitables. Des références
bibliographiques complètes ainsi que des résumés des publications RTO et AGARD figurent dans les journaux suivants :
Scientific and Technical Aerospace Reports (STAR) Government Reports Announcements & Index (GRA&I)
STAR peut être consulté en ligne au localisateur de ressources publié par le National Technical Information Service
uniformes (URL) suivant: http://www.sti.nasa.gov/Pubs/star/Star.html Springfield
STAR est édité par CASI dans le cadre du programme Virginia 2216
NASA d’information scientifique et technique (STI) ETATS-UNIS
STI Program Office, MS 157A (accessible également en mode interactif dans la base de
NASA Langley Research Center données bibliographiques en ligne du NTIS, et sur CD-ROM)
Hampton, Virginia 23681-0001
ETATS-UNIS
 NORTH ATLANTIC TREATY ORGANISATION RESEARCH AND TECHNOLOGY ORGANISATION

BP 25 DISTRIBUTION OF UNCLASSIFIED
F-92201 NEUILLY-SUR-SEINE CEDEX • FRANCE RTO PUBLICATIONS
Télécopie 0(1)55.61.22.99 • E-mail [email protected]
AGARD & RTO publications are sometimes available from the National Distribution Centres listed below. If you wish to receive all RTO reports,
or just those relating to one or more specific RTO Panels, they may be willing to include you (or your Organisation) in their distribution.
RTO and AGARD reports may also be purchased from the Sales Agencies listed below.
Requests for RTO or AGARD documents should include the word ‘RTO’ or ‘AGARD’, as appropriate, followed by the serial number. Collateral
information such as title and publication date is desirable.
If you wish to receive electronic notification of RTO reports as they are published, please visit our website (www.rto.nato.int) from where you can
register for this service.

NATIONAL DISTRIBUTION CENTRES

BELGIUM HUNGARY PORTUGAL
Royal High Institute for Defence – KHID/IRSD/RHID Department for Scientific Analysis Estado Maior da Força Aérea
Management of Scientific & Technological Research Institute of Military Technology SDFA – Centro de Documentação
for Defence, National RTO Coordinator Ministry of Defence Alfragide
Royal Military Academy – Campus Renaissance P O Box 26 P-2720 Amadora
Renaissancelaan 30 H-1525 Budapest
1000 Brussels ROMANIA
ICELAND Romanian National Distribution
CANADA Director of Aviation Centre
DRDKIM2 – Knowledge Resources Librarian c/o Flugrad, Reykjavik Armaments Department
Defence R&D Canada 9-11, Drumul Taberei Street
Department of National Defence ITALY Sector 6, 061353, Bucharest
305 Rideau Street, 9th Floor General Secretariat of Defence and
Ottawa, Ontario K1A 0K2 National Armaments Directorate SLOVENIA
5th Department – Technological Ministry of Defence
CZECH REPUBLIC Research Central Registry for EU and
LOM PRAHA s. p. Via XX Settembre 123 NATO
o. z. VTÚLaPVO 00187 Roma Vojkova 55
Mladoboleslavská 944 1000 Ljubljana
PO Box 18 LUXEMBOURG
197 21 Praha 9 See Belgium SPAIN
SDG TECEN / DGAM
DENMARK NETHERLANDS C/ Arturo Soria 289
Danish Acquisition and Logistics Organization (DALO) Royal Netherlands Military Madrid 28033
Lautrupbjerg 1-5 Academy Library
2750 Ballerup P.O. Box 90.002 TURKEY
4800 PA Breda Milli Savunma Bakanlığı (MSB)
FRANCE ARGE ve Teknoloji Dairesi
O.N.E.R.A. (ISP) NORWAY Başkanlığı
29, Avenue de la Division Leclerc Norwegian Defence Research 06650 Bakanliklar – Ankara
BP 72, 92322 Châtillon Cedex Establishment
Attn: Biblioteket UNITED KINGDOM
GERMANY P.O. Box 25 Dstl Knowledge Services
Streitkräfteamt / Abteilung III NO-2007 Kjeller Information Centre
Fachinformationszentrum der Bundeswehr (FIZBw) Building 247
Gorch-Fock-Straße 7 POLAND Dstl Porton Down
D-53229 Bonn Centralny Ośrodek Naukowej Salisbury, Wiltshire SP4 0JQ
Informacji Wojskowej
GREECE (Point of Contact) Al. Jerozolimskie 97 UNITED STATES
Defence Industry & Research General Directorate 00-909 Warszawa NASA Center for AeroSpace
Research Directorate, Fakinos Base Camp Information (CASI)
S.T.G. 1020 7115 Standard Drive
Holargos, Athens Hanover, MD 21076-1320
SALES AGENCIES
NASA Center for AeroSpace The British Library Document Canada Institute for Scientific and
Information (CASI) Supply Centre Technical Information (CISTI)
7115 Standard Drive Boston Spa, Wetherby National Research Council Acquisitions
Hanover, MD 21076-1320 West Yorkshire LS23 7BQ Montreal Road, Building M-55
UNITED STATES UNITED KINGDOM Ottawa K1A 0S2, CANADA

Requests for RTO or AGARD documents should include the word ‘RTO’ or ‘AGARD’, as appropriate, followed by the serial number (for example
AGARD-AG-315). Collateral information such as title and publication date is desirable. Full bibliographical references and abstracts of RTO and
AGARD publications are given in the following journals:
Scientific and Technical Aerospace Reports (STAR) Government Reports Announcements & Index (GRA&I)
STAR is available on-line at the following uniform resource published by the National Technical Information Service
locator: http://www.sti.nasa.gov/Pubs/star/Star.html Springfield
STAR is published by CASI for the NASA Scientific Virginia 2216
and Technical Information (STI) Program UNITED STATES
STI Program Office, MS 157A (also available online in the NTIS Bibliographic Database
NASA Langley Research Center or on CD-ROM)
Hampton, Virginia 23681-0001
UNITED STATES
ISBN 978-92-837-0042-5