See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/326339847

The Rise and Evolution of Agile Software Development

Article in IEEE Software · July 2018

DOI: 10.1109/MS.2018.290111318

CITATIONS READS
244 13,691

3 authors, including:

Rashina Hoda Norsaremah Salleh

Monash University (Australia) International Islamic University Malaysia
177 PUBLICATIONS 4,167 CITATIONS 82 PUBLICATIONS 2,148 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Rashina Hoda on 31 January 2019.

The user has requested enhancement of the downloaded file.

 FOCUS: SOFTWARE ENGINEERING’S 50TH ANNIVERSARY

The Rise and process and project measurement,

documentation, and other support-
ing practices, the human side of

Evolution of
engineering software was seen by
many to have become, or to certainly
be becoming, lost in mainstream

Agile Software
software development. Emerging in
the late 1990s in response to the
then-prevalent complex methods,
agile methods offered disciplined

Development yet lightweight processes while plac-

ing human effort and experience at
the core of software development,
through its central focus on people
Rashina Hoda, University of Auckland and interactions.1 Agile methods
retain the rigor of engineering pro-
Norsaremah Salleh, International Islamic University Malaysia cesses and best practices while better
helping both stakeholders and soft-
John Grundy, Monash University ware engineers build, deploy, and
maintain complex software.
Agile has now become a major
// This article provides a historical overview software engineering discipline in
of agile’s main focus areas and a holistic both practice and research. Formally
introduced through a set of four core
synthesis of its trends, their evolution values and 12 principles laid out in
over the past two decades, and agile’s the Agile Manifesto, 2 agile is now
the mainstream software develop-
current status and likely future. // ment method of choice worldwide.1

Agile in Practice
The latest State of Agile survey, the
FPO largest and longest-running survey
of its kind, reports 97% of respon-
dents’ organizations practicing agile
somewhere within their organiza-
tion in 2018,1 compared to 84% in
ORIGINALLY, COMPUTER SOFT- approach—software engineering was the first survey in 2007.3 Likely to be
WARE was written in an ad hoc developed as a discipline to provide en- completed by those predisposed to
manner. The programmers often gineering rigor to the profession. In the agile, the latest survey also showed
had no formal training but great do- ’70s, ’80s, and early ’90s, the growth 52% having all or more than half of
main knowledge and aptitude, most of software systems, range of domains their teams practicing agile.
commonly using large-scale non- of applications, number of developers, Scrum increased its prominence
networked machines and lacking a advent of the web, and diverse range as the most popular agile method,
common set of principles and prac- of challenging software engineering from 40% reported in the first survey
tices. This process was really more problems resulted in a set of printci- in 2007 to 56% on its own and 70%
akin to a cottage industry than an ples, methods, practices, and tools to when combined with other methods
engineering discipline. assist the engineering of such systems. in 2018. At the same time, Extreme
Subsequently—owing to all the In this move to complex pro- Programming (XP) lost ground
expected problems of such an cesses, project management, tools, from being the second most popular

2 I E E E S O F T WA R E | PUBLI S HED BY THE IEEE COMPUTER SO CIE T Y 0 74 0 -74 5 9 / 1 8 / $ 3 3 . 0 0 © 2 0 1 8 I E E E

method (23%) to being used in com-
bination with Scrum at 6%. Mean- 8
while, Kanban on its own (5%) and in
combination with Scrum (Scrumban,
8%) replaced DSDM (dynamic sys-
tems development method, 8%). In 6
another new development, 71% of

No. of reviews and studies

organizations report planning or in-
vesting in DevOps initiatives now.
In the first survey, reported con- 4
cerns with adoption centered around
lack of up-front planning, documen-
tation, and predictability, and loss
of management control. In the 2018
2
survey, the top challenges reported
in adopting and scaling agile orga-
nizational culture were general orga-
nizational resistance to change and
0
inadequate management support.
s

y
GSE

ods

cts

MI

tion

ility

g
s
One of the most interesting find-
tice

bilit

tem

erin
CM
spe
eth

l ag
dop
nd

ings is that an overwhelming 84%

rac

usa

sys

ine
nd
ial a
le m
le a

ona
le a
le p

eng
le a
nd

ded
of organizations are still maturing in
Agi

soc

zati
Agi

Agi
le a
Agi

Agi

line
bed

their agile practice. This highlights

ani
and
Agi

em

uct
continued opportunities for agile re- Org
an

rod
nd
Hum

search on the challenges of agile adop-

le a

re p
Agi

tion and practice.

twa
Sof

Agile Research
The phenomenal growth of agile FIGURE 1. Systematic literature reviews on agile topics.5 GSE 5 global software
practice is mirrored by agile research engineering, and CMMI 5 Capability Maturity Model Integration.
becoming a significant subdiscipline
of software engineering in the last conferences such as the International reviews and mapping studies, captur-
two decades and continuing today. Conference on Software Engineering ing two decades of agile research.5
In April 2018, a search for the key- (ICSE) and Foundations of Software We identified 10 key agile research
words “agile software development” Engineering (FSE) and in numerous areas: agile adoption, agile methods,
in Google Scholar for a period up to other reputable conferences. In addi- agile practices, human and social as-
2001 produced just over 13,000 re- tion, the rise and sustained growth pects, agile and GSE (global software
sults. The same search led to over of agile research has been chronicled engineering), agile and usability, agile
260,000 results for today. by 19 years of the International Con- and CMMI (Capability Maturity
Agile research has featured prom- ference on Agile Software Develop- Model Integration), organizational
inently in many premier software ment (XP) and 15 years of the North agility, agile and embedded systems,
engineering periodicals, including American Agile Conference (Agile), and software product line engineer-
IEEE Transactions on Software En- two of the largest dedicated annual ing, summarized in Figure 1.
gineering, IEEE Software, Empiri- agile conferences, and by numerous The agile-practices area, which
cal Software Engineering, Journal regional agile conferences and events covered topics such as test-driven
of Systems and Software, and Infor- around the world. development, metrics, effort esti-
mation and Software Technology. We conducted a research retro- mation, and requirements, was the
Agile research has also been published spective in the form of a tertiary most significant research area, with
in flagship software engineering study4 of 28 systematic literature seven systematic reviews. Agile and

S EP TEMBER /OCTOBER 2018 | I E E E S O F T WA R E 3

 FOCUS: SOFTWARE ENGINEERING’S 50TH ANNIVERSARY

Usability
RE

DevOps AI End-user
Knowledge management development
information systems

Internet of Things
Human aspects CMMI Product lines big data
adoption practices AR/VR

Early 2000s Late 2000s Mid-2010s Early 2020s

Late 1990s Mid-2000s Early 2010s Late 2010s

Large-scale agile

Methods Game design Microservices

Security
Global SE
Architecture
Safety-critical
systems

Embedded systems

Establishing Fundamentals Integrating with SE Beyond Development Beyond Software Emerging Technologies

FIGURE 2. The emergence of trends in agile software development, based on the first relevant publications in the IEEE and ACM
digital libraries. SE 5 software engineering, RE 5 requirements engineering, and AR/VR 5 augmented reality or virtual reality.

usability had the second highest the numerous research studies under Indicative papers charting this time-
number of reviews, five, and focused that banner. line can be found at http://dx.doi
on topics such as integrating the user .org/10.21227/H2T08T.
experience with agile. Given the role Agile Evolution The early days of agile saw ex-
of human–computer interaction in There is little consensus among indus- ploration of fundamental agile con-
maintaining a focus on engineering trial reports and sources on the emer- cepts such as agile methods,6 agile
for people and its synergies with agile gence of particular trends in agile adoption, agile practices, and the
software development, this is not sur- software development over time. human and social aspects, combined
prising. This was followed by agile However, the first relevant publica- into the Establishing Fundamentals
and GSE, with four reviews. tions in each of these areas are well trend (see the dark-green elements in
Agile education, an active and vi- documented in some of the larg- Figure 2). For example, the single
brant research area, is not included est publication archives and digital largest review (with 333 papers) was
in our study, as we focus on indus- libraries—e.g., the IEEE and ACM on the role of communication, a hu-
trial research. Another significant digital libraries. We used the infor- man and social aspect fundamen-
agile research area is pair program- mation from those digital librar- tal to agile software development.
ming, one of the most popular XP ies to devise a timeline illustrating With the latest State of Agile survey
practices. That area deserves a sec- the rise and evolution of agile soft- reporting 84% of organizations to
ondary review to collate and present ware development (see Figure 2).1–4 be still maturing in agile,1 many of

4 I E E E S O F T WA R E | W W W. C O M P U T E R . O R G / S O F T W A R E | @ I E E E S O F T WA R E
these fundamental issues continue to would seem a natural fit. However, these domains rapidly. For example,
be relevant in practice. Similarly, the many technical, sociotechnical, and this rapid advance is occurring in the
agile research community continues organizational challenges present automotive industry, as it becomes
to call for establishing firmer theo- themselves. When and how should more and more software-intensive
retical foundations of agile research, customer feedback be captured and with the advent of autonomous-
keeping this trend very much alive. actioned, and when should changes vehicle technologies.
The Integrating with Software be rolled out? What about software Finally, in the late 2010s, we
Engineering trend (see the magenta deployed across different organiza- see an interest in exploring the ten-
elements in Figure 2) saw synergies tions and user groups with different sions and synergies between agile and
explored between agile software requirements? And when software micro­services. Emerging microservice-
development and some of the more infrastructure changes significantly, based architectures take software by
well-established software engineer- how is continuously deployed soft- composition to a new level. This is
ing concepts and subdisciplines, ware effectively tested? What is the impacting software design and de-
such as usability, requirements en- impact of DevOps transformations ployment and raises questions such
gineering, software security, GSE, on agile practices? as, how does a team balance its agile
software architecture, and safety- Another significant trend involved practices with emerging micro­service
critical systems, in the early and extending agile Beyond Software, its architectures that require some level
mid-2000s. Other new topics in this original domain, and into related dis- of design up front?
trend emerged over time, such as in- ciplines such as knowledge manage-
tegrating agile and CMMI in the late ment and information systems in the Agile in the Future
2000s and agile and game design early 2000s. Given its central focus With current advancements in tech-
in early 2010s. Most of these topics on human and social aspects, agile nologies such as the Internet of
have continued to be popular, and has brought the complementary dis- Things (IoT), a wide range of devices
some—in particular, security and ciplines of software engineering and are being integrated into systems,
safety-critical systems—have wit- information systems closer than vast amounts of big data are becom-
nessed a strong surge in renewed ever before. Software engineering ing available for analysis, various
interest with recent technological ad- has gained from the theoretical ro- augmented- and virtual-reality sys-
vancements such as blockchains and bustness of information systems re- tems are being developed, and in-
cryptocurrencies. search, and the information systems telligent solutions are increasingly
Small, colocated teams, with an field has gained from the practical expected. At the same time, these
onsite or easily available customer, relevance of software engineering emerging technologies have renewed
an emphasis on programming and studies on agile topics. interest and opened new possibili-
early testing, and frequent feedback Agile also spread into closely re- ties in exploring the full potential
on iterative delivery of working lated areas such as embedded sys- of not-so-new paradigms such as AI
software, marked the original agile tems, starting in the mid-2000s, and and end-user development,8 going
sweet spot.7 The mid-2000s saw product line engineering, starting in forward in the 2020s.
agile step outside its comfort zone, the mid-2010s. Traditionally, these We predict a strong role for agile
scaling beyond the confines of small domains have had their own pro- software development in partner-
development teams into large-scale cesses, practices, measurements, and ing with and enabling the Emerging
agile, applying hybrids of agile soft- team cultures. Embedded systems Technologies trend in the foreseeable
ware development at the intrateam traditionally have been dominated future. We expect a number of ques-
level and traditional planning ap- by engineers using waterfall-style tions to be explored:
proaches at the interteam level. processes heavy on planning, docu-
Once again, in the mid-2010s, mentation, measurement, and model- • How will agile practices enable
agile ventured Beyond Development driven tool support. AI-based software engineering?
to acknowledge operations along- Applying agile software develop- In the last three years, there has
side development, through DevOps. ment philosophies, practices, and been a large increase in publica-
Continuous delivery and continuous cultures to these domains is challeng- tions on new AI-based software
feedback from users to developers ing. And yet, agile has advanced into tools. Conversely, how can AI be

S EP TEMBER /OCTOBER 2018 | I E E E S O F T WA R E 5

 FOCUS: SOFTWARE ENGINEERING’S 50TH ANNIVERSARY

used to augment agile software continue to increase. Can agile of its original comfort zone of small,
development? practices support the develop- colocated teams, agile has spread Be-
• Can agile improve data analytics ment of software by non-tech- yond Development into DevOps im-
and data sciences practices in the nical experts who nonetheless plementations and large-scale agile
way it has improved software want to quickly and effectively on the business and enterprise levels.
engineering? Is there an agile ap- improve and deploy parts of After more than two decades of
proach to data science that lever- the software systems they use? practice, organizations consider them-
ages practices similar to those of Where does an agile software selves still maturing in successfully
agile software engineering? development team end and end- deploying, improving, and contex-
• To fulfil the demand of the IoT, user developers of their own tualizing their agile practices to their
to what extent can agile methods (parts of a) software system teams, customers, and specific project
revolutionize the IoT industry? begin? conditions. Researchers continue to
How do hardware, embedded, • How do we successfully lever- study these issues and to help practi-
creative, visual, source, touch, age agile across multiple emer- tioners comprehend and address them.
and other interface designers gent technology domains and Another fundamental issue, that of
work effectively with, or indeed practices? For example, what managing change within a process
within, agile software develop- does “agile, secure DevOps for that actively promotes embracing
ment teams? IoT solutions may data-intensive intelligent sys- change, demands further inquiry.
be composed from hardware and tems” mean for researchers and However unsure as we may feel
software components. How do practitioners? about our collective maturity in agile
we produce more agile hardware software development, software en-

S
solutions? gineers are indeed looked upon as
• Security continues to be a major ince its inception in the the experts in agile practice by those
concern for developers and us- late 1990s, agile software in disciplines Beyond Software, such
ers. While agile practices and development has come to as embedded systems and product
continuous-deployment ap- dominate the latter half of the past lines. Agile practitioners can assist in
proaches theoretically allow for 50 years of software engineering. It agile transformations outside of soft-
quick fixes of emerging security started off with Establishing Fun- ware development—e.g., in human
issues, extensive security testing damentals such as agile adoption, resources, sales and marketing, proj-
before deployment is increas- methods, and practices, and human ect management, and R&D. They
ingly being required. Similarly, and social aspects. Then it moved can do this by abstracting out the
zero-day security threats can’t on to Integrating with Software En- lessons learned from agile transfor-
be fully designed or tested for, gineering topics and subdisciplines mation in software teams, applying
but an agile fix may not be ac- such as usability, requirements engi- them to new contexts, and helping
ceptable in many circumstances, neering, GSE, software architecture, adapt agile to fit new contexts.
either. How do agile methods CMMI, and game design. Of late, Finally, peeking into the future,
ensure that security require- agile development has seen renewed much give and take can be expected
ments are continually met? interest in security and safety-critical between agile and Emerging Tech-
• How can agile processes sup- systems and seems likely to move nologies such as the IoT, augmented
port the development of safety- into exploring synergies with AI and and virtual reality, big data services,
critical systems in increasingly end-user development. paradigms such as AI, and end-user
software-intensive autonomous Research has been directed at development.
vehicles, software-defined net- understanding how agile is made to
working, and robotics develop- work in practice within and along- References
ment and integration? side these preestablished software en- 1. 12th Annual State of Agile Survey
• End-user development of gineering paradigms. Barriers, areas Report, VersionOne, 2018; http://
complex software, whether by of conflict, synergies, strategies, and stateofagile.versionone.com.
coding, configuration, composi- workarounds have been researched 2. M. Fowler and J. Highsmith,
tion, or a mixture, is likely to and presented. Moving further out “The Agile Manifesto,” Software

6 I E E E S O F T WA R E | W W W. C O M P U T E R . O R G / S O F T W A R E | @ I E E E S O F T WA R E
 Development, vol. 9, no. 8, 2001, pp.

ABOUT THE AUTHORS

28–35; http://agilemanifesto.org.
3. Survey: “The State of Agile Devel-
opment,” VersionOne, 2007; http:// RASHINA HODA is a senior lecturer (associate professor)
stateofagile.versionone.com. and the founder of the Software Engineering Processes Tools
4. B.A. Kitchenham and S. Char- and Applications research group at the University of Auckland.
ters, Guidelines for Performing Her research interests include agile software development,
Systematic Literature Reviews in cooperative and human aspects of software engineering,
Software Engineering, tech. report grounded theory, and serious-game design. Hoda received a
EBSE-2007-01, Keele Univ. and Univ. PhD in computer science from Victoria University of Wellington.
of Durham, 2007. Contact her at [email protected].
5. R. Hoda et al., “Systematic Literature
Reviews in Agile Software Develop-
ment: A Tertiary Study,” Information NORSAREMAH SALLEH is an associate professor in the
and Software Technology, vol. 85, Department of Computer Science, International Islamic
2017, pp. 60–70. University Malaysia. Her research interests include empirical
6. M. Aoyama, “Web-Based Agile Soft- software engineering (SE), evidence based SE, the human and
ware Development,” IEEE Software, social aspects of SE, and computer science and SE education.
vol. 15, no. 6, 1998, pp. 56–65. Salleh received a PhD in computer science from the University
7. D. Reifer, F. Maurer, and H. Erdog- of Auckland. Contact her at [email protected].
mus, “Scaling Agile Methods,” IEEE
Software, vol. 20, no. 4, 2003, pp.
12–14. JOHN GRUNDY is the senior deputy dean of Monash
8. J. Segal and C. Morris, “Developing University’s Faculty of Information Technology. His research
Scientific Software,” IEEE Software, interests include automated software engineering, software
vol. 25, no. 4, 2008, pp. 18–20. tools, human-centric software engineering, visual languages,
software architecture, software security engineering, and user
interfaces. He’s a Fellow of Automated Software Engineer-
ing and of Engineers Australia. Contact him at john.grundy@
Read your subscriptions
monash.edu.
through the myCS
publications portal at
http://mycs.computer.org

S EP TEMBER /OCTOBER 2018 | I E E E S O F T WA R E 7

View publication stats