To Agile, or Not To Agile
To Agile, or Not To Agile
Abstract
arXiv:1704.07469v1 [cs.SE] 24 Apr 2017
Since the Agile Manifesto, many organizations have explored agile development methods
to replace traditional waterfall development. Interestingly, waterfall remains the most widely
used practice, suggesting that there is something missing from the many “flavors” of agile
methodologies. We explore seven of the most common practices to explore this, and evaluate
each against a series of criteria centered around product quality and adherence to agile prac-
tices. We find that no methodology entirely replaces waterfall and summarize the strengths
and weaknesses of each. From this, we conclude that agile methods are, as a whole, un-
able to cope with the realities of technical debt and large scale systems. Ultimately, no one
methodology fits all projects.
1 Introduction
Agile software development, a term introduced in 2001 in famous Agile Manifesto [1], took the
world by a storm and quickly became, if not the most popular then certainly the most fashionable,
development style. However, there is a lot of misunderstandings surrounding the agile development
style. As teams adopt and change this methodology they raise questions about the limitations [2]
and applicability [3] of this approach.
In this paper we explore the state-of-the-art methodologies which are widely used by software
companies today with a focus on how agile methods have been altered to facilitate the need for
planning. We compare different software development practices, both agile and traditional, in an
attempt to find an answer to the questions like "Why does the waterfall persist?" and "What
makes popular agile methods popular?"
This report is structured in the following way: Section 2 consists of the historical idealogical
foundation of agile development, describing of the most famous agile method, extreme program-
ming, and a quick discussion of architectural agility. Section 3 describes the evaluation methodol-
ogy that we used to compare various methods and the rationale behind our evaluation framework.
Section 4 contains descriptions of each studied methodology as well as short discussions on how
they function in real-world circumstances. This section includes the evaluation results of seven of
the most popular software development methods. Finally, ins section 5 we draw conclusions and
outline possible future work.
2 Agile
In this section we will describe the original agile approach including its origins and its current
state. This will act as a baseline on which we will compare modern reformulations of these core
principles. Furthering this baseline we discuss Extreme Programming and Architectural Agility.
The former is the principle agile methodology, the latter is a more modern approach with attempts
to incorporate architectural design into the process. Although neither method is extraordinarily
popular today, these serve to bookend the extremes of the agile development described in latter
sections.
1
changing requirements” [4]. This approach is motivated by what he calls the “runaway inflation” of
the software development processes. This inflation begins when a team fails to deliver a product on
time, and begin over-engineering a methodology to attempt to improve their performance. If the
new methodology also leads to a failed project, an even more complicated replacement may be put
in place. From there the cycle continues. As a result of this inflation, many companies were stuck
in a quagmire of increasingly complex processes. A group of industry experts met and decided to
create a set of values and principles that would help companies overcome these problems calling
themselves the “Agile Alliance” [5].
There are many methods developed around the principles described in the agile manifesto, and
it is important to separate the principles of agile from the methods and practices of agile software
development (e.g., SCRUM). Failing to make this separation often leads to blindly following the
word of practices of a certain method and ignoring the spirit of them. This can result in danger-
ous and inefficient cargo-cult mentality and in failure to deliver a good product. As a baseline
methodology, we begin by exploring extreme programming (XP) before moving onto the seven
most popular modern design methods.
2
The end goal of architectural agility is an increased focus on the dependencies between user
stories. This begins with an additional category added to the traditional agile release planning
board, and ends with a dependency structure matrix. By focusing on these dependencies, as well
as quality attributes, a team should be able to quickly identify the software components which are
most central to the proposed system. This allows a team to understand which elements require
fleshed-out architectures in the upcoming iteration.
Ultimately, it would seem that the project management structure proposed by Brown et. al.
attempts to find a middle ground between the speed of agile, and the patience of waterfall. By
taking the time each iteration to look a couple steps ahead, this method should be able to improve
agile teams with complex software architectures.
3 Evaluation Methodology
In the following sections we detail the basis for our methodology evaluation and go on to explain
our approach and the specifics of our evaluation criteria.
3
to a specific organizational structure or design selection, but instead will instead influence these
topics in an indirect manner. For example, traditional agile with scrum [15] tends to produce very
“wide” team hierarchies where all developers have very similar standings when making decisions.
Additionally, we evaluate the agility of each methodology in an effort to identify if a strong corre-
lation exists between software quality and agility. We do so in a similar manner, also described in
the following section.
In addition to our quality criteria, we also present a series of criteria for evaluating the agility of
a given software method. We do so in order to detect if there is a measurable correlation between
method quality and method agility. This series of questions is directly inspired from the set of
project characteristic discriminators defined in [14].
4
6. Relies heavily on customer feedback The method includes customers at multiple points
in the design process, as opposed to a single requirement gathering phase.
7. Susceptible to unforeseen risks The method is weak to unforeseen major risks or compli-
cations which were not foreseen. Waterfall, for example, spends substantial time documenting
these risks and is therefore less susceptible.
4 Evaluated Methods
Vijayasarathy et. al. in [16] conducted a survey in which they asked 153 developers to describe
their software development process. One of the most important results of this study is a listing
of the most commonly used software development practices as of 2016. We use these results to
inform our methodology reviews, allowing us to review the most widely used methodologies. The
list of software methodologies which address in later sections follows the same order of popularity.
1. Waterfall
2. Agile Unified Process
3. Scrum
4. Test-Driven Development
5. Rapid Application Development
6. Joint Application Development
7. Feature-Driven Development
4.1 Waterfall
In [17], Royce describes the process which would eventually become known as waterfall develop-
ment. This method includes seven steps which are completed in order and should result in stable
software as a result. These steps include, system requirements, software requirements, analysis,
program design, coding, testing, and operations. Royce notes that these steps do not typically form
a linear flow from one to the next, instead many backward pointing edges between these states may
exist. For example, as testing reveals bugs, a project might need to return to the program design
phase. Additionally, these main phases represent sub-trees within them selves. For example, Royce
points out that whole departments might be assigned to just the testing phase.
Waterfall typically requires a long time of requirements gathering and project planning before
any code is written. This process attempts to identify the full set of project requirements. Other
methods, such as [18] by Benington and [19] by Ramamoorthy also incur a large up front require-
ment gathering cost and were published around time same time. Since then, numerous other flavors
of waterfall development have been introduced such as the iterative model described in [20], which
closely resembles the “iterative waterfall method.”
It is unlikely that those survey respondents who reported using waterfall in 2016 followed the
original specification without modification [16]. Waterfall has changed a lot over the years as
developers have began experimenting with a more iterative approach. For the purposes of our
analysis, we consider all of the above waterfall variants together, including iterative agile. As a
whole, these methods are characterized by large upfront costs in requirement gathering, inexpensive
bug fixing, accurate time estimates, and inflexibility to changing or uncertain requirements. It
seems clear that there are still many projects which are sufficiently well-defined to facilitate such
a starched methodology.
5
AUP applies agile techniques, such as Test Driven Development (TDD) and Agile Model Driven
Development (AMDD) to improve the productivity of a team. It acts as a unifying framework for
multiple agile methodologies and processes.
The AUD approach is “serial in the large” and “iterative in the small”. It consists of four
subprocesses or work flows: Modeling, Implementation, Testing and Deployment, each of which
goes through four phases: Inception, Elaboration, Construction and Transition.
The principles of AUD emphasize simplicity. The simplest possible tools are preferred to
complex products, minimal viable documentation is preferred to detailed documentation. It’s
aiming to prioritize high-value activities over trying to define everything that can possibly happen
in a lifespan of a project.
Success of Agile Unified Process over other agile methodologies can be attributed to its flexi-
bility, as well as to its simplicity.
4.3 Scrum
Scrum [15] is one of the most popular frameworks for implementing agile. Its defining characteristic
is commitment to short iterations of work.
In scrum, a product is developed through a series of fixed-length iterations, called sprints, that
allow for software updates at a regular cadence. The part of scrum that is the most attractive to
a team is the idea that some call “continuous inspiration” [15]. Team members are motivated by
tangible, visible progress at the end of each sprint, as well as by the ability to "show off" during
the sprint demo.
Sprint consists of four “ceremonies” [22]:
1. Sprint planning – team meeting where the next sprint is outlined.
2. Daily stand-up – daily 15 minute meeting for the team to sync up.
3. Sprint demo – weekly meeting where teammates showcase what they will ship during the
week.
4. Sprint retrospective – weekly analysis of what went wrong and what went right during the
previous week.
Scrum has three specific roles: product owner, scrum master, and the development team.
The product owner works with the business requirements and gives requirements to the team.
The scrum master coaches the team and make sure the team observes scrum practices. The
development team works closely together in an open and collaborative manner, talking regularly
at scrum meetings. A potential point of failure, of course, comes from the quality of the product
manager who is the sole point of contact between the development team and the customers.
6
A more recent study [26] looks to explore the effect TDD has in the modern software production
process. By aggregating over a thousand papers evaluating TDD, Bissi et. al. note that most TDD
studies report an increase in code quality, and a decrease in productivity. Interestingly the study
also found that almost every TDD project was written in C++ or Java. This speaks to the limited
scope of TDD, and how tied it is to specific testing frameworks.
7
are grouped into sets, and each set describes a two week iteration [39]. FDD promotes very modular
architectures, as each feature should, in a perfect world, be concurrently developed.
Typically, FDD start with UML modeling. As features are planned, the necessary software
components and their connections should grow organically. Arguably, this is the opposite of typ-
ical architectural design processes where a large system is split into encapsulated submodules.
Additionally, FDD defines a hierarchy of developers, appointing some to be “chief programmers”
responsible for small teams and planning sprints [14]. These leaders act as a way to fluidly create
and dismantle teams around features, as well as to regulate the quality of software produced in an
iteration. Interestingly though, as opposed to many other agile methods, FDD actively discourages
refactoring. Instead the development cycle focuses on added value to an extreme. After-all once a
feature is present in the software, the customer is satisfied [40].
It would seem that FDD is a very natural, albeit naive, approach to software design. If a team
is good and spends adequate time each iteration on the closing code review, it is possible that
code quality may not deteriorate too quickly. It does appear that the method requires substantial
upfront cost in the FDD version of requirements gathering in order to put together an initial plan
that will not change too drastically during the lifetime of the project, otherwise an FDD project
would require deep refactoring. Lastly, FDD fails to account for many non-functional qualities
which typically depend on interactions between features. For example, FDD could easily produce
a system had to both produce a shopping cart as well as track purchasing data, but would be ill
equipped to ensure a latency requirement between user actions and the tracking system.
Methodology
Waterfall AUP Scrum TDD RAD JAD FDD
Requirements flexibility No Yes Yes Yes Yes Yes No
Requirements fulfillment guarantee Yes Yes Yes No 1 Yes No Yes
Quality Criteria
5 Conclusion
In this paper, we have presented a comparative analysis of a number of the most popular software
development methodologies. The results show that each of the “agile” methodologies has an im-
portant basic criteria that it doesn’t cover, such as technical debt control or cost estimation. In
smaller projects these issues can be ignored since the costs are small and the potential for technical
1 Only Functional
2 via TDD
3 via weekly demos
4 via AMDD
5 Accomplished with lengthy documentation and planning.
6 Via continuous user engagement in the development process
7 Works for all sizes
8
debt is limited. For really large and complex projects, these limitations render the “agile” methods
inapplicable.
Therefore, we believe that these innate qualities of “agile” methods make them applicable only
for a certain subset of the problems: small and solvable in small teams. However, the agile methods
that we’ve looked at do not scale well. For large and sprawling projects, a paradigm that over=-
emphasizes many person-to-person interactions stops working.
We find that these limitations explain the persistence of the traditional waterfall approach.
Despite being bulky and prone to generating redundant documentation, it is a tried and true
method that has proved to be able to deliver working products. It is understandable that many
managers decide to err on the side of caution, especially considering the fact that benefits of using
many agile methods are at best unclear.
We find it unsurprising that Vijayasarathy noticed over 45% of software teams are using a
hybrid approach [16]. As we show in our table above, no one methodology properly addresses all
important aspects of the software design process. By using a combination of approaches, greater
results can be achieved. For example by using waterfall for project planning, scrum for short term
goals, and TDD for a guarantee on software correctness, a team can mitigate the downfalls present
in each method while highlighting their strengths.
References
[1] Agile Manifesto. [Online]. Available: http://agilemanifesto.org
[2] D. Turk, R. France, and B. Rumpe, “Limitations of agile software processes,” arXiv preprint
arXiv:1409.6600, 2014.
[3] A. Qumer and B. Henderson-Sellers, “An evaluation of the degree of agility in six agile methods
and its applicability for method engineering,” Information and software technology, vol. 50,
no. 4, pp. 280–295, 2008.
[4] R. C. Martin, Agile software development: principles, patterns, and practices. Prentice Hall
PTR, 2003.
[5] A. Alliance, “Agile manifesto,” Online at http://www. agilemanifesto. org, vol. 6, no. 1, 2001.
[6] K. Beck, Extreme programming explained: embrace change. addison-wesley professional,
2000.
[7] ——, “Embracing change with extreme programming,” Computer, vol. 32, no. 10, pp. 70–77,
1999.
[8] M. C. Paulk, “Extreme programming from a cmm perspective,” IEEE software, vol. 18, no. 6,
pp. 19–26, 2001.
[9] K. Beck and M. Fowler, Planning extreme programming. Addison-Wesley Professional, 2001.
[10] N. Brown, R. Nord, and I. Ozkaya, “Enabling agility through architecture,” DTIC Document,
Tech. Rep., 2010.
[11] B. A. Kitchenham, “Evaluating software engineering methods and tool part 1: The evaluation
context and evaluation methods,” ACM SIGSOFT Software Engineering Notes, vol. 21, no. 1,
pp. 11–14, 1996.
[12] R. Sorensen, “A comparison of software development methodologies,” CrossTalk, vol. 8, no. 1,
pp. 10–13, 1995.
[13] R. H. Thayer, A. B. Pyster, and R. C. Wood, “Major issues in software engineering project
management,” IEEE Transactions on Software Engineering, no. 4, pp. 333–342, 1981.
[14] M. Awad, “A comparison between agile and traditional software development methodologies,”
University of Western Australia, 2005.
[15] K. Schwaber and M. Beedle, Agile software development with Scrum. Prentice Hall Upper
Saddle River, 2002, vol. 1.
9
[16] L. R. Vijayasarathy and C. W. Butler, “Choice of software development methodologies: Do
organizational, project, and team characteristics matter?” IEEE Software, vol. 33, no. 5, pp.
86–94, 2016.
[17] W. W. Royce et al., “Managing the development of large software systems,” in proceedings of
IEEE WESCON, vol. 26, no. 8. Los Angeles, 1970, pp. 1–9.
[18] H. D. Benington, “Production of large computer programs,” Annals of the History of Com-
puting, vol. 5, no. 4, pp. 350–361, 1983.
[19] C. Ramamoorthy, W. Tsai, T. Yamaura, and A. Bhide, METRICS GUIDED METHODOL-
OGY. IEEE, 1985, pp. 111–120.
[20] N. M. A. Munassar and A. Govardhan, “A comparison between five models of software engi-
neering,” IJCSI, vol. 5, pp. 95–101, 2010.
[21] S. Ambler, Agile modeling: effective practices for extreme programming and the unified process.
John Wiley & Sons, 2002.
[22] S. Alliance, “Learn about scrum,” Scrum Alliance, 2016.
[23] D. Janzen and H. Saiedian, “Test-driven development concepts, taxonomy, and future direc-
tion,” Computer, vol. 38, no. 9, pp. 43–50, 2005.
[24] L.-O. Damm, L. Lundberg, and D. Olsson, “Introducing test automation and test-driven
development: An experience report,” Electronic Notes in Theoretical Computer Science, vol.
116, pp. 3–15, 2005.
[25] E. M. Maximilien and L. Williams, “Assessing test-driven development at ibm,” in Software
Engineering, 2003. Proceedings. 25th International Conference on. IEEE, 2003, pp. 564–569.
[26] W. Bissi, A. G. S. S. Neto, and M. C. F. P. Emer, “The effects of test driven development
on internal quality, external quality and productivity: A systematic review,” Information and
Software Technology, vol. 74, pp. 45–54, 2016.
[27] J. Martin, Rapid application development. Macmillan Publishing Co., Inc., 1991.
[28] I. Jacobson, G. Booch, J. Rumbaugh, J. Rumbaugh, and G. Booch, The unified software
development process. Addison-wesley Reading, 1999, vol. 1.
[29] A. Gerber, A. Van Der Merwe, and R. Alberts, “Practical implications of rapid development
methodologies.” Computer Science and Information Technology Education Conference, 2007.
[30] P. Beynon-Davies, C. Carne, H. Mackay, and D. Tudhope, “Rapid application development
(rad): an empirical review,” European Journal of Information Systems, vol. 8, no. 3, pp.
211–223, 1999.
[31] H. Mackay, C. Carne, P. Beynon-Davies, and D. Tudhope, “Reconfiguring the user: using
rapid application development,” Social studies of science, vol. 30, no. 5, pp. 737–757, 2000.
[32] E. Carmel, R. D. Whitaker, and J. F. George, “Pd and joint application design: a transatlantic
comparison,” Communications of the ACM, vol. 36, no. 6, pp. 40–48, 1993.
[33] R. Hirschheim and M. Newman, “Symbolism and information systems development: myth,
metaphor and magic,” Information Systems Research, vol. 2, no. 1, pp. 29–62, 1991.
[34] B. Ives and M. H. Olson, “User involvement and mis success: A review of research,” Manage-
ment science, vol. 30, no. 5, pp. 586–603, 1984.
[35] E. J. Davidson, An exploratory study of joint application design (JAD) in information sys-
tems delivery. Center for Information Systems Research, Sloan School of Management, Mas-
sachusetts Institute of Technology, 1993.
[36] S. A. Becker, E. Carmel, and A. R. Hevner, “Integrating joint application development (jad)
into cleanroom development with icase,” in System Sciences, 1993, Proceeding of the Twenty-
Sixth Hawaii International Conference on, vol. 3. IEEE, 1993, pp. 13–21.
10
[37] J. Hunt, “Feature-driven development,” Agile Software Construction, pp. 161–182, 2006.
[38] M. Benoit, R. Anthony, and L. B. Wee, “Feature-driven development,” 1999.
[39] S. R. Palmer and M. Felsing, A practical guide to feature-driven development. Pearson
Education, 2001.
[40] S. Khramtchenko, “Comparing extreme programming and feature driven development in aca-
demic and regulated environments,” Feature Driven Development, 2004.
11