Master of Science in Software Engineering

June 2017

An empirical study of Agile planning critical

success factors

Di Liu
Zhichao Zhai

Faculty of Computing
Blekinge Institute of Technology
SE-371 79 Karlskrona Sweden
Th
isthe
sisissubmi
ttedtotheFacul
tyofComput
ingatBle
kingeIn
st
itu
teofTe
chn
ologyin
pa
rt
ialful
fi
llmentofther equi
rement
sforthedegreeofMas
terofS c
ien
ceinSof
twar
e
En
gin
eer
ing. Theth
esi
sisequi
val
entto20w
eek
sofful
ltimes
tud
ies
.

Con
tactInformat
ion:
Au
thor
(s)
:
D
iL i
u
E-m
ai
l:6 54459373@qq .com
li
di15@student
.bth.
se
Zh
ichaoZ h
ai
E-m
ai
l: 350250301@qq .com
zhz
a 15@st
ud en
t.bth
.se

U
niv
ers
itya
dvi
sor
:

R
ica
rdo B
ri
t
to
D
epar
tmen
to fSo
ftw
areE
ngi
nee
ring

Fac
ultyofCom pu
ti
ng In
te
rne
t :www.bt
h.s
e
B
lek
i ng
eI n
st
it
uteofT ec
hno
logy Ph
one :+46455385000
SE-37179K a
rlsk
rona
,Sw e
den Fa
x :+46455385057

Ⅰ i
i
 ABSTRACT

Context. With the popularity of Agile methods, many studies about Agile software development has
been done by researchers. Among the phases in Agile software pro jects, planning is crit ical because it
provides an overview o f the pro ject and a guiding of future wo rk. In addit ion, success factors are also
mandatory to the success of Agile software development. The current literature focus on the success
factors during the whole lifecycle rather than planning phase, and they don’t make a n in-depth
analysis on the factors. In this thesis, we perform an emp irical study to deeply study the critical
success factors at agile planning phase.
Objecti ves. The main aim of our research is to identify the critical success factors at Agile planning
phase and challenges associated with each factor. We list four objectives to support our main aim.
First is to investigate the factors that are mandatory to the success of Agile software development at
planning phase. Second is to investigate the challenges associated with each factor. Third is to find out
the ways to address these challenges . The last is to identify the consequence of not ensuring these
factors.
Methods. We employed two research methods: systematic mapping and survey. Systematic mapping
is used to identify the crit ical success factors of entire lifecycle in current literature. To find crit ical
success factors at agile planning phase and make in -depth analysis, we conducted a survey based on
an online questionnaire. The online questionnaire was consisted of open-ended questions and was sent
to respondents who have experience on Agile development.
Results. Through systematic mapping, we identified 13 papers and 47 crit ical success factors for
Agile software development. We also made a frequency analysis for these factors and they will be the
effective evidence to support the results of survey. Through the survey, we identified 13 crit ical
success factors at agile planning and made an in-depth analysis for these 13 factors. These 13 factors
are divided into two categories: people factor (individual-level, team-level) and process factor.
Through the contrastive analysis of mapping results and survey results, we found that 7 factors of
survey results are same or similar with some factors shown in mapping. The other 6 factors of survey
are first shown.
Conclusions. The factors proposed in this thesis are proved that they are important to the success of
the project at planning phase. Failure to consider these critical success factors may lead to inefficient
planning and even result in the failure of the whole project. The challenges and corresponding
solutions can help organizat ions well manage these critical success factors. In conclusion, these
detailed descriptions of critical success factors can be used as a guideline to help people increas e the
chance of successfully developing software with high quality and low cost in practice.
Keywords: Agile software development, critical
success factors, Agile planning phase

I
 ACKNOWLEDGEMENT
Firstly, we express our sincere thanks to our supervisor Ricardo Britto for his
guidance, supporting and trust on our thesis report. His patient guidance helps us
clarify the research direction, modify the research methods, and improve the quality
of the thesis. He send our questionnaire to his friends which gives us great helpful on
completing our survey.

Secondly, we thank all people who participated in our survey by sharing their
lived experience to support our research. Because of their positive cooperation, we
collect the valuable answers.

We also would like to thank our examiner Dr. Jürgen Börstler. His comments on
our thesis topic and plan provide us a good start and guide for our continue work.

Last but not least, we express gratitude to our parents and friends who give us
support on the life and study. Without their support, we cannot complete our thesis.

II
 CONTENTS
ABSTRACT .................................................................................................................I

ACKNOWLEDGEMENT ........................................................................................ II

CONTENTS ..............................................................................................................III

LIST OF FIGURE..................................................................................................... V

LIST OF TABLE...................................................................................................... VI

1 INTRODUCTION............................................................................................... 7
1.1 AIM AND OBJECTIVES .................................................................................... 7
1.2 RESEARCH QUESTIONS ................................................................................... 8
1.3 EXPECTED OUTCOMES .................................................................................... 8
1.4 THESIS STRUCTURE ........................................................................................ 8
2 RELATED WORK ........................................................................................... 10
2.1 AGILE SOFTWARE DEVELOPMENT (ASD) .................................................... 10
2.2 AGILE METHODOLOGY.................................................................................. 11
2.2.1 DSDM....................................................................................................... 11
2.2.2 Extreme programming (XP)..................................................................... 12
2.2.3 Scrum ....................................................................................................... 12
2.2.4 Comparison .............................................................................................. 13
2.3 ASD LIFECYCLE ........................................................................................... 13
2.3.1 Overview of ASD lifecycle........................................................................ 13
2.3.2 Planning in ASD....................................................................................... 14
2.4 CRITICAL SUCCESS FACTORS ....................................................................... 15
2.5 GAPS OF CURRENT LITERATURE.................................................................... 16
3 METHODOLOGY ........................................................................................... 17
3.1 SYSTEMATIC MAPPING DESIGN .................................................................... 17
3.1.1 Motivation of selecting SMM ................................................................... 17
3.1.2 Search process ......................................................................................... 18
3.1.3 Study selection process ............................................................................ 19
3.2 SURVEY DESIGN ........................................................................................... 20
3.2.1 Motivation of selecting survey ................................................................. 20
3.2.2 Target population..................................................................................... 20
3.2.3 Sampling method...................................................................................... 20
3.2.4 Data collection method ............................................................................ 21
3.2.5 Data analysis method............................................................................... 23
4 RESULTS .......................................................................................................... 25
4.1 RESULT OF SYSTEMATIC MAPPING ................................................................ 25
4.1.1 Classification of papers ........................................................................... 25
4.1.2 General results ......................................................................................... 27
III
 4.1.3 Summary of CSFs..................................................................................... 29
4.2 RESULT OF SURVEY ...................................................................................... 31
4.2.1 Demographic of respondents ................................................................... 31
4.2.2 CSFs statistics .......................................................................................... 33
5 ANALYSIS AND DISCUSSION ..................................................................... 35
5.1 ANALYSIS OF DATA COLLECTED FROM SYSTEMATIC MAPPING...................... 35
5.2 ANALYSIS OF DATA COLLECTED FROM ONLINE QUESTIONNAIRE .................. 36
5.2.1 Analysis of each factor ............................................................................. 36
5.2.2 Summary of collected factors................................................................... 48
5.3 DISCUSSION .................................................................................................. 50
5.3.1 Relations of survey results ....................................................................... 50
5.3.2 Comparison of two methods results ......................................................... 52
6 CONCLUSION AND FUTURE WORK ........................................................ 55
6.1 CONCLUSION OF RESEARCH QUESTIONS ........................................................ 55
6.2 CONTRIBUTION OF THIS THESIS ..................................................................... 56
6.3 VALIDITY THREATS ...................................................................................... 57
6.4 FUTURE WORKS ............................................................................................ 58
REFERENCES ......................................................................................................... 59

APPENDIX A. SURVEY QUESTIONNAIRE ...................................................... 62

APPENDIX B............................................................................................................ 63

IV
 LIST OF FIGURE
Figure 1.1 The structure of this thesis .......................................................................... 8
Figure 2.1 DSDM lifecycle ........................................................................................ 11
Figure 2.2 XP lifecycle............................................................................................... 12
Figure 2.3 Scrum lifecycle ......................................................................................... 12
Figure 2.4 The comparison for three methods ........................................................... 13
Figure 2.5 Agile software lifecycle ............................................................................ 14
Figure 3.1 Correspondence between research methods and research questions ........ 17
Figure 3.2 Searching process ..................................................................................... 18
Figure 3.3 Searching processes and results ................................................................ 19
Figure 3.4 Analysis processes .................................................................................... 23
Figure 4.1 Publication year of papers......................................................................... 26
Figure 4.2 Country of respondents ............................................................................. 31
Figure 4.3 Domain of the company........................................................................... 32
Figure 4.4 Years of experience working with agile software development............... 32
Figure 5.1 The answer of respondent about Expertise .............................................. 36
Figure 5.2 The answer of respondent about Joint understanding of uncertainty ....... 37
Figure 5.3 The answer of respondent about Productive team of good level .............. 38
Figure 5.4 The answer of respondent about Conciliate deadline between client and
team ..................................................................................................................... 39
Figure 5.5 The answer of respondent about Top engineers ....................................... 40
Figure 5.6 The answer of respondent about Communication .................................... 41
Figure 5.7 The answer of respondent about Priorities ............................................... 42
Figure 5.8 The answer of respondent about Coordination ......................................... 43
Figure 5.9 The answer of respondent about Level of details ..................................... 44
Figure 5.10 The answer of respondent about Team’s maturity.................................. 45
Figure 5.11 The answer of respondent about Team knowledge ................................ 46
Figure 5.12 The answer of respondent about Team collaboration ............................. 47
Figure 5.13 The answer of respondent about Requirement change ........................... 48
Figure 5.14 The relations of defined categories ......................................................... 50
Figure 5.15 The relations among CSFs collected by survey...................................... 51
Figure 5.16 The relations between mapping results and survey result ...................... 52

V
 L
ISTOFTABLE
Ta
ble3.1 Keyw ordsandsynonym s..
..
...
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
.18
Ta
ble3.2Searchingresul
tsofe a
chd a
t aba
se..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
.19
Ta
ble3.3Surveyqu est
ions.
..
..
...
..
..
..
..
..
...
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
.22
Ta
ble4.1Public
ationv enue
..
..
...
..
..
..
..
..
...
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
.26
Ta
ble4.2TheCSF sextrac
tedfrome achpaper..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
.27
Ta
ble4.3Thefrequ encyofeachCSF(F≥4 )..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
29
Tab
le4.4Th efrequencyofea chCSF(F ≤3 )..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
30
Tab
le4.5Th eoverviewo fdem og
raphicd a
taofrespondent
s..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
.33
Tabl
e4.6 Thesumm a
ryofCSF sb a
se dontheansw e
rsofsurvey.
..
..
..
..
..
..
..
..
..
..
..
..
..
.33
Ta
ble5.1Frequencyan dpercent
ageofe achCSF ..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
.35
Ta
ble5.2Categoryo feachCSF ..
..
..
..
..
...
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
.49

V
I
1 INTRODUCTION
Since the 1990s, Agile software development (ASD) as a development approach
has attracted more and more attention. Because it allows for dealing with rapidly
changing demands, ASD is widely accepted and applied by organizations [1]. This
attracts researchers to get an in-depth understanding so that it can be applied
smoothly and effectively. Through these studies, we can clearly know its value,
principles, development process, development method, and other detail parts. The
study field of ASD is very widely so that the opinions on each field may differ from
researches.
Considering the goal of all organizations that developing high-quality software
with less time and cost will never change [2], more and more people put their
attention on the development process. ASD decomposes the developme nt lifecycle
into four main phases: Planning, Iteration, Release, and Retirement [3]. During the
whole development process, planning can be divided into two parts: release plan and
iteration plan. Planning as the initial activity plays an important role to guide people
reducing the risk, gaining better decision making, conveying information and
satisfying customers [4]. Thus, it can be seen the cornerstone of the project
development.
Although many studies provide the knowledge and analysis of ASD, there are
still many problems in development practices, such as low quality of products and
low success chance of development. So researchers focus on the mandatory factors to
ASD to address these problems. To the best of our knowledge, we find that there are
some gaps in existing studies, the detail of gaps are presented in section 2.5.
Generally, most literature (e.g. [1] [2] [5] [21] [22] [23] [24] [25]) just collected
critical success factors (CSFs) of ASD without any in-depth analysis. Additional,
these papers focused on CSFs for whole development lifecycle of product rather than
planning phase. Thus, these CSFs are vague and broad so that it is difficult for people
to refer them to development practices. In order to fill up these gaps and increase the
success chance of ASD, we write this thesis. Different from existing studies, we pay
our attention to the CSFs that are mandatory to the success of ASD at planning
phases and make an in-depth analysis for them.
The remainder of this chapter is organized as follows: The details of aim and
objectives are presented in the section 1.2. The section 1.3 sets some research
questions (RQ) which can help us deeply understand the topic and easily meet the
aim and objectives. The section 1.4 gives the expected outcomes of each question.
The section 1.5 gives a picture to make the structure of our research visible.
1.1 AIM AND OBJECTIVES
The aim of the research is to identify the CSFs at Agile planning phase and
challenges associated with each factor. To achieve the aim of the research, we
formulated four objectives as follows:
Objective 1. To identify the CSFs which are mandatory to the success at Agile
planning. In order to increase the success chance, we want to identify CSFs at Agile
planning phase.
Objective 2. To find out the challenges which are associated with each factor. In
order to ensure the success factors, we want to identify the challenges that during the
process of ensuring factors.

7
Objective 3. To investigate the ways for addressing the proposed challenges
associated with each factor. In order to better manage the factors, we want to find
out the ways to address the challenges associated with each factor.
Objective 4. To identify the consequences if not ensuring these factors. In order to
emphasize the importance of success factors, we identify the consequences of not
ensuring these factors.

1.2 RESEARCH QUESTIONS

Based on the specific objectives, we proposed four corresponding research
questions for each objective. RQ1.a can provide the fundamental background of
research topic and collect the CSFs in literature. The rest questions can collect the
CSFs, challenges, solutions and consequence in practice. The research questions are
defined as below:
RQ1: What are critical success factors for the success at Agile planning phase?
RQ 1.a. What state of the art on the CSFs in the existing literature?
RQ 1.b. What state of the art on the CSFs at Agile planning phase in practice?
RQ2: What are the challenges associated with each factor?
RQ3: How do practitioners address these challenges?
RQ4: What are the consequences of not ensuring the factors?

1.3 EXPECTED OUTCOMES

According to the aim and research questions, the expected outcomes are
presented as follows:
1. A list of CSFs of Agile planning.
2. A list of challenges associated with each factor.
3. A list of the solutions to address each challenge for factors.
4. A list of the consequences of not ensuring these factors.

1.4 THESIS STRUCTURE

This thesis is decomposed into chapters: Introduction, Related work,
Methodology, Results, Analysis and Discussion, Conclusion and Future work. The
figure 1.1 can make the structure more visible.

Figure 1.1 The structure of this thesis

8
 The remainder of this thesis is organized as follows: Chapter two is used to
present the related work of this research. It provides the introduction about five
aspects: ASD, Agile methodology, ASD lifecycle, critical success factors, and the
gaps of existing studies. Chapter three presents the methodologies used in this paper.
The two methods are separately introduced in two subsections to collect necessary
data. Chapter four shows the results obtained from the data collected through
systematic mapping and survey. Chapter five presents what are found based on the
results shown in chapter four. Chapter six draws a conclusion according to the
analysis of the collected results. Threats validity and a description of the future work
are also presented in this chapter.

9
2 RELATED WORK
Nowadays, many organizations pay more attention to software development.
However, numerous software projects are failed in many situations, such as delivery
on time, over budget and low customer satisfaction. A major cause of this
phenomenon is that the software development approach is not suitable for the pro ject.
The testimonial data indicate that no more than twenty percent of projects meet the
characteristics of traditional methods, but the project manager still uses this
traditional method to develop the software projects [5].
The traditional methods rigidly follow a pre-established plan so that it is
difficult to adapt to the dynamic environment, while the Agile methods increase
customer satisfaction and require projects quickly to adapt the changes. The
traditional methods use the linear development lifec ycle. The stages of this lifecycle
are executed only one time. The projects can move to next stage until the predefined
milestones or objectives of the current stage are achieved [5]. This indicates the
project can be delivered to the customer until the end of the development lifecycle. In
contrast, the Agile methods are based on the iterative lifecycle to accept and embrace
change [5]. In each iteration phase, the project performs almost every phase of the
development cycle. The future features should not or need not be prepared in the
current iteration due to the feedback of current iteration may influence the next
iteration [5]. At the end of each iterative lifecycle, the project will be delivered to the
customer to get feedback to increase the customer satisfaction, to improve the project
quality and to make some adjustments for next iteration. This shows that the Agile
development methods are more suitable for the current dynamic environment
compared with traditional development methods.
Agile software development becomes more and more popular due to its
characteristics. Thus, we prefer to study the CSFs of Agile development rather than
traditional development. The following subsections provide a comprehensive
introduction for Agile software development.

2.1 Agile Software Development (ASD)

Agile Software Development (ASD) is a currently emerging software
engineering approach for people to develop software quickly by dividing the whole
development into a series of iterations [6]. In the environment of rapidly changing
requirements, these small and typically iterations show their obvious advantages.
Additional, “Agile” well indicates the ability to create and respond to this
environment. With Agile methods, ASD aim at making the development more agile
to gain more profit and productivity. It makes Agile development gain huge interest
from organizations all over the world.
Since the “Agile manifesto” was proposed by a group of software practitioners
in 2001, it has been well known and widely used by more and more people. The
“Agile method” is people-based rather than plan-based, so ASD focus on customer
satisfaction [7]. The four values were put forward in manifesto to underlie the
philosophy of ASD [8]. The details are as follows:
● Individuals and interaction over process and tools
● Working software over comprehensive documentation
● Customer collaboration over contract negotiation
● Responding to change over following a plan

10
Besides, there are 12 principles mentioned in the Agile manifesto. Even though the
practitioners have their own philosophies and experience on developing software,
they have common degree on these principles [2]: using face-to-face conversation to
communicate; close cooperation between teams of business and development;
frequent delivery of working software; meeting customer satisfaction earlier;
adaptive team capability; accepting changing requirements by customers. It indicates
that ASD has been widely acknowledged and utilized.
2.2 Agile methodology
ASD methodologies are accepted and widely used in organizations because of
their benefits and characteristics. There are no less than 10 ASD methods known to
us. The widely used methods are Extreme Programming (XP), Scrum, and Dynamic
Systems Development Method (DSDM). Due to the agility of these methods, ASD
can be widely used in both traditional environment and Agile environment [6]. On
one hand, these ASD methods have common features, such as iterative development,
interaction, communication, and so on [9]. On the other hand, they also highlight
their own characteristics. In the following, the details of three frequently used
methods are introduced.
2.2.1 DSDM
The Dynamic Systems Development Method (DSDM) provides a framework
and best practices to control software development projects with short timeline s [10].
This method can truly understand the needs of a business, and deliver software
solutions as quickly as possible [11]. It also provides documentation for better
understanding and maintenance of software. It works well to address the common
failures of projects, such as over budget, over time and so on. The DSDM lifecycle
include six stages: Pre-project, Feasibility Study, Business Study, Functional Model
Iteration, Design and Build Iteration, Implementation, and Post-project [9]. The
details are presented as fig 2.1:

Figure 2.1 DSDM lifecycle

11
2.2.2 Extreme programming (XP)
As Cohen said, XP is a very popular Agile method in Software Engineering area
[9]. It is used as a software-development discipline that organizes people to focus on
producing higher quality software [12]. Following this purpose, the characteristic of
multiple short development cycles of XP is helpful to reduce the cost of vague and
incessantly changing requirements. Aiming at requirements is also one reason why is
XP so successful on customer satisfaction. Although there are 12 rules introduced by
Beck and Jeffries in XP, it is easy for people to implement XP by following five key
principles: communication, simplicity, feedback, courage, a nd quality work [11].The
development lifecycle of XP is presented as fig 2.2:

Figure 2.2 XP lifecycle

2.2.3 Scrum
Scrum is one of the popular methods in Agile software development [9]. It
provides the iterative and incremental development framework to manage product
development. Through the framework and a series of practice, everything becomes
visible, which allows practitioners to exactly know what’s going on, to easily deal
with unpredictable challenges and to keep the project moving toward the desired goal
[13]. This method works well for the projects where the requirements are not clear in
the early stages of the project. In scrum, the each iteration called sprint which
includes the sprint planning meeting, the daily scrum, the sprint review and the sprint
retrospective [13]. The development lifecycle of scrum is presented as fig 2.3:

Figure 2.3 Scrum lifecycle

12
2.2.4 Comparison
We perform a comparison for these three methods from four aspects: process,
roles, practice and scope, the detail as figure 2.4:

Figure 2.4 The comparison for three methods

2.3 ASD lifecycle

2.3.1 Overview of ASD lifecycle
Considering that there are different ASD methods, the development processes
are more agility and various from methods and projects. As Waters said that there are
no obviously distinct stages during the development [3]. According to the key
activities of development, the ASD lifecycle can be divided into 4 phases: Planning,
Iteration, Release, and Retirement [3]. In order to give a visualized introduction, we
drew a flowchart. The working process of the whole ASD lifecycle and the details of
each phase are narrated as fig 2.5.

13
 Figure 2.5 Agile software lifecycle
1. Planning - This phase contains two different level of
planning: release planning and iteration planning. Release planning is a high-
level planning which outlines the scope of the deve lopment project. Iteration
planning is a low-level planning which provides the detail on the build of the
product use stories at each iteration.
2. Iteration - Iteration is a fixed, short period of time that the team chooses to
work within [3]. In this phase, the product is divided into features to complete.
This period is varying from the Agile development method.
3. Release - This phase may contain several releases for complex systems. The
important aspects of releasing include testing, finalization of any system and
user documentation, training, and final release of the iteration into production.
4. Retirement - Retirement is the last phase of ASD lifecycle, which includes
system decommissioning or system unsetting and customer notification and
migration.
2.3.2 Planning in ASD
Planning is essential and critical to the success of ASD. The plan can be
reflected in the value of guiding our investment decisions. Besides, it also can help
us know how to allocate people to the project. Because of the plan, we can continue
projects avoiding a number of problems and barriers. The purpose of planning is
providing an overall view of the scope of ASD products to find an optimal answer of
what to build [4]. In addition, a good planning process can support organization to
reduce risk and uncertainty, gain better decision making, establish trust, and convey
information [4]. The Agile planning phase can be roughly divided into two levels.
One is release-level planning, the other is iteration- level planning.
A. Release plan
Release planning is a high- level planning which occurs at the start of a project.
It provides an overview framework to outline the scope, schedule, and resources for a
project. Generally, it lasts three to six months and includes three to twelve or more
iteration, depending on the duration of iteration [4].
A good release plan provides a view of “whole” for customer and team rather
than the details of iteration. In release plan, the high- level requirements were
decomposed into a user story. And the plan presents the time frame of a user story
and the expectations of each iteration at a cursory level. Unlike the iteration planning,
it needn’t provide that which developers will work on which user stories or tasks, or
the sequence in which work will be performed within iteration [4]. The release
planning as a high- level planning just outlines the whole overview for a development
project.

14
 B. Iteration plan
Different from release planning, iteration planning is conducted at the start of
each iteration which provides the detail on the build of the product user stories.
During the iteration planning meeting, the participants who involved in taking a raw
idea and turning it into a functioning product use spreadsheet or note cards to create
the iteration plan [4]. At this meeting, they only focus on the tasks of one iteration.
Generally, one iteration plan will last two to four weeks. The components of iteration
plan are smaller and more detail than release plan. This change increases the
accuracy of the estimates [14]. Thus, one of the purposes of iteration plan is that
decomposing the user stories into a set of small tasks.
There are two kinds of approaches can be broadly and successfully used by
teams to plan iteration: velocity-driven and commitment-driven. Although they are
different in some details, the general steps are similar. Through adjusting priorities of
user stories, teams select and identify the high-priority ones to put them into a new
iteration so that they can be transformed into functioning product.
2.4 Critical Success Factors
Most of the primary studies are from the manager’s perspective to evaluate the
success of projects. They indicated that there are three key components of any
project’s success: time, cost, and quality [15]. Cohn and Lindvall et al. [16, 17]
added “Scope” as another key success attributes. No matter how many key attributes
of success, the goals that developing high-quality software with less time and the
cost will never change [2]. Aiming at that goal, people have collected many factors
through analyzing the case studies and research theories in Agile implementation and
Agile software development project [1]. Because of the failures and successes of
cases, these factors were divided into two aspects: failure factor and success factor.
Even though the failure factors also can help people to increase the chance of success
by avoiding certain serious pitfalls [1], people prefer to focus on the success factors
which can directly notice them measure or evaluate.
A. Definition
The critical success factors were first introduced by Rockhart [18] to identify
and measure the performance of organizations. At 1981, Bullen and Rockhart
defined the CSFs as “the limited number of areas in which satisfactory results will
ensure successful competitive performance for the individual, department, or
organization. CSF’s are the few key areas where ‘things must go right’ for the
business to flourish and for the manager's goal to be attained” [19]. As Pinto and
Rouhiainen said the CSFs are factors that will significantly improve the chances of
project success if they are addressed appropriately [20].
As the rise of software development, more and more studies apply the CSFs on
the software development. Because of the different research fields, each article has
precise definition for their CSFs. In our research, we defined the CSFs as factors
which influence the success of ASD at planning phase.
B. Current research status
Agile as a new development approach is becoming more and more popular. The
researchers pay more attention to the successful development of software with Agile.
Thus some studies focus on the CSFs of Agile software development.
The paper [1], [2], [21] and [22] conducted a survey to investigate which factors
have the association with successful development in Agile. The paper [1] and [21]
used the same possible factors to perform a survey. But the most respondents of
paper [21] worked in distributed environment of ‘mixed’ companies. Due to the

15
respondents in the different project environment, the results of the surve y are
different. The paper [5], [23], [24], and [25] collected the CSFs using literature
review and perform the frequency analysis to priority the collected factors. In paper
[24], the author presented a systematic literature review on challenges and success
factors for large-scale Agile transformations. The paper [26] focused on the CSFs of
outsourcing projects which developed by Agile. The paper [27] used literature
review to highlight critical success factors and barriers during software process
improvement. The paper [28] was based on authors’ experience to explore the CSFs
while implementing Agile Software Development and Testing across distributed
teams. The paper [25] used the content analysis method to analyze and produce a list
of CSFs from the extensive literature. The author of paper [29] used Evidence-Based
Software Engineering (EBSE) and Systematic Literature Review (SLR) to propose a
set of success and failure factors. The paper [41] used the qualitative interview to
validate the conceptual model through examining the relationship between the
candidate success factor and Agile project success. Paper [42] as an empirical study
conducted local interview to collect what factors in the successful projects are
indicative of software development agility. These studies well support our research
to perform in-depth analysis.
In some of these studies, researchers used various symbols to categorize these
failure factors and success factors separately, such as organizational factor, process
factor, project factor, people factor, team factor, technical factor, non-technical factor
etc. The standards of categories vary from the context of project, and most of these
studies didn't illustrate how they categorize these factors. In terms of categorization,
researchers and practitioners have no broad consensus [5].

2.5 Gaps of current literature

As mentioned above, these studies provide some effective data for our research.
However, we still find some gaps in the current literature, the details as follows:
1. All these papers investigate the CSFs that cover the entire lifecycle of project
rather than focus on the single stage. Thus, these CSFs are too vague and
broad, and it is difficult for people to apply them in development practices.
The planning phase, as an early stage of project, has a significant impact on
the success of the project. Thus we focus on the factor at Agile planning
phase rather than entire lifecycle. To address this gap, we proposed the RQ1
to identify the CSFs at agile planning phase.
2. Some papers just conduct the survey to judge whether these given factors are
related to the successful development of software projects with Agile, such as
paper [1], [2], [21] and [22]. Some papers use the literature review to collect
the factors, such as paper [5], [23], [24] and [25]. All these papers don’t make
an in-depth study on when these factors impact success and how to manage
them to improve the chances of project success. To address this gap, we
proposed the RQ2, RQ3 and RQ4 to make an in-depth analysis for these
factors.

16
3 METHODOLOGY
To answer the RQs, we use two kinds of qualitative research method: systematic
mapping and survey based on a questionnaire. The systematic mapping is used to
present the existing literature related to CSFs of ASD. As mentioned in the section of
related work, there is no paper just focusing on the factors of the Agile planning
phase. Thus we collect the factor associated with whole development lifecycle of
Agile. In order to in-depth study, we conduct an online questionnaire to investigate
factors and challenges associated with each factor at the planning phase. The
respondents who have experience on Agile development give their opinions with
freedom through the open-ended questions. These two research methods can well
address our research questions. The details are shown in Fig 3.1:

Figure 3.1 Correspondence between research methods and research questions

3.1 Systematic Mapping Design
To answer RQ1.a, we applied systematic mapping method (SMM) to identify
the CSFs in existing literature. This method is divided into the following sections to
introduce in details.
3.1.1 Motivation of selecting SMM
The SMM is chosen to identify the critical success factor of Agile software
development in existing literature. SMM works well for broad and weakly defined
research areas [30]. Conducting the SMM can return a large number of studies. It is
this advantage that allows researchers to link the evidence with their study at a high
level of granularity. Comparing with mapping, although Literature Review (LR) as a
qualitative method covers a wide range of subjects, it lacks an explicit intent to
maximize scope or analyze data collected [31]. Thus, it is usually used as
fundamental to get the status and effort being done in primary studies. In our case, it
cannot provide enough evidence to support our research questions.
Systematic Literature Review (SLR) is used to systematically search for,
appraise and synthesize research evidence. To conduct SLR, researchers must follow
strict and systematic pre-defined process to identify and report whether the research
supports their research questions or hypothesis [32]. This indicates that this method

17
r
equi
rescons
ide
rabl
ym or
ee ffo
rtthantr
adi
ti
onall
it
eratur
erevi
ews[32]
. Duetothe
l
imi
ta
tionoftim
ea nde
ff
ort
,t hemappin
gism ores
ui
tableforou
rst
udy.
Taki
ngtheaboven
arr
ationint
oa cc
ount
,w epe
rformm appi
ng m
ethodt
oc o
llec
t
t
heinf
ormat
ionfrompr
imarystudi
estoa n
swerRQ1 .
a.
3
.1.2 S
ear
chp
roc
ess
Th
ea imofthisp
arti
stofindou
texis
tings
tud
ies whi
chhavest
rongre
lat
ion
wi
thourstudyt
opic
.W edes
ignedfou
rste
pst oa
chi
evethi
spurpo
se,andth
ed e
tai
ls
ofth
esearchproc
essa
redi
splay
edasFig3
.2.Fol
lowingth
ispr
oce
ss,wefin
al
lyg ot
1231paper
s.

F
igu
re3.2S e
arch
ingp ro
cess
I
nfirs
tste
p,wed ef
inedthek eywords andsynonymt
ohe
lpu
sfu
rth
er
u
nde
rs
tan
dt h
est
udyt
opi
c,th
ed e
ta
ilastab
le3.1:
Tabl
e3.1 Keywordsa n
ds y
nonym
s
K
eyw
ord
s Sy
non
ym

c
ri
ti
cals
ucc
essf
act
or s
ucc
essf
act
or,f
act
or

A
gi
lep
lan
ningph
ase A
gi
les
oftw
ared
eve
lopm
ent

I
nthes econdstep
,w e def
inedthes e
archst
ringtop rope
rlyconne
ctthe
k
eywo
rdsw i
ththei
rsynonymstoge
ther
.Theop er
ato
ro fkeyword
sis“AND ”,and
t
heope
rat
orofthesynonymis“OR ”
.T ohigh
ligh
ttheconne
cti
onam ongkeywor
ds
a
ndsyn
onyms
,t hesea
rchs
tr
ingw a
slis
tedasbelow
.

（“ c
ri
ti
calsuc
ces
sfac
tor
”OR“suc
ces
sfact
or”OR “
fac
tor
”） AND (
“Ag
i
le
s
oftwar
ed e
vel
opmen
t”OR “A
gi
lepla
nni
ngphas
e”）

Inthethi
rdstep
,w es
elect
e dappropr
iatedata
b a
s e
stosearchpape
rs.Ch oosinga
g
oodd ataba
sec an wel
lsupportustoc ol
lectm o
rec om p
rehens
iverese
archess othat
w
ec and eeplykn owth eprimarystudya boutourt opi
c. Com p
endexan dIn spec,
S
copus,an dW ebo fS c
ienc
ea rethew ide
lyu seddatabase
sn owadays
,du etoth em
a
ggregatem os
tr e
levantda
tabases[30]
.T able3.2des c
rib
esthenum be
ro fresult
sf o
r
e
achd at
abase
.
Inthelaststep
,w ei
denti
fythes ea
rchf i
elds
.Fo rCom p
endexa n
dIn spec,the
s
ear
chf ield
sw eappli
e done achterma resubjec
t,tit
lean dabs
trac
t. Ford a
tabase
W
ebo fS c
ienc
e,these a
rchf
ieldistopi
c.

1
8
 Table 3.2 Searching results of each database
Database Search results

Compendex and Inspec Scopus Web of Science Total

229 162 840 1231

3.1.3 Study selection process

The aim of this part is to identify the studies which can support or address our
research questions. Through the selection process which is displayed in Fig 3.3, we
finally identified the 13 studies, the detail as follows:

Figure 3.3 Searching processes and results

A. Selection criteria
We firstly defined the include criteria and exclusion criteria to select the s tudies.
The details as follows:
● Inclusion criteria
1. The studies propose or extend critical success factor.
2. The studies are within Agile software development.
● Exclusion criteria
1. Published before 2001 when the Agile manifesto was issued.
2. Language is not English.
3. The full text cannot be read for free.
4. The papers are not reported in a peer-reviewed workshop, conference, or
journal.
5. The paper that are duplicates
B. Scan strategy
According to the inclusion and exclusion criteria, we performed a two-stage
scan to identify the final studies which can support or address our research questions.
In the first stage, we just considered the abstracts and title of studies based on the
inclusion/exclusion criteria. For more accurate results, we retained ambiguity studies

19
which are difficult to judge whether related to our research questions based on the
abstracts and title. Through the first scan (rough scan), the 235 studies were left. In
the second stage, we read the full text to refine our search results. Through the
second scan (elaborate scan), just 13 studies were left. These studies can well support
our research.

3.2 Survey Design

To answer the RQ1.b, RQ 2, 3, 4, we use survey which is a retrospective form
of investigation to gather data from wider population [30]. To well implement the
survey, we clarify the mandatory parts: the motivation, sampling method, target
population, data collection and analysis method. The details of each part are
presented as follows.
3.2.1 Motivation of selecting survey
The survey provides a way to measure a population’s characteristics, self-
reported and observed behavior, awareness of programs, attitudes or opinions, and
needs [33]. It also allows us to collect data from a wider population. We want to
collect the opinions from experienced people of Agile development to identify the
factors and challenges at the planning phase. Obviously, survey is more suitable for
us to conduct the research rather than case study and experiment.
Case study is an in-depth study of a particular case compare with the sweeping
statistical survey [34]. It is generally used to validate a model or a framework in a
particular case which can be a small team or a specific industry. In our research, the
case study would be ok as well. However, it is harder to identify companies willing
to run case studies. On the other hand, it is easier to talk to people from companies
and ask them to fill up a questionnaire.
The experiment is a systematic and scientific approach to study the relationship
between the variables. The researcher can manipulate one or more variables, control
and measure any change in other variables [34]. It is also a suitable way to evaluate
which approach is more effective. So the experiment is also not suitable for us to
conduct our research.
3.2.2 Target population
People who have experience on Agile software development is our target
population. According to the aim and objectives of this paper, people who have more
experience on Agile development can provide more helpful and effective answers.
3.2.3 Sampling method
There are two categories of sampling methods: probability and nonprobability.
Since the target population is the individuals who have experience in Agile
development, probability sampling method is not a good choice for our paper. As it
is known to all, non-probability sampling method contains four types: quota
sampling, convenience sampling, judgment sampling, and snowball sampling.
Quota sampling firstly segments population into some sub-groups, then use
convenience or judgment sampling to select the sample size of each subset [35].
Since we don’t have a majority of the sample population, this sampling method was
excluded.
Convenience sampling selects the sample based on the only standard:
convenience [35]. It has many advantages, such as easily operate, wide application,
and so on. Using this sampling method, each individual of the overall is regarded as
people who have the same essence. However, in our thesis, experienced people and

20
inexperienced people cannot be treated in a unified way. Thus, convenience sampling
method was excluded.
Judgment sampling selects sample based on the knowledge and judgment of
authorities or the researchers. This sampling method is known as purposive sampling
and authoritative sampling. We want to find the people who have experience on ASD
to participate our survey, this indicates our sampling have strong purpose. Thus,
judgment sampling was included.
Snowball sampling contains two steps. The first step uses other sampling
methods to select target population. The second step is let the selected people of the
first step explore more sample. It can help people extend the samples. Thus, snowball
sampling was also included.
According to the above analysis, we conducted the judgment sampling and
snowball sampling to select the sample in our thesis. Judgment sampling as the first
step contains two criteria: we used working experience to judge whether the
respondents are experienced, and used domain of the company to judge whether they
have experience on ASD. At the second step, according to the two criteria of
judgment sampling, they asked their other friends to conduct this survey.
Through the sampling method, we got 9 experienced respondents who are
working for different companies. The companies are ESIG, Ericsson, Qvantel,
Federal University of Piauí, ICBC software development center, Infoway, IFMA.
The working domains of these respondents belong to ASD.
3.2.4 Data collection method
A. Motivation
In this thesis, we use a questionnaire to collect necessary data and information.
There are two kinds of methods can be utilized in the survey: questionnaire and
interview.
The interview would be ok for our research. It seems simple and self-evident,
and the interviewer coordinates a conversation aimed at obtaining desired
information [36]. However, we didn’t choose this method, because when we contact
the people to conduct the interview, some of them don’t like face to face
communication, and some respondents have no appropriate time for an interview.
Take into account these cases, the respondents suggest us to send a questionnaire to
them. Thus, we choose the online questionnaire as our data collection method which
is close to interview.
Compared to interview, the respondents can fill it out at their own convenience.
Beside of this, due to our English is not very good, the results of the questionnaire
we got can be more effective and accurate compared with the interview. To increase
the response of the questionnaire, we select online-questionnaire instead of a mail
survey. In addition, most of the respondents are familiar with our supervisor or us,
so they can provide higher attention on it. All respondents have rich experience on
ASD, so they can give more accurate and effective answers.
B. Questionnaire design
This study employed the online questionnaire to gather data from experienced
people of agile development. The questionnaire was divided into three parts. The
first part was an introduction about the simple description of research, aim of survey
and authors’ information.
The second part was designed to gather demographic data, which included both
the respondent’s demographic information as well as the company information, such
as domain of the company, company size and so on.

21
 The third part focused on the critical success factors and the challenges in terms
of managing factors. The respondents can describe their opinions through the open-
ended questions.
For the third part, we designed questionnaire around four open-ended questions.
The open-ended questions provided the freestyle for respondent to describe their
lived experience. If we provide multiple choices to them, it will limit their thoughts
and influence their opinions. In addition, through their description we can collect
more realistic data rather than the settled data. Thus we can identify some factors
which cannot find in the book. The following table shows the questions and
motivations in details.
We also designed a document questionnaire for Chinese so that we can send it
to our friends who have experience on agile development.
Table 3.3 Survey questions
Question Motivation

What factors are mandatory to This question is used to collect the opinions of
the success at Agile planning? experienced people towards factors at Agile
planning phase.

What are the challenges This question is used to collect the opinions of
associated with each factor? experienced people about the challenges
associated with each factor.

How do you address these This question is used to collect the opinions of
challenges? experienced people for addressing the
proposed challenges.

What are the consequences of not This question is used to collect the opinions of
ensuring this factor? experienced people about the consequences of
not ensuring this factor.
C. Pilot test
In order to test content validity and readability of questionnaire, we conducted a
pilot test before the survey invitation. We send our questionnaire to two PhD
students who are from BTH (Blekinge Institute of Technology). They have extensive
experience on Agile development. According to their feedback and suggestions, we
improved our questionnaire so that the other respondents can easily understand our
survey.
D. Implementation Plan
The questionnaire was designed at December 2016 and was made available to
target population from December 15th to January 30th. Due to the limitation of the
resource of the target population, the selection of respondents will take our more
time. So we considered six weeks is the appropriate time to conduct a questionnaire.
Considering Google is not allowed in China, we use Google form and Microsoft
Word to design two kinds of the questionnaire. The questionnaire designed by
Google form can be sent to respondents through a survey link. The questionnaire can
be accessed: https://docs.google.com/forms/d/e/1FAIpQLSdYhyKEUb2hZAEUvQ-
wBUwureAxC3CyaLjjO1AkQ6B5lezy2Q/viewform?usp=sf_link.
The document questionnaire designed by Microsoft Word can be downloaded and
sent to get feedbacks in China. The structure of questionnaire has been introduced in
the previous part, and the details of these are presented in Appendix A and B.

22
3.2.5 Data analysis method
The results of mapping and demographic data of respondents were analyzed
using descriptive statistics. The reason why descriptive statistics is used to describe
the basic features of the data in this study is that it helps us to simplify large amounts
of data in a sensible way. This method provides simple summaries about the sample
and the measures. Some measures that are commonly used to describe a data set are
measures of central tendency which includes the mean, median and mode [37]. In
this case, the mode was used to describe the central tendency of collected data, such
as factors by mapping, demographic data of respondents and so on.
Due to our survey questions are open-ended and subjective, the answers of
respondents were analyzed using qualitative content analysis. Qualitative content
analysis has come into widely use in analyzing text data [38]. The text data is
various. It might be in verbal, print, or electronic form and might have been obtained
from narrative responses, open-ended survey questions, interviews, focus groups,
observations, or print media such as articles, books, or manuals [39]. In our case, the
text data is obtained from open-ended survey questions. Taking into account the
above interpretation, we choose the qualitative content analysis as data analysis
method.
Qualitative content analysis is not only merely counting words to examining
language, but also classifying large amounts of text into an efficient number of
categories that represent similar meanings [39]. Besides, the categories relate to the
overall research questions [40]. In this research, we classify the answers of the
survey into four categories: description of factor (at Agile planning phase), challenge
(associated with each factor), solution (in terms of each challenge), and consequence
(not ensuring the factors). We can follow these four categories to analyze the
opinions of experienced people to get the effective results for our paper.
Base on the process of qualitative content analysis written in paper [43], we
analyze the online questionnaire results. These analyses are performed to help us
understand what the mandatory CSFs in practices are. Figure 3.4 indicates the
general steps of content analysis, and the details of each step are described at below.

Figure 3.4 Analysis processes

The process of qualitative content analysis can be divided into three parts:
preparation, organizing, and reporting [43]. The first step in the picture is that read

23
the responses carefully belonging to preparation. We read through each response
carefully at least triple to make full sense of their opinions. If there is confusion
about the understanding of the responses, we send an email to the respondent to
make the verification.
At the process of organizing, we decomposed it into three steps. The first step is
to identify coding categories. According to the four open-ended questions in survey,
we identified four coding categories to sort the descriptive data we have collected:
description of factor, challenge, solution, and consequence.
The second step is thinking about the meaning of the response. If we have the
conflict opinions on same response, we send the email to respondent until this
conflict addressed.
The third step is to label each response with four coding categories. Based on
the understanding for these responses, we divided these responses into suitable
categories. We used one column to present the coding categories and responses in the
next column, the detail as shown in pictures of chapter 5.
At the process of reporting, we wrote up the analysis processes and results.

24
4 RESULTS
Agile as a new development approach can better adapt to the dynamic
environment comparing with the traditional development. However, there are still
some failures in Agile software development. To improve the successful chance of
project, we focus on the factors that have significant impact on the success of ASD at
planning phase. Firstly, we performed a systematic mapping to collect the CSFs of
ASD in current studies. We used the keywords and search strategy described in
chapter 3.1 to search the papers in the databases, and then we used the
inclusion/exclude citeria to select papers. We finally identified the 13 papers that can
be used to provide the effective factors for us. Secondly, for in-depth analysis of
CSFs, we conducted a survey using online questionnaire to collect the open opinions
form experienced people. As the details of the questionnaire introduced in chapter
3.2, we collected six aspects of the CSFs: the name of factors, the description of each
factor, the type of factor, the challenges associated with each factor, methods or ways
to address each challenge, and the consequences of not ensuring these factors. The
detail data of each aspect will provided at following section.
As we described in chapter 3, we used two methods to collect necessary data.
These data will be discussed at following two subsections. The structures of
subsections are arranged as follow: section 4.1 presents the results collected from
systematic mapping, section 4.2 presents the results extracted from online
questionnaire.

4.1 Result of systematic mapping

As described in section 3.3, we finally identified the 13 papers to support our
research. In this thesis, we just focused on CSFs for Agile development. Some papers
mentioned CSFs are not included, due to they are not targeted for Agile development,
and some papers proposed factors which are limited in a specific project. Therefore,
we just got 13 papers that are close to our research. The details of each paper are
presented in follow sections. The publication year and venue of paper are counted in
4.1.1. The section 4.1.2 provides an overview of these papers. The summary of
factors mentioned in current papers is presented in 4.1.3.
4.1.1 Classification of papers
This part presents the category results of the 13 papers extracted using publish
year and publication venue as standard. Figure 4.1 obvious shows that the earliest
paper is published at 2008, and the papers published in 2015 are the central tendency
which has the highest frequency. It indicates that the CSFs gain more attention since
2008.

25
 Figure 4.1 Publication year of papers
As the exclusion criteria mentioned in 3.1.3, the paper referred by our thesis
must be reported in workshop, conference or journal. In table 4.1, RN means the
reference number and F represents the frequency that the total number shown in each
publication venue. It displays that 69.2% (9) of the studies were reported in journal,
and 23% (3) of this studies were published in conference. Only paper was reported in
Workshop.
Table 4.1 Publication venue
Category Publication venue RN F

Workshop IEEE/ACM 8th International Workshop on 41 1

Cooperative and Human Aspects of Software
Engineering

Conference IEEE, International Conference on Computing 23 3

for Sustainable Global Development
(INDIACom),

International Joint Conference on Software 27

Technologies (ICSOFT)

International Conference On Software 28

Engineering

Journal Journal of Systems and Software 1,2,21,24 9

Journal of Enterprise Information Management 5

Scientific research and essays 25

International Journal of Project Management 42

Journal of Theoretical & Applied Information 29

Technology

Journal of Software 22

26
4.1.2 General results
Through the mapping using systematic processes mentioned in chapter 3, we
found that no papers just study the CSFs at agile planning phase. Due to planning
phase is part of the whole lifecycle, we extend the searching scope to the whole
development lifecycle. Finally, 13 papers were chosen. All these selected papers
studied the factors in the entire lifecycle. This part provides an overview of these
selected papers. Table 4.2 describes the details of each paper. It includes reference
number, publication year, CSFs of each paper, and category.
From this table, we can clearly see the CSFs and categories of these factors in
each paper. Different papers have different standards to classify factors, but the
authors didn’t give the details of partition criterion. The factors generally can be
divided into four categories: organization factors, people factors, project factors,
process factors. Whereas the same category in different papers may present different
scope, such as “Top- level management support” in the paper [5] was divided into
organizational factors, but paper [25] divided it into people factors. Some paper
didn’t use the categories to classify the factors, such as paper [27], [28] and [29]. The
paper [5] decomposes people factors into customer factors and team factors. This
situation significantly indicates that the CSFs of primary studies don’t have a
uniform scope of the category.
Table 4.2 The CSFs extracted from each paper
RNo. Year Critical Success Factors (CSFs) Category
Organization
Team Environment
factors
1 2008 Team Capability, Customer Involvement People factors
Project Management Process Process factors
Agile Software Techniques, Delivery Strategy Technical factors
Customer satisfaction, Customer collaboration, Organizational
Customer commitment, Decision time, Corporate factors
2 2009 Culture, Control
Personal characteristics, Societal culture, Training People factors
and learning,
Top-level management support, Organizational Organizational
culture, Level of project planning, Leadership, factors
Monitoring and controlling, Change management
skills
Project team commitment, Internal project Team factors
communication, Project team empowerment,
Project team’s composition, Project team’s
expertise with the task, Project team’s general
5 2015 expertise, Lack of development team skill, Project
team’s experience with SDM
User participation, User support, Customer training Customer factors
and education, Customer (client) experience, Lack
of end user experience
Technological uncertainty, Development Project factors
methodologies, Project complexity, Urgency,
Relative project size, Specifications changes,
Project criticality
21 2013 Customer involvement, People factors

27
 Project Management Process, Process factors
Agile Software Techniques Technical factors
Project Nature Project factors
Training, Management involvement, Access to
22 2008 external resources, and
Corporation size
Organization
Culture, Management support
factors
Customer involvement, Customer collaboration,
23 2014 Experience, Team size, Access to external resource, People factors
Training
Duration, Size, Complexity, On-time delivery Project factors
Dynamism, Methodology, Process training Process factors
Management support, Commitment to change,
Leadership, Choosing and customizing the agile
approach, Piloting, Training and coaching,
24 2016
Engaging people, Communication and
transparency, Team autonomy, Requirements
management
Effective project management skills/ methodologies people-related
(project manager), Support from top management, factor
User/client involvement, Skilled and sufficient
staff, Good leadership, Committed and motivated
team,
Clear requirements and specifications, Clear process-related
objectives and goals, Realistic schedule, Effective factors
communication and feedback, Realistic budget,
Frozen requirement, Proper planning, Appropriate
25 2011 development processes/ methodologies (process),
Effective monitoring and control, Adequate
resources, Risk management, Effective change and
configuration management, Clear assignment of
roles and responsibilities, End-user training
provision
Familiarity with technology/ development Technical-related
methodology; Complexity, project size, duration, factors
number of organizations involved; Supporting tools
and good infrastructure;
Technological uncertainty, Development
methodologies, Project complexity, Urgency,
27 2015
Relative project size, Specifications changes,
Project criticality
Assumptions, Cut Communication Loops, Facilitate
28 2009 Tool Driven Query Resolution, Initiate Test Drives,
Assess internal quality, Manager effort variance
Project management methodology, Tools, Delivery
strategy, Management process, Organizational
29 2015
environment, Agile methods, Project teams, Skilled
clients/providers

28
 Tr
a n
sfo
rm ationall eadersh
ip,Comm uni
cations ty
le
,
41 2015
Valuec ongruence
,De greeofa g
il
i
ty,Pr o
jectsize
.
Agi
li
tys upportedbyt opm anagement( fromp l
a n
- P r
ojec
t
dr
ivent oA gil
e),Le velo fri
sk-taki
ngw i
ll
ingness
, e nv
ir
onm en
t
In
stab
il
ityo ftheorgan
i zat
ionale nv
ironm ent
,Low
powerdistance
Lo wproject c
rit
ica
li
ty, Ex pe
ri
e n
cel eve
lo fthe Pro
ject
42 2013
team,Te chnologi
calu ncer
tain
ty,Re qui
r ement
s
unce
rta
inty, Cl osecus
t om e
rc ol
labor
ation,Agil
ity
suppor
te dbyt h
ec us
tom er(fromp l
an-dr
ivent o
Ag
ile
), Lowp ro
jectsi
z e(m anhours
),Co locat
iono f
theprojectt e
am m emb ers
4
.1
.3 Summ
aryo
fCSF
s
Consideringth ec ategorie
sa red iff
erentfromth ep apers
,w edon ’
tf ocuson
thes
ec ategori
e s when m e
rgethem .Ina ddit
ion,thefactor
sind if
ferentp ap
ers which
havesam em eaninga red escr
ibedi nthed if
feren
tn am e
.Thu s,w em erg edtheCSF s
ofcurrentlite
r ature whichh assam eo rsim i
larm eaningusingth eun iformn ame.If
onefacto
rh asw iderm eaningthanth eoth e
ron e
,w eth i
nkth eydon ’th avesam eo r
simi
lar m eani ng. Fore ach CSF i den
tifie
d, w ec ounted th ef requ ency o fits
occu
rrenceth a
th asb eencitedinp reviousstudies
,an dthenw ep riori
tizedth eCSF s
indescendingo rdera ccordingtothef requency.W ealsolis
te dth ere f
er encenum ber
whi
ch m ent
i one dthec o
rrespondingf actors
. Thed etai
lsared isplaysi nfol
low ing
tab
les
.
Factor
sw iththefr equencyg rea
terth anthreeareshowni ntable4 .3.F romth i
s
tabl
e,w ec anc learl
ys eewh i
chf ac
tori sm en
tionedatm o
stbyp reviouss tudi
es.Th e
mor
ef requentlyc i
ted CSF sten dt ob ef ac
torsr e
latedto m anagem en tsu ppor
t,
org
anizat
ion alcu ltur
ea ndenv ironm ent
,t ra
ining,d eve
lopm entm eth odologya nd
techn
iques
.
T
a bl
e4 .3T hefrequencyo feachCSF(F≥4 )
ID F
act
orn
ame RN F
requ
enc
y

1 m
ana
gem
ents
uppo
rt(
comm
i
tmen
t) 5,23,24
,25
,42(1
,21
, 9
22,27)

2 o
rgan
iz
ati
onalc
ul
tu
rea
nd 1
,2,5
,21
,23,29
,42 7
e
nvi
ronment
3 d
eve
lopmen
tme
thodo
logya
nd 1
,5,21
,23,25
,27,2
9 7
t
ech
niqu
es
4 t
ra
in
ing 2
,5,22
,23
,24
,25
,27 7

5 c
ust
ome
rin
vol
vem
ent 1
,2,21
,23,2
5,42 6

6 c
omm
uni
cat
ion 5,24
,25
,27,2
8,41 6

7 p
roj
ects
iz
e 5
,23,2
5,41
,42 5

8 t
eame
xpe
ri
enc
e 5
,23,2
7,42 4

2
9
9 t
eame
xpe
rt
is
e 1
,5,21,2
5 4

10 r
esou
rce 22
,23,25
,27 4

11 D
el
iv
ery S
tra
teg
y 1
,21
,23,2
9 4

12 l
ead
ers
hip 5,24
,25,41 4

13 p
roj
ectp
lan
/sc
hedu
le 5,2
5/1
,21 4

14 M
oni
to
ringa
ndc
ont
rol
l
ing 2
,5,25
,27 4

15 r
equ
irem
ent
san
dsp
eci
f
ica
ti
on 24
,25
,27,42 4
u
nce
rt
ain
ty

16 p
roj
ectm
ana
gem
ent 1
,21,2
5,2
9 4

Asshownintab
le4.4
,t h
er emainde
r CSFse x
tra
cte
dfroms tu
die
sareal
so
p
rio
ri
ti
zedindes
cend
ingorde
r.Althoughthef
requencyofth
ese CSFsi
sle
ssthan
f
our
,itdoe
sn’
tmeanthe
ycanb ei
gnored
.
Tabl
e4.4Thefre
quencyofeachCSF(F≤3 )
ID F
act
or RN F ID F
act
or RN F
1 c
omp
lex
i
ty 1
,5,25 3 p
ers
ona
l
17 2 1
c
har
act
er
is
ti
cs
2 t
ool 25
,28
, 3
18 t
eamc
omm
i
tmen
t 5 1
29

3 m
ana
gem
ents
ki
ll 5,2
5 2 19 a
ssum
pt
ion
s 28 1

4 p
roj
ectd
ef
in
it
ion as
ses
sin
te
rna
l
1
,21 2 20 28 1
p
roce
ss qu
al
it
y

m
anag
eref
for
t
5 t
eame
nvi
ronm
ent1
,21 2 21 28 1
v
ar
ian
ce

c
lea
rob
jec
ti
vea
nd
6 p
roj
ectn
atu
re 1
,21 2 22 25 1
g
oal
s

7 p
roj
ectt
ype 1
,21 2 23 u
sers
uppo
rt 5 1

c
lea
rass
ignmento
f
P
roj
ect
8 5,42 2 24 r
ole
sand 25 1
c
ri
ti
cal
ity
/urg
ency
r
espon
sib
il
it
ies
9 du
ra
ti
on 23,2
5 2 25 v
alu
econ
gru
enc
e 41 1

c
ust
omer
/us
er/
cli
e
1
0 5
,29 2 26 d
egr
eeo
fag
i
li
ty 41 1
n
texpe
ri
enc
e

1
1 A
gi
lem
eth
od 24,2
9 2 27 u
serp
art
ic
ipa
ti
on 5 1

3
0
 commitment to
12 risk management 25, 42 2 28 24 1
change

13 team autonomy 24 1 29 team's composition 5 1

14 staff involvement 27 1 30 team empowerment 5 1

15 review-feedback 27 1 31 dynamism 23 1

16 decision time 2 1

4.2 Result of Survey

This subsection presents the result of survey based on an online questionnaire
regarding the CSFs at Agile planning phase. We divided this section into two parts.
The first part describes demographic data of respondents. The in-depth study results
of CSFs were presented in second part.
4.2.1 Demographic of respondents
This part provides the statistics of respondents by counting the frequency of
each category. The information of respondents investigated by the online
questionnaire contains the following items: respondents’ name, email, country,
company, the domain of the company, company size, role, and working experience.
We focus on some representative item to systematically analyze these respondents at
this part, and the details are as follows.
As shown in figure 4.2, the 9 respondents come from different countries. It
indicates that 44.4% (4) of the respondents come from Brazil, 22.2% (2) of the
respondents come from China, 11.1% (1) of the respondents come from Pakistan,
11.1% (1) of the respondents come from Portugal, and 11.1% (1) of the respondents
come from Sweden. This paper applies snowball sampling method which means our
supervisor sent the questionnaire to his contacts from Brazil. This is the reason why
Brazil has the highest frequency.

Figure 4.2 Country of respondents

31
 Figure 4.3 displays the domain of each company that the respondents are
working for. Although the domains of these companies are various, they all involve
in ASD. We can clearly see that ASD is popular and adopted by various aspects of
life.

Figure 4.3 Domain of the company

The years of experience working with ASD of respondents are shown in Figure
4.4. This picture clearly shows that 88.9% of the respondents have greater than 2
years’ experience on ASD. Only one respondent have two years’ experience. This
fully demonstrates that our respondents have a wealth of experience in Agile
development, which is consistent with our criteria for selecting respondents.

Figure 4.4 Years of experience working with agile software development

32
 Table 4.5 provides an overview of demographic data of respondents. They come
from different countries and work for different companies as various roles. This
indicates the results of the questionnaire are more comprehensive and representative.
Table 4.5 The overview of demographic data of respondents
Respondent Country Company name Company Role Experience
ID size

1 Brazil Infoway 60 Manager 10

2 Sweden Ericsson 100000 Manager 10

3 Brazil ESIG 120 Manager 10

4 Portugal Qvantel 500 Developer 12

5 Pakistan Qvantel 700 Developer 7

6 Brazil Federal 12 Developer 3

University of
Piauí

7 Brazil Federal Institute 40 Developer 2

of Education,
Science and
Technology of
Maranhão
(IFMA)

8 China ICBC software 2000 Developer 3

development
center

9 China ICBC software 2000 Developer 3

development
center

4.2.2 CSFs statistics

This part presents the results obtained from online questionnaire regarding an
in-depth study of CSFs. In table 4.6, we simply list the CSFs, their type, challenges
associated with each factor, and the consequences of not ensuring factors. These
items all associate with factor. Considering the solutions are provided for the
challenges, so we don’t list them in this table. The details answers of each respondent
will be presented and analyzed in next chapter.
Table 4.6 The summary of CSFs based on the answers of survey
ID Factor name Factor Challenge Consequence
Type

1 Expertise Both Select or train the Low quality and

team overtime

2 Joint Both Get all stakeholders Impossible to do Agile

33
 understanding involvement planning
of uncertainty

3 Productive team Both Hard have a good 1.Inaccurate plan

of good level level team 2.Overtime and even
failure.

4 Conciliate Both conciliate deadline Team demotivation

deadline between client
between client proposal and team
and team estimation

5 Top engineers Both Differences of 1.Fail to deliver top

people results
2.Lose team
3.Lose the ability to
adapt and innovate fact

6 Communication Both 1.Not effective Pain points not

communication addressed
2. Hierarchy and too
much management
3. Distributed teams

7 Priorities Both 1.Fast change Not mentioned

2.Depend on
business strategy
3.Defined ahead
4.Back fed by teams
velocity
5.Too detached from
reality

8 Coordination Iteration Human factors 1.Failure to deliver in

planning time
2.Demotivated team
members

9 Level of details Both Time and preventing Reduce accuracy of

unforeseen estimates

10 Team's maturity Both Constant change on Estimates based on

development team beliefs

11 Team Release Estimate accurately Low quality and late

Knowledge planning delivery

12 Team Both Other teams do not Project delay and quality

collaboration cooperate reduction

13 Requirement Both Reduce the risk of Increase cost and effort,

change change low quality, late deliver

34
5 ANALYSIS AND DISCUSSION
In this chapter, we make an in-depth analysis of the results of systematic
mapping and survey. They will be described separately in section 5.1 and 5.2. The
comparison between the results of two methods will be discussed in section 5.3
5.1 Analysis of data collected from systematic mapping
Through searching the databases, we finally identified the 13 papers that studied
CSFs for ASD projects in the last 16 years. We make a detailed summary of the
CSFs mentioned in these articles in section 4.1. For each CSF, we count the
frequency (number of literatures) of its occurrence and then express it using a
percentage of the total citation count (n=13). In this section, we just display the
analyzing of the representative factors whose frequency over than 3. The details are
shown in table 5.1.
The analysis of mapping results can help us understand the status of the art of
the success factors in the existing literature. As shown in table 5.1, the frequency and
percentage of each factor were counted to indicate the degree of attention. These
factors are cited much more frequently by existing literature. Obvious ly, there are
four factors that have more than 50 percent: management support (commitment),
organizational culture and environment, training, and development methodology and
techniques. Factor that has more percentage means it gains more attention. For
example, management support and commitment has the highest frequency (69.2%).
Therefore, from researcher's perspective, this factor was valued and can affect ASD
to a great extent.
Table 5.1 Frequency and percentage of each CSF
ID Factor name Frequency Percentage%

1 Management support (commitment) 9 69.2

2 Organizational culture and environment 7

53.8
3 Development methodology and techniques 7

4 Training 7

5 Customer involvement 6
46.2
6 Communication 6

7 Project size 5 38.5

8 Team experience 4

9 Team expertise 4 30.8

10 Resource 4

35
 11 Delivery Strategy 4

12 Leadership 4

13 Project plan/schedule 4

14 Monitoring and controlling 4

15 Requirements and specification uncertainty 4

16 Project management 4

5.2 Analysis of data collected from online questionnaire

This section presents a detailed description of the data analysis. Subsection
5.2.1 displays the detailed analysis of each factor. Subsection 5.2.2 shows the
summary of factors collected from the survey.
5.2.1 Analysis of each factor
We list the 13 critical factors identified by respondents in details. The answers
of respondents are presented in following figures. All these answers belong to the
respondents’ opinions. In order to make the original answers more readable and
understandable, we simply corrected the grammar and spelling mistakes. According
their answers, we provide the analysis of each answer below each picture.
A. Expertise

Figure 5.1 The answer of respondent about Expertise

The development team as a core component has a great influence on the success
of project. The respondent 1 (mentioned in Table 4.5) proposed that the expertise of
team as a CSF directly impacts the results of the project. There was a clear definition
for team expertise which includes five abilities: ability to work with uncertain

36
objectives, ability to work with top management, ability to work effectively as a team,
ability to understand human implications of a new system, and ability to carry out
tasks effectively [45]. The projects carried out by development team with expertise
have a high possibility to be success [5].
However, the selection of team members or training the team is a big challenge
for gathering a team with expertise. For example, paper [27] indicated that the high
training coasts is a barrier of team training for organizations. To address this
challenge, Respondent 1 simply proposed that the organization should try to match
the best team for the project. Due to he didn’t provide the specific strategies, we give
some supplements about this. For selecting the team members with expertise, the
organization can organize some tests to select members. For training the team, the
organization can use mentoring, and professionally guided discussions to transfer the
tacit knowledge between individuals [2].
The development team with rich expertise can successfully develop software
with the high quality, low cost at predetermined time. If this factor is not paid enough
attention, the project may have a low quality and cannot be completed on time and
budget.
B. Joint understanding of uncertainty

Figure 5.2 The answer of respondent about Joint understanding of uncertainty

The Agile methods provide an effective way to deal with uncertainty. As
respondent 2 said, Joint understanding of uncertainty is a CSF for Agile software at
planning phase. Having a better understanding of the uncertainty of Agile
development, we can better solve the problems caused by uncertainty.
However, not all stakeholders can understand the uncertainty well in practice,
which leads managers to give enforce long-term commitments on development team.
For example, the customers don’t understand the agile principle. In order to satisfy
the requirements of customers, the manager may forcibly make a release deadline for
the feature of product. This forcible long-term commitment violates the idea of agile
development [8]. Thus, getting all stakeholders to understand why it is not possible
to make exact estimates on time taken to develop a feature is a big challenge. To
address this challenge, the respondent 2 suggested that making close collaboration
between development team and business people to help all stakeholders have a
uniform understanding on uncertainty. In a small product the business owner can be

37
part of the team. There is no problem to make close collaboration for this situation.
But this is not possible in large-scale development with more than 100 teams. This
needs the business people to have regular meetings with development teams to better
understand uncertainty.
If all stakeholders understand the uncertainty well, the development team can
work in a pleasant environment so that the project can be de veloped in a high
velocity and quality. Some papers also demonstrated that uncertainty is negatively
associated with project success [5]. If this factor is not paid enough attention, the
project may not be completed as agile planning.
C. Productive team of good level

Figure 5.3 The answer of respondent about Productive team of good level
The level of productive team directly determines the outcome of the project. As
respondent 3 said, if the organization uses a limited team to develop software, this
may influence the accuracy of estimation and reliability of delivery date. The limited
team means that the level of productive team is low, due to the ability in some
aspects of the team is poor, such as expertise, experience, technology and so on. The
good level team illustrates that team members with high competence and expertise,
with great motivation, the team managers have light-touch or adaptive management
style and appropriate technical to train the team [1]. This factor is similar with the
expertise proposed by respondent 1, all of them are considered from the team aspect.
The respondent 1 just focus on the expertise of team and respondent 3 emphasizes
the whole level of team. Thus, team level has a wider scope including expertise,
technology and so on.
Although the productive team with good level can bring many benefits, it is
hard to gather a team like this. For addressing this challenge, the respondent 3
suggested that the development team take a presentatio n every fortnight to introduce
the good skills of each one and weak of the others. This provides a motivation for
people to constant learn to improve the level of the productive team.
The level of team determines the productivity of the team. The good level of
productive team dictates not only the pace of ASD, but also delivering the software
satisfying the requirements of the customers [2]. If this factor is not paid enough
attention, the plan may be not accurate, and the project cannot be delivered on time
and even lead to failure.

38
 D. Conciliate deadline between client and team

Figure 5.4 The answer of respondent about Conciliate deadline between client and
team
At Agile planning phase, whether it is the release planning or iteration planning,
the most important task is to make estimation on time. As respondent 3 said, the
deadline established in the contract may conflict with the team estimation. Like this,
the client should participate in the ASD and communicate with development team
about making deadline. However, it is still difficult to balance the deadline between
client and team some times. This is the reason why Conciliate deadline between
client and team as a CSF is mandatory to the success at Agile planning. In this case, a
software development manager needs to conciliate deadline between client proposal
and team estimation.
The respondent 3 thinks that the client deadline is more important than team
estimation. Thus it is hard for the development manager to explain for the team
members that the estimative deadline is less important. For addressing this challenge,
the respondent 3 suggested that manager should explain the consequences of
deadline extrapolation to team members, such as client complaints, investment
decrease and so on.
If this factor is not paid enough attention, the members of team are demotivated
due to they don't understand why client deadline is the most important. Team
members lose their enthusiasm for the work, resulting in products that fail to be
completed or delivered on time and quality.

39
 E. Top engineers

Figure 5.5 The answer of respondent about Top engineers

Engineers as executor of the project can directly affect the success of ASD so
that they should be selected and allocated appropriately. There are many aspects of
assessing whether he or she is a top engineer, such as their attitude to work and
difficulties, their experience, their knowledge, their background and so on. It greatly
increases the difficulty of choosing top engineers.
For addressing this challenge, respondent 4 as a software engineer proposed
some solutions. He listed some standards to hire people with the skills required for
Agile:
● Good communication
● Good skills
● A positive attitude instead of complaining no matter how hard it gets
● Willing to deliver important work fast
● Willing to work alone or in pair after the vision and daily goal is clear
It is obviously that not all engineers fit in the above standards. The respondent 4
provided corresponding solutions. If the engineers fit in the above standards, the
manager can empower them. If some engineers’ performance has been very poor, it’s
time to let them go. However, the employee turnover has a serious economic impact
for organization [46]. Considering the negative influence, if engineers are worth
trained, the manager can leave them and train them to avoid the economic impact.
Top level engineers deliver top level results. If organization cannot hire them, it
is not possible to expect to deliver top results. If organization manage to hire at least
some of them but do not empower them you will very likely lose them. The
organization might also create a culture around hierarchy and process (endless
managers and few top engineers) where every problem is hard to solve and every
change creates problems, so you lose the ability to adapt and innovate fast.

40
 F. Communication

Figure 5.6 The answer of respondent about Communication

Communication is one of the core activities to support the development going
smoothly. It was proved to be the critical success factor to the software project
success [5] [47]. Effective communication can increase the success chance of
development [41]. Close communication can bring many positive effects, such as
helping people find problems and solve them in time, helping people quickly
comprehend and respond to the risk, enabling the creation of shared values, and
encouraging team building [5] [47] [48]. Due to these advantages, it should be
managed at both release planning phase and iteration planning phase. The all agile
practices used in the projects had positive effects on the communication inside the
development teams, such as the Daily Scrum, Pair Programming, Iteration and
Release Planning meetings [44]. Among many methods of communication, face-to-
face conversation is highly effective.
As respondent 4 said, there are many challenges when managing this factor.
Firstly, people don’t know the effective ways to communicate. He proposed some
specific cases to interpret this, like don't know how to articulate themselves toward a
common goal, can't listen carefully before speaking, don't have proactive attitude and
so on. This challenge can be solved through Agile practices such as stand-ups and
retrospective. Stand- ups meeting can do a big service if developers are aware of what
they should communicate, such as process and blocker issues. Agile retrospectives
can help the teams to access the current situation, identify internal and external pain
points, and define actions to address them.
Secondly, hierarchy and too much management hurt communication. To address
this challenge, people at management level should empower their staff and give them
more freedom to communicate and to express their opinions.
Thirdly, communication becomes more difficult when geographically separated
teams need to work together. Respondent 4 proposes four solutions according to his
working experience:
● Letting teams deliver in different speeds.
● Giving freedom for anyone to add work on each other teams board, based on
the vision and goals (features that need to be delivered)
● Documenting decisions and making them available to all

41
 ● Communicating the vision and goals on every possible occasion
If the communication is limited, the agile projects will suffer and even may fail
[5]. For instance, pain points can be seen as problems which determine the success or
failure of software development project. If the pain points are found early and
addressed in time, it positively impact on the success of ASD. As the first solution
said, the pain points can be identified at Agile retrospectives. If there is no Agile
retrospective or limited communication on pain points, they may not be addressed
well, even the project may fail.
G. Priorities

Figure 5.7 The answer of respondent about Priorities

Priorities are another important task for both release planning and iteration
planning. Prioritization provides an effective way to deal with competing demands
for high customer expectations, short timelines, and limited resources [49] [50].
Establishing the relative priority base on the importance of product features can well
construct plan to provide the highest value at the lowest cost [50]. An optimal
priority can save time and cost, reduce redundancy, and improve productivity. It is
closely related to the success of the project. As respondent 4 said, priorities as a CSF
at Agile planning phase are essential for a number of reasons, such as keep people
motivated, give meaning to people work, measure progress and so on.
However, there are many challenges for people to manage this factor.
According to the experience of respondent 4, he proposed the following challenges:
● It can change very fast.
● It depends on business strategy.
● It usually defined way ahead of any delivery done.
● It can be made too detached from reality.
● It is back fed by teams’ velocity.
The respondent 4 said, there is no good way to solve all of the above challenges.
But he suggested that the development team basically need to know what the weekly
priorities are and make the cards to reflect that. The agile methods promote dynamic
prioritization to adapt dynamic environment. This makes agile method can better
deal with the above challenges comparing with traditional methods. Dynamic
prioritization refers that at the start of a new iteration, the customer can reprioritize
the features of this iteration, such as discarding originally planned feat ures and

42
adding new ones [51]. For example, the Scrum organize the daily meeting and spring
meeting, this can help people adjust the priority in time, according to the current
situation. And this also provides the effective ways to solve priority changing fast,
depending on business strategy, detaching from reality and back feeding by teams’
velocity. One principle of agile is to ask the customer to interact with the
development team daily [8], and this helps people to make the more reasonable
priority.
The respondent 4 did not proposed the specific consequences of not ensuring
this factor, due to there are too many negative results of not ensuring this factor, such
as low productivity, time-consuming, over budget and so on. People cannot predict
what will happen if this factor are not ensured well. For example, the poor priority
may lead to an unreasonable planning, and then have influence on the project in
terms of time, cost and so on.
H. Coordination

Figure 5.8 The answer of respondent about Coordination

At iteration planning phase, the most important thing is the coordination. It
refers to team-situated interactions in paper [52]. However, respondent 5 emphasize
the coordination as the interactions between teams and stakeholders. Every teams
and stakeholders should understand and practice the core concepts of Agile. A third
of Agile principles have relation with the interactions between teams and
stakeholders, such as sponsors, developers and users should maintain the constant
pace indefinitely, business people and developers need work together every day, and
so on [8].
No matter team or stakeholders, all of them are the key important components
during ASD. Thus, human factors become one of the challenges when managing
coordination. It includes many unpredictable aspects, such as the personal
characteristics, the individual’s turnover, and the absence of stakeholders [2]. All
these aspects have significant impact on the coordination between team and
stakeholders.
To mitigate the impact of human factor, respondent 5 advocated two solutions.
Firstly, for the teams, the managers should motivate them to provide good
performance and make close coordination among not only team members but also
teams. For stakeholders, the managers should motivate them to attend the core

43
practices and express their thoughts frequently. Secondly, the managers try to make
sure the involvement of teams and stakeholders in all core activities of ASD.
If we break the interactions between teams and stakeholders, the project may not
be satisfied by the customers and may even fail to deliver in time. Besides, the team
members are demotivated.
I. Level of details

Figure 5.9 The answer of respondent about Level of details

Estimation is an important and indispensable part of ASD. Although there are
many metrics, technologies, and theories supporting people to estimate, there are still
more spaces for people to improve so that they can make a more accurate estimate.
As respondent 6 said, the level of details is a CSF at Agile planning phase which can
impact estimation. The level of details about activities and tasks should be scoped
based on the type of plan. For example, the release plan just provides the overview
framework to outline the scope, schedule, and resources for the project [3]. So it only
needs a lower level of detail. But the iteration plan needs a higher level of details to
tell people the specific tasks. An appropriate detailing for different plan can gain
more accuracy estimates.
However, there are many challenges for people to manage this CSF, such as
time and preventing unforeseen. Time as one of the uncertainty can make a great
impact on the details of activities. This increases the difficulty of time estimation.
For this challenge, respondent 6 provided a solution that people should make
temporal delimitation instead of fixed plan so that can improve the agility of detail
plan and estimate. There are also many unforeseen things lead to that people cannot
provide enough details of activities. For this challenge, respondent 6 provided a
solution that giving a suitable level of detail rather than excessive detailing.
If the level of detail is not appropriate, it has significant impact on estimation.
For example, if the release plan has excessive detailing, it is hard to estimate and
may cost more effort and time. If the iteration plan is too general, it may reduce the
accuracy of the estimate.

44
 J. Team's maturity

Figure 5.10 The answer of respondent about Team’s maturity

Team maturity can be analyzed from three dimensions: team technical
competency, team motivation, and teamwork skills [53]. Respondent 6 as a
developer pays more attention to technical competency. Thus, he described team
maturity as team’s experience on project and domain. If a team has rich experience,
he believes the maturity of this team at a high level. In practice, some managers
prefer to select the development team with ability of self- management which may
have little experience on project. Whereas, some other managers may prefer to the
team who has rich experience. This will lead to the different maturity among teams.
The reason why team maturity should be noticed at both release planning and
iteration planning is that this CSF can directly impact on the quality and delivery of
the project.
One of the challenges people meet when managing this factor is the constant
changes within a development team. For instance, a developer leaves the job or
switch to another project. This staff turnover changes the maturity of the team and
may lead to the delay of delivery or other negative impacts, such as the experienced
people turnover or novice added. Respondent 6 provided a solution for this challenge.
Team members should keep on study to improve their skill and knowledge, and to
reduce the gap of experienced and novice members. What’s more, organizations
should provide a good working environment for their staff, and reduce unsatisfied
developers who have many chances to leave work.
It is very dangerous if ignoring this factor. For example, estimation as one of
activities at planning phase has a significant impact on the success of development.
The team maturity can impact on the success of estimation. For example, the
experienced people can make the more accurate estimation for project comparing
with the novices who make estimation based on the beliefs. So the consequence of
not ensuring this factor is the negative impact on estimation.

45
 K. Team Knowledge

Figure 5.11 The answer of respondent about Team knowledge

Team knowledge can be seen as a fundamental element in ASD. As respondent
7 said, team knowledge as a CSF factor at Agile planning phase represents team’s
knowledge about the problem and technology. Figure 5.12 display the reason why
team knowledge is mandatory to the success at Agile planning is that it has a great
impact on the delivery time and the quality of the solutio n. This factor can directly
and indirectly influence the success of ASD. If the team has enough knowledge,
many problems can be avoided in advance or be addressed in time. So it can directly
increase the success chance. According to the definition of team expertise in paper
[45], we can clearly find that team knowledge is one component of expertise.
Respondent 7 said this factor can indirectly influence many aspects. So he only
list one example to describe the importance of it. Team knowledge is directly related
to correctly estimate the time spent on a task. However, it is difficult to have an exact
estimation with team knowledge due to the differences between the developers. For
this challenge, the respondent 7 provided two solutions. The first is improving the
detail of the tasks to allocate people appropriately. In other words, according to the
degree of difficulty of each task, people who have different knowledge should be
allocated to appropriate task to make a better contribution. Second is investing in the
professionals employment to provide a more authoritative and accurate estimation.
Considering the vast impact of this factor, respondent 7 recommended that team
knowledge should be noticed as early as possible. If this CSF is not paid enough
attention, it may reduce the quality of the project or make the project later delivered.
What’s worse, the whole project may fail.

46
 L. Team collaboration

Figure 5.12 The answer of respondent about Team collaboration

The team as an important component of software development directly affects
the result of project. Thus the collaboration between not only the team members but
also the teams is particularly important. The close collaboration of teams can help the
team work toward same goal and improve the project performance of organization.
As respondent 8 said, a team sometimes cannot do the tasks separately which
needs other teams to cooperate with their works. However, the other teams may
unwilling to cooperate with their works. For addressing this challenge, the
respondent 8 suggested that the team should take the initiative to communicate with
other teams when facing the questions that need to be confirmed. Communication is
an effective way to enhance information sharing and collaboration between the
development teams and then the amount of team conflict was red uce and the teams
keep stable [5]. However, if multiple requests for the communication fail to respond,
the team should let leaders communicate with each other. The participation of leaders
can motivate the collaboration of teams, make them work together.
The close collaboration of teams promotes the process of development and
inspired team enthusiasm of working so that the project can be completed with high
quality on time. If the team treats collaboration negatively, the project cannot be
delivered as planned and the quality may be reduced.

47
 M. Requirement change

Figure 5.13 The answer of respondent about Requirement change

Due to the dynamic aspects environment of ASD, organizations need short
delivery cycles to cope with uncertainty and rapid change in requirements [42].
Requirement change was proved to be the CSF that can contribute to ASD success
[25]. The change indicates the characteristic of requirements is volatility. This
volatility is usually caused by the defects or the changes from the product owner. The
volatile of requirements has been the common problem of early estimation for more
than fifty years [25]. The changes of requirements sometimes are genera ted within
the development team. For instance, a new requirement is added to complete a new
functionality [42].
Although the requirement change is inevitable, we should devote ourselves to
reduce the negative influence of the project, such as the increasing of investment,
efforts, and risks. Thus the challenge for us is to reduce these negative effects.
Respondent 9 provided two solutions to address these challenges. The first is adding
requirements validation phase and ensuring the effectiveness of validation. After
defining the requirements with all stakeholders, they should confirm the
requirements and scenarios again. Second is the assessment of change. If the change
of requirements doesn't have a strong effect, these changed requirements can be
developed in next version.
This CSF closely relate to the Agile practices: prioritization and estimation. If
ignore it, the cost and effort of prioritization and estimation will increase, the quality
of project will reduce, and the product cannot be delivered on time.
5.2.2 Summary of collected factors
Through the above analysis of each factor, we can clearly see the challenges and
solutions when managing these factors. In this section, due to the respondents
answered questions from different angles, we divided these factors into different
categories identified by 13 papers at table 4.2. Then we focus on analyzing the
collected factors from different categories. The following table presented the
categories of the 13 collected factors.

48
 Table 5.2 Category of each CSF
ID Factor name Category

1 Top engineers
People factor
2 Communication (Individual- level)

3 Expertise

4 Productive team of good level

5 Team’s maturity
People factor
6 Team knowledge (Team-level)

7 Team collaboration

8 Coordination

9 Joint understanding of uncertainty

10 Conciliate deadline between client and team

11 Priorities Process factor

12 Level of details

13 Requirement change

From the table 4.2 mention in chapter 4, we can clearly see that CSFs are
usually divided into four widely used categories: organization factor, people factor,
process factor and project factor. Based on these four categories, the 13 collected
factors are divided into people factor and process factor. The people factor is also
divided into individual- level and team- level. There is one important reason why
organization factor has not been mentioned by the respondents. From table 4.2, we
can clearly see that the most factors of organization category belong to whole
development lifecycle rather than agile planning phase. Thus, the respondents didn’t
mentioned the organization factor due to our survey investigate the CSFs at agile
planning phase. The existing papers still have argument about whether project factor
are related to project success. The paper [1] used the project factor as candidate CSFs
to conduct the survey. The result indicated that this category has little impact on
project success. Whereas, some papers (e.g. [5] and [23]) advocated that project
factor has influence on project success. But these papers based on literature review
didn’t conduct the survey to validate these factors. In our thesis, the respondent

49
didn’t propose factors that belong to project category, such as project size and project
complexity.
From table 5.2, we can clearly see that there are eight factors that belong to
people category which describes the factors from people aspect. Six of them belong
to team- level, the rest belong to individual- level. The individual- level factors focus
on individual capability, such as skills, communication etc. The team- level factors
are more concerned about the overall level of team, such as team knowledge,
experience and expertise and so on. The individual- level factors may directly affect
the team- level factors, due to a good team is made up with excellent individuals.
There are five factors that belong to process category. It contains the factors which
are involved in activities of the process, such as requirement analysis, priority,
estimation etc. They focus on the details of activities at agile planning phase.
From the above table, we can clearly see that the proportion of people factor is
biggest. This indicates that people category as a most important category at agile
planning phase need be paid more attention. The process factor as another important
category also needs be taken seriously. Based on the description of existing
papers, Organization category less focus on CSF at agile planning phase, it prefer to
the factors of whole development lifecycle. Thus, it can be paid less attention at agile
planning phase. However, this not indicates that organization category is unimportant
for ASD. It is difficult to judge the importance of project category, due to existing
papers have argument on this and our respondent didn’t proposed factors that belong
to this category.
5.3 Discussion
In this section, we will deeply discuss the CSFs collected through both
systematic mapping and survey. Subsection 5.3.1 provides the relations among CSFs
collect by survey. Subsection 5.3.2 discusses the comparison of results of two
methods. It is divided into two parts.
5.3.1 Relations of survey results
This section displays the analysis of the relation of survey results through two
aspects. At part A, we will analyze the relation between people category and process
category. At part B, we focus on the internal relations of team level CSFs. The
detailed analyses are displayed as follow.
A. Relations of defined categories
As we mentioned above, we divided the 13 collected factors into two categories:
people factor and process factor. In this part, we analyze the relationship between
these two categories. The following figure presented the detail relationship between
them.

Figure 5.14 The relations of defined categories

From figure 5.14, we can clearly see that the people factor has two sub-
categories: individual- level and team- level. A good team can be made up with
excellent individuals, so the individual- level factors can impact on the team- level

50
factor. Communication can impact on the Team collaboration and Coordination that
belong to the team- level factors. The factor communication emphasizes the ability of
communication which is useful for development teams to collaborate and coordinate.
The top engineering can impact on the Expertise, Productive team of good level,
Team’s maturity and Team knowledge. The top engineering illustrated that people
who has various abilities and these abilities are useful for team-level factors.
The people factor as the fundamental category which has most proportion of
factors can directly impact on the process factor. The individual- level and team- level
as the two sub-category of people factor also have influence on the process factor.
The top engineering that belongs to individual- level can impact the process factor,
such as Priorities, Level of details and Requirement change. Top engineering as
skilled people has enough ability to make a good priority, make a suitable level of
details for planning, and to solve the problems that caused by requirement change.
The communication that belongs to individual- level also can impact on the conciliate
deadline between client and team that belong to the process factor. If a manager has
a good communication skill, this will help himself to better solve the conflicts
between client and team.
A good development team has better capability to accomplish various activities
of process. Thus, the team- level of people factor also can impact on the process
factor. The team who has enough expertise, experience and knowledge can well deal
with prioritization, planning with reasonable detail level and requirement change.
Therefore, the Expertise, Productive team of good level, Team’s maturity, Team
knowledge and Team collaboration has impact on the Priorities, Level of details and
Requirement change. The Coordination emphasizes the close collaboration between
stakeholders and development teams. This factor has influence on the Joint
understanding of uncertainty that illustrate the stakeholders should understand that
the uncertainty of agile development. If stakeholder and team have a better
coordination, the stakeholder can understand the uncertainty well.
B. Relation of CSFs
Through the analysis of each factor, except the impact relations among CSFs in
different categories, we also found there are some inclusion relations among the
CSFs in team level category. There are six factors belong to team level category:
Expertise, Productive team of good level, Team’s maturity, Team knowledge, Team
collaboration, Coordination. Four of team have significant relations that shown as
figure 5.16.

Figure 5.15 The relations among CSFs collected by survey

51
 Figure 5.15 indicates the relationship of the internal factors of team level
category. Expect team collaboration and Coordination, the other four factors have
inclusion relationship. The inclusion relationship of these four factors is defined
according to the description of each CSF in subsection 5.2.2. Factors that have
inclusion relationship are seen as similar factors. Productive team of good level can
be described from different aspects, such as team expertise, knowledge, experience,
skill etc. All these aspects can illustrate the level of the productive team. Besides, the
team maturity more emphasizes the experience of team by respondent. Thus,
productive team of good level has the widest scope than the other three CSFs. As
analyzed above, team expertise contains many abilities that include the knowledge of
team. So it has wider scope than team knowledge.
5.3.2 Comparison of two methods results
As it is displayed in section 4, we collected CSFs through mapping and survey.
In the above two subsections, we analyzed the results of these two parts separately.
The comparative analysis of these two results will be presented in this subsection.
Through this analysis, we can clearly find the similarities and differences of these
two kinds of results.

Figure 5.16 The relations between mapping results and survey result
Ps: A B represents: Factor A has same meaning with factor B.
A B represents: Factor A is included in factor B, but factor B is not included in
factor A. In other words, the factor B has wider meaning than A.
In this figure, the far left list displays the CSFs that belong to the results of
systematic mapping. The frequency of all these factors is higher than 3. The middle
and far right lists display the CSFs that belong to the results of the survey. The
arrows are used to describe the relationships between the factors that came from
different results of methods. At following subsections will separately introduce these
similarities and differences of these two kinds of results. Part A provides the analysis
of the similar CSFs shown in two methods. The rest CSFs of survey are new
proposed factors that are discussed in Part B.
A. Similarities
From figure 5.17, we can clearly see that communication, team expertise, team
experience, requirements and specification uncertainty are mentioned in both results
52
of methods. The percentage of communication in systematic mapping is over than
45%. The percentage of experience, expertise, and requirements and specification
uncertainty is 30.8%. Although the percentage of these factors is less than 50%, the
papers identified by mapping focus on the factors of the whole lifecycle rather than
agile planning phase. Thus, the percentage of these factors has been relatively high.
And the respondents use their working experience expressing these factors are
mandatory to the success of development at agile planning phase. These data indeed
demonstrate that these factors are CSFs at agile planning phase and should be given
enough attention.
Team knowledge as a CSF proposed by respondent recommends the importance
of the knowledge about the problem and used technology. The description of team
expertise in current literature includes many abilities, such as ability to work with
uncertain objectives, ability to work with top management, ability to work effectively
as a team, ability to understand human implications of a new system, and ability to
carry out tasks effectively [45]. So team expertise has wider scope and meaning than
team knowledge. In other words, team knowledge is included in team expertise. Due
to these two factors directly influence in the results of a project, they are mandatory
to the success of software development at Agile planning.
As the respondent said, productive team of a good level as a CSF has great
influence on agile planning, such as accuracy of estimation and reliability of delivery
date. This factor can be described from different aspects. For example, both team
expertise and experience can illustrate the level of the productive team. The
productive team of good level proposed by the respondent has a wider meaning than
the other two factors mentioned in papers. Thus, the team expertise and team
experience are included in the productive team of good level. These factors should be
paid enough attention at agile planning phase.
As the respondent said, Coordination emphasizes the collaboration between the
stakeholders and development teams. The customer involvement illustrates that the
customers should involve in the activities of ASD. These two factors all indicate that
the customers and development teams should have close collaboration. However, the
stakeholder didn’t just refer to customers, it also contains other roles, such as users,
architect, manager etc. Thus, the coordination has wider meaning than the customer
involvement.
From the above picture, we can clearly see that the seven factors proposed by
respondents have relationship with the results of mapping. This indicated that the
results of mapping strongly support the survey results. These factors need be
managed in agile planning phase.
B. Differences
Except the factors mentioned above, respondents according to their experience
proposed six new factors: Joint understanding of uncertainty, Level of details, Team
collaboration, Top engineers, Priorities, Conciliate deadline between client and
team. Although these factors are not mentioned in papers, they are mandatory to
success at agile planning phase in practice.
Joint understanding of uncertainty emphasizes the importance of understanding
uncertainty with agile thinking, and this factor directly affects whether the project
can be implemented as planned. The level of details emphasizes the details of
activities. If an activity has an appropriate detail, it is very helpful for us to make a
more exact estimation. Team collaboration illustrates the demand of collaboration
between teams. This factor also influences the delivery and quality of the project.
Top engineers are considered from people category, this factor emphasizes the

53
people with high skills can deliver top level results and have the ability to innovate
fast. Priorities as a basic activity in agile planning phase play a pivotal role. The
optimal priority can reduce redundancy and improve productivity so that the project
with high quality can be delivered with less time and money. Conciliate deadline
between client and team focuses on balancing the conflict between client deadline
and team estimation. This factor may affect the developer's enthusiasm, thereby
affecting the development process of the project. As a result, these factors also
should be managed at Agile planning phase.

54
6 CONCLUSION AND FUTURE WORK
Critical success factors are proved to be mandatory to the success of ASD at
Agile planning. The goal of this paper was to identify the critical success factors at
Agile planning phase and challenges associated with each factor. Systematic
mapping was conducted to collect CSFs of ASD in existing research. The survey
based on the questionnaire was performed to identify 13 CSFs and in-depth study of
each factor. The respondents come from different companies, and they have enough
experience on ASD. Thus they can provide the effective answers for our research.
Statistical analysis and qualitative content analysis were respectively used to analyze
the collected data of systematic mapping and survey.
This section presents a description of the conclusion drawn from this thesis. The
description of each research question will be displayed in subsection 6.1. Subsection
6.2 will illustrate the contribution of our thesis. Finally, the validity threats and future
work will be separately presented in subsection 6.3 and 6.4.

6.1 Conclusion of research questions

RQ1: What are critical success factors for the success at Agile planning phase?
In order to explore CSFs that are mandatory for the success of ASD at Agile
planning, we perform two methods. Firstly, we conduct systematic mapping to
understand the status of the art on the CSFs in existing literature (RQ1.a). Secondly,
we perform a survey to investigate the factors that are mandatory for the success of
ASD at Agile planning phase in practice (RQ1.b).
For systematic mapping, the numbers of papers that focus on CSF of ASD at
Agile planning are limited. As a result, the search scope is extended as agile software
development. The lifecycle of agile software development include the planning phase,
thus the collected data from the systematic mapping is effective. Through the
systematic mapping, we finally identified 13 papers and 47 critical success factors
for agile software development. In order to better compare with the results of the
survey, we also make a frequency analysis for identified factors. There are 16 factors
that the frequency of total citation count is greater than 3. The same or similar factors
with survey results are derived from these 16 factors. Comparing with the factors
which frequencies are lower than 4, these 16 factors need to be paid more attention in
agile software development.
For survey with an online questionnaire, there are four main questions that can
help us deeply study the CSF at Agile planning. Nine experienced people of different
companies contributed their opinions to each question. Eventually, we collected 13
effective answers of each question, which means 13 factors had been described in
detail. There are four of them have same meaning with the factors collected through
mapping. They are communication, expertise, team’s maturity and requirements
change. Another three factors are similar to the mapping results. They are team
knowledge, productive team of good level and Coordination. So the results of two
methods can support each other to prove that these CSFs can significantly impact on
the success of ASD. The rest six factors are Joint understanding of uncertainty,
Conciliate deadline between client and team, Level of details, Team collaboration,
Top engineers, and Priorities. These factors are initially proposed by the 9
respondents according to their development experience. To draw a conclusion, the 13
factors are proved should be managed in practices.
RQ2: What are the challenges associated with each factor?

55
 In order to make an in-depth study of factors, we perform a questionnaire to
investigate the challenges that encountered in managing these factors collected by
survey. Due to different factor has different challenges, we list the details of the
challenge for each factor in section 5.2. Through the collected challenges, the
organization can prepare for these challenges ahead of time. Like this, the agile
planning may be more accurate and the project may be completed more smoothly.
RQ3: How do practitioners address these challenges?
The experienced people proposed the corresponding solutions according to the
challenges of each factor. Since the solutions vary with the different challenges, the
details of them are presented in section 5.2. The solutions can be seen as guidelines
to help organizations address the challenges or make the preparations. Through the
answers of RQ2 and RQ3, we can get the specific measures to manage critical
success factors in practice.
RQ4: What are the consequences of not ensuring the factors?
The four key components of project’s success are time, cost, quality, and scope.
The consequence of not ensuring these factors influence s directly or indirectly
impact on the success of ASD. For the directly impaction, if not ensuring each factor
the ASD projects may have low quality, deliver late, increase the cost etc. For
instance, frequent changes in requirements can increase the cost, reduce the quality,
and delay the delivery time. For the indirectly impaction, if not ensuring each factor
the activities of ASD may not be done as supposed. For example, if the factors level
of details is not managed properly, people may not make accurate estimation. The
consequences vary form the CSFs. The details of consequences of each factor are
presented in section 5.2.2. These consequences can highlight the importance of
managing these CSFs so that people can pay more attention on them at planning
phase.
6.2 Contribution of this thesis
Through answering the research questions, we achieve all the objectives of
thesis successfully. Compare with the existing literature, our thesis contributions are
shown as follows.
For the mapping results, we have three contributions as follows:
1. Due to there is no paper just focus on the CSF at agile planning phase, we
extend the search scope as whole lifecycle of ASD. Thus, we present the
CSFs of ASD proposed by existing literatures and give a priority for these
collected factors. This make reader better understand the state of CSFs of
ASD.
2. The results of mapping method were compared with the results of survey
method. The four factors are both mentioned in two method s and three factors
have similar meaning. This can illustrate that these results of two methods
can be support each other and the data obtained from two methods are
effective.
3. At the start of the thesis, we want to conduct two questionnaires to investigate
the CSFs at agile planning phase. One is used to collect the free opinions of
respondents based on the open-ended questions which are not limit the mind
of respondents. We have conducted this questionnaire. The other is used to
extract the CSFs at agile planning based on the results of mapping. However,

56
 we didn’t have enough time and people resource to conduct this questionnaire.
Thus, the results of mapping method can be seen as the input to support the
future work.
For the survey results, we also have three contributions as follows:
4. Identify the critical success factors at Agile planning phase that are
mandatory to the success of ASD. Comparing with existing literature, we
identified 13 CSFs at agile planning phase rather than whole lifecycle and the
6 of 13CSFs are new proposed by respondents.
5. Identify the challenges that are associated with each factor, and investigate
the ways for addressing the proposed challenges. We make an in-depth study
on CSFs rather than just identify these factors. This in-depth study on CSFs is
the novel and high innovation degree study compared to existing research.
People can use these data as guidelines when they come across same or
similar troubles in practices.
6. Identify the consequences of not ensuring these factors. We gain a list of
consequences that are not mentioned in existing literature. The consequences
can remind people the importance of managing these CSFs for the success of
ASD at planning phase. It can also prove that the list of factors we provided
is all mandatory to the success of ASD.

6.3 Validity threats

Construct validity is concerned with the degree to which the theory behind the
experiment measures the observation [30] [34]. In this case, considering the survey
method we used in this thesis, the threat to this validity is related to the difference of
respondents. It was mitigated by collecting answers from various types of
respondents who are working in different companies, regions and countries.
Conclusion validity is the degree to which conclusions we reach about relationships
in the observations are incorrect due to errors in the data sources [30, 34]. We
performed three measures to mitigate this kind of threat. Firstly, we conducted a
pilot test by other researchers to validate the reliability of survey to ensure the
questions are clear and straight forward to answer [30]. Secondly, the target
population we selected has rich experience on agile software development and they
can give more accurate answers for our questionnaire. Lastly, we sent the e-mail to
every respondent to confirm whether our analysis accurately expressed their views.
We improved our analysis based on their suggestions until the respondents agree
with our analysis.
Inte rnal validity is the approximate truth about inferences regarding cause-effect or
causal relationship between the treatment and outcomes [30, 34]. To mitigate the
learning affect the answers of survey, we allow pre respondent just to answer the
questionnaire one time. Besides, to mitigate discomfort affect the respondents since
the face to face communication, we conducted an online-questionnaire. Finally, the
time of answering the questionnaire was no more than 30 minutes, this may remove
the impaction of fatigue on respondents.
External validity is concerned with the extent to which the results obtained from
study can be generalized to other context and other people [30] [34]. The selection
of participants and content may affect external validity. In this case, the small
number of respondents is a threat to validity. To mitigate this threat, we used the

57
 snowball sampling method to collect data from empirical respondents. However, we
only receive nine feedbacks because of many reasons, such as time and people
resource limitations.

6.4 Future works

Firstly, we used the mapping results as an evidence to support the survey result
in this thesis. Thus, those factors shown in mapping will be studied in depth in
follow-up studies. For instance, whether these factors are mandatory to the success at
Agile planning? What consequences will happen if ignore them? The second is
adding the number of respondents to enrich and extend survey results. The survey in
this paper is very close to the method of interview and has gained effective results.
Thus, we can conduct both survey and interview as much as possible in future work
to collect more effective data. Considering the importance of CSFs, these
investigations are meaningful and necessary.

58
REFERENCES
[1] T. Chow and D.-B. Cao, “A survey study of critical success factors in Agile
software projects,” Journal of Systems and Software, vol. 81, no. 6, pp. 961–971,
2008.
[2] S. C. Misra, V. Kumar, and U. Kumar, “Identifying some important success
factors in adopting Agile software development practices,” Journal of Systems and
Software, vol. 82, no. 11, pp. 1869–1890, 2009.
[3] K. Waters, All About Agile: Agile Management Made Easy! Createspace
Independent Publishing Platform, 2012.
[4] M. Cohn, Agile Estimating and Planning. Pearson Education, 2005.
[5] A. Ahimbisibwe, R. Y. Cavana, and U. Daellenbach, “A contingency fit model of
critical success factors for software development projects: A comparison of Agile
and traditional plan-based methodologies,” Journal of Enterprise Information
Management, vol. 28, no. 1, pp. 7–33, Feb. 2015.
[6] I. Sommerville, Software Engineering, 9 edition. Boston: Pearson, 2010.
[7] S. M. Biju, “Agile Software Development,” E-Learning, vol. 5, no. 1, p. 97, 2008.
[8] Kent Beck, James Grenning, Robert C. Martin, Mike Beedle, Jim Highsmith,
||Steve Mellor, Arie van Bennekum, Andrew Hunt, Ken Schwaber, Alistair Cockburn,
Ron Jeffries, Jeff Sutherland, Ward Cunningham, Jon Kern, Dave Thomas, Martin
Fowler, Brian Marick (2001). "Manifesto for Agile Software Development". Agile
Alliance. Retrieved 14 June 2010
[9] Cohen, David, Mikael Lindvall, and Patricia Costa. "Agile software
development." DACS SOAR Report 11 (2003).
[10] S. Sultana, Y. H. Motla, S. Asghar, M. Jamal, and R. Azad, “A hybrid model by
integrating Agile practices for Pakistani software industry,” in 2014 Internat ional
Conference on Electronics, Communications and Computers (CONIELECOMP),
2014, pp. 256–262.
[11] Highsmith, J., Agile Software Development Ecosystems, Boston, MA: Addison-
Wesley, 2002a.
[12] B. Choudhary and S. K. Rakesh, “An approach using Agile method for software
development,” in 2016 International Conference on Innovation and Challenges in
Cyber Security (ICICCS-INBUSH), 2016, pp. 155–158.
[13] K. Schwaber, “Agile Project Management with Scrum”. Microsoft Press, 2004.
[14] Hubert Smits, “5 Levels of Agile Planning: From Enterprise Product Vision to
Team Stand-up,” Rally Software Development Corporation, 2006.
[15] PMI, 2004. Project Management Body of Knowledge (PMBOK). third ed.
Project Management Institute. USA.
[16] Cohn, M., Ford, D., “ Introducing an agile process to an organization”. 2003.
Computer 36 (6), pp. 74–78.
[17] Lindvall, M., Muthig, D., Dagnino, A., Wallin, C., Stupperich, M., Kiefer, D., et
al., “Agile software development in large organizations”. 2004. Computer 37 (12),
26–34.
[18] Rockart, J., “Chief executives define their own information needs,” Harvard
Business Review March/April, p.81–92, 1979.
[19] Bullen, C.V., Rockart, J.F., “ A Primer on critical success factors”, Information
Systems Research no. 69, p. 385, 1981.
[20] Pinto JK, Rouhiainen PJ. “Building Customer-Based Project Organizations”,
John Wiley and Sons, p. 87. 2001.
[21] D. Stankovic, V. Nikolic, M. Djordjevic, and D.-B. Cao, “A survey study of

59
critical success factors in Agile software projects in former Yugoslavia IT
companies,” Journal of Systems and Software, vol. 86, no. 6, pp. 1663–1678, Jun.
2013.
[22] J. A. Livermore, “Factors that significantly impact the implementation of an
Agile software development methodology,” Journal of Software, vol. 3, no. 4, pp.
31–36, 2008.
[23] D. Shahane, P. Jamsandekar, and D. Shahane, “Factors influencing the Agile
methods in practice - Literature survey amp; review,” in 2014 International
Conference on Computing for Sustainable Global Development (INDIACom), 2014,
pp. 556–560.
[24] K. Dikert, M. Paasivaara, and C. Lassenius, “Challenges and success factors for
large-scale Agile transformations: A systematic literature review,” Journal of
Systems and Software, vol. 119, pp. 87–108, Sep. 2016.
[25] M. H. N. Nasir and S. Sahibuddin, “Critical success factors for software projects:
A comparative study,” Scientific research and essays, vol. 6, no. 10, pp. 2174–2186,
2011.
[26] T. Gradel and J. T. Nosek, “Model of Critical Factors for Outsourcing Agile
Development,” AMCIS 2009 Proceedings, p. 468, 2009.
[27] E. Kouzari, V. C. Gerogiannis, I. Stamelos, and G. Kakarontzas, “Critical
success factors and barriers for lightweight software process improvement in Agile
development: A literature review,” in 2015 10th International Joint Conference on
Software Technologies (ICSOFT), 2015, vol. 1, pp. 1–9.
[28] R. Bavani, “Critical success factors in distributed Agile for outsourced product
development,” 2009.
[29] S. MOHANARAJAH and M. A. JABAR, “An improved adaptive and dynamic
hybrid Agile methodology to enhance software project success deliveries,” Journal of
Theoretical & Applied Information Technology, vol. 75, no. 3, 2015.
[30] R. Britto, Blekinge Tekniska Högskola, and Institutionen för programvaruteknik,
“Knowledge classification for supporting effort estimation in global software
engineering projects,” Department of Software Engineering, Blekinge Institute of
Technology, Karlskrona, 2015.
[31] M. J. Grant and A. Booth, “A typology of reviews: an analysis of 14 review
types and associated methodologies,” Health Information & Libraries Journal, vol.
26, no. 2, pp. 91–108, Jun. 2009.
[32] B. Kitcheham and S. Charters. Guidelines for performing Systematic Literature
Reviews in Software Engineering. Technical report, Keele University, 2007.
[33] D. Kahneman, A. B. Krueger, D. A. Schkade, N. Schwarz, and A. A. Stone, “A
Survey Method for Characterizing Daily Life Experience: The Day Reconstruction
Method,” Science, vol. 306, no. 5702, pp. 1776–1780, Dec. 2004.
[34] Wohlin, Claes, et al. Experimentation in software engineering. Springer Science
& Business Media, 2012.
[35] Linaker, Johan, et al, ―Guidelines for Conducting Surveys in Software
Engineering. 2015.
[36] J. F. Gubrium and J. A. Holstein, Handbook of Interview Research: Context and
Method. SAGE, pp.3-32. 2002.
[37] Trochim, William M. K. (2006). "Descriptive statistics". Research Methods
Knowledge Base. Retrieved 14 March 2011.
[38] L. Jacoby and L. A. Siminoff, Eds., Empirical methods for bioethics: a primer, 1.
ed. Amsterdam: Elsevier JAI, 2008, pp. 39-62
[39] H.-F. Hsieh, “Three Approaches to Qualitative Content Analysis,” Qualitative

60
Health Research, vol. 15, no. 9, pp. 1277–1288, Nov. 2005.459–472, Apr. 2013.
[40] U. Flick, The SAGE Handbook of Qualitative Data Analysis. SAGE, 2013.
pp.170-180
[41] E. v Kelle, J. Visser, A. Plaat, and P. v d Wijst, “An Empirical Study into Social
Success Factors for Agile Software Development,” in 2015 IEEE/ACM 8th
International Workshop on Cooperative and Human Aspects of Software
Engineering, 2015, pp. 77–80.
[42] J. Sheffield and J. Lemétayer, “Factors associated with the software
development agility of successful projects,” International Journal of Project
Management, vol. 31, no. 3, pp, 2013.
[43] M. Vaismoradi, H. Turunen, and T. Bondas, “Content analysis and thematic
analysis: Implications for conducting a qualitative descriptive study,” Nurs Health
Sci, vol. 15, no. 3, pp. 398–405, Sep. 2013.
[44] Pikkarainen, M., Haikara, J., Salo, O., Abrahamsson, P. & Still, J. (2008) 'The
impact of agile processes on communication in software development'. Empirical
Software Engineering. (13). pp. 303-337.
[45] Jiang. J. and Klein, G., “Software development risks to project effectiveness”,
Journal of Systems and Software, Vol. 52 No. 1, pp. 3-10. 2000.
[46] G. Melnik and F. Maurer, “Comparative analysis of job satisfaction in agile and
non-agile software development teams,” in International Conference on Extreme
Programming and Agile Processes in Software Engineering, 2006, pp. 32–42.
[47] S. Nerur, R. Mahapatra, and G. Mangalaraj, “Challenges of migrating to agile
methodologies, Communications of the ACM, vol. 48, no. 5, pp. 72-78, 2005.
[48] J.R. Turner, and R. Müller, “Communication and co-operation on projects
between the project owner as principal and the project manager as agent,” European
Management Journal, vol. 22, no. 3, pp. 327-336, 2004.
[49] Wiegers K. First things first: prioritizing requirements [J]. Software
Development, 1999, 7(9): 48-53.
[50] K. E. W. and J. Beatty, “First things first: Setting requirement priorities,” in
Software Requirements, 3rd ed., Microsoft Press, 2013.
[51] J. Highsmith and A. Cockburn, “Agile software development: the business of
innovation,” Computer, vol. 34, no. 9, pp. 120–127, Sep. 2001.
[52] Faraj Samer and Spruoll Lee, Coordinating Expertise in Software Development
Teams, Management Science, Vol 46, No 12. 2000.
[53] S. Zhang, M. Tremaine, J. Fjermestad, A. Milewski, and P. O’Sullivan,
“Delegation in Virtual Team: the Moderating Effects of Team Maturity and Team
Distance,” in 2006 IEEE International Conference on Global Software Engineering
(ICGSE’06), 2006, pp. 62–68.

61
APPENDIX A. SURVEY QUESTIONNAIRE

Critical success factors in agile planning phase

Introduction:
We are two students at BTH who are working on the master thesis. In order to
improve the probability of successful software development, we want to focus on the
CSFs (critical success factors) of agile software development at planning phase. This
phase contains two levels: release planning and iteration planning. Release planning
is a high- level planning, which outline the scope of the development project.
Iteration Planning is a feature- level plan which outlines the scope of product feature
in detail. The planning phase as early stage of development life cycle has a great
influence on the successful development of the software. Our goal is to identify
CSFs related to agile planning.
It may take about 30 minutes of your time. We really appreciate your input!
If you have any questions about the survey, please email us:
[email protected], [email protected].
Part one: Individual information:
Name: Country:

Email: Domain of the company:

Working role: Company size (number of employees):

Company name:

Years of experience working with

agile software development:

Part two: Open-ended questions:

Please fill out the forms according to the following questions.
1. What factors are mandatory for the success of agile planning?
2. What are the challenges associated with each factor?
3. What do you use to do to address these challenges?
4. What are the consequences of not ensuring these factors?
Details of questions:
1. Factor name:___________
2. Factor type:
 Release planning
 Iteration planning
 Both
3. Description: ________________
4. Challenges: _________________
5. How to address each challenge?
6. Consequences: _______________

62
APPENDIXB
敏捷软件成功开发的影响因素调查
序号 :

个人信息：
姓名 国家

电子邮件 公司的领域

公司规模（员 职位
工的人数）

从事敏捷软件开发的年限

相关简介：
我们是瑞典布莱京理工学院的在读研究生。我们的研究论文方向是敏捷软
件开发。敏捷开发方法从 2001开始备受关注，由于它尽早和不断交付有价值
开始备受关注，由于它尽早和不断交付有价值
的软件满足客户需求而大受欢迎。然而由于一些因素导致了软件开发失败的例
子比比皆是。为了解决这个问题，我们想通过问卷调查明确有哪些因素可以在
计划阶段影响敏捷软件开发。计划阶段作为软件开发生命周期中的早期阶段，
计划阶段影响敏捷软件开发。计划阶段作为软件开发生 命周期中的早期阶段，
对软件的成功开发有着举足轻重的作用。在敏捷开发当中，计划阶段又可分为
r
elea
se计划和 sprin
t计划。
这个问卷调查大概会占用你 30分钟的时间，非常感谢你抽出时间完成我们
的调查，你的意见和观点会给我们很大的帮助。问卷调查中涉及到的个人信息
我们会保密。如果对此调查有困惑或者疑问，可以通过以下邮箱联系我们，我
们会尽快回复。
翟志超：350250301@qq
翟志超： .
com
刘迪：
刘迪 ： 654459373@qq
.com

6
3
问卷调查：（注：每个表格填写一个影响因素，如有多个因素，请自行复制表格，谢谢合作！）
表格一
表格 一

影响因素（在开发阶段影
响软件成功开发的因素）

因素的类型 R
elea
se计划  Sp
rin
t 计划  两者都是 

因素的描述

在管理这个影响因素时
会遇到什么挑战/
会遇到什么挑战 困难

如何解决各项挑战
如何解决各 项挑战/
困难

如果忽视这个因素会有
什么后果

6
4