Information and Software Technology xxx (2013) xxx–xxx

Contents lists available at SciVerse ScienceDirect

Information and Software Technology

journal homepage: www.elsevier.com/locate/infsof

An empirical study on the implementation and evaluation

of a goal-driven software development risk management model
Shareeful Islam a,⇑, Haralambos Mouratidis a, Edgar R. Weippl b
a
School of Architecture, Computing, and Engineering, University of East London, 4-6 University Way, London E16 2RD, United Kingdom
b
SBA Research gGmbh, Sommerpalais Harrach, Favoritenstrasse 16, 1040 Wien, Austria

a r t i c l e i n f o a b s t r a c t

Article history: Context: Building a quality software product in the shortest possible time to satisfy the global market
Received 17 October 2012 demand gives an enterprise a competitive advantage. However, uncertainties and risks exist at every
Received in revised form 23 June 2013 stage of a software development project. These can have an extremely high influence on the success of
Accepted 25 June 2013
the final software product. Early risk management practice is effective to manage such risks and contrib-
Available online xxxx
utes effectively towards the project success.
Objective: Despite risk management approaches, a detailed guideline that explains where to integrate
Keywords:
risk management activities into the project is still missing. Little effort has been directed towards the
Software risk management
Goal modelling language
evaluation of the overall impact of a risk management method. We present a Goal-driven Software Devel-
Requirements engineering opment Risk Management Model (GSRM) and its explicit integration into the requirements engineering
Empirical study phase and an empirical investigation result of applying GSRM into a project.
Goal-risk model Method: We combine the case study method with action research so that the results from the case study
directly contribute to manage the studied project risks and to identify ways to improve the proposed
methodology. The data is collected from multiple sources and analysed both in a qualitative and quanti-
tative way.
Results: When risk factors are beyond the control of the project manager and project environment, it is
difficult to control these risks. The project scope affects all the dimensions of risk. GSRM is a reasonable
risk management method that can be employed in an industrial context. The study results have been
compared against other study results in order to generalise findings and identify contextual factors.
Conclusion: A formal early stage risk management practice provides early warning related to the prob-
lems that exists in a project, and it contributes to the overall project success. It is not necessary to always
consider budget and schedule constraints as top priority. There exist issues such as requirements, change
management, and user satisfaction which can influence these constraints.
Ó 2013 Elsevier B.V. All rights reserved.

1. Introduction failure [7]. An advantage of considering risk management during

the requirements engineering phase is that such integration en-
Software projects, by inherent nature, contain a significant ables the identification of expensive and persistent requirement
number of uncertainties such as time-to-market, budget and sche- problems [8]. However, the literature fails to provide comprehen-
dule estimation, technology evolution, and stakeholders’ expecta- sive and detailed guidelines and clear evidence on how to integrate
tions. Failure to control such uncertainties imposes potential risk management activities at the early development stages
risks on a project. Software risk management can be used as a tool [9,10,6]. Moreover, although several study results exist in the
to manage these risks and to reason about the uncertainties in- literature on identifying risk factors in software projects, only
volved. There exist several risk management methods for manag- few reports are available on evaluating the impact of an overall risk
ing risks in software projects [1–5]. Practitioners and researchers management method onto a software project [4]. Research studies
agree that for a risk management practice to be effective, it needs indicate that risk management is not well applied in practice [11]
to be included at the early phase of the development process [6], and practitioners are more concerned on the tangible development
since requirement problems are one of the main causes of project cost, which provides direct benefits in terms of project deliverables
[9]. For a successful project, it is difficult to prove that any part of
⇑ Corresponding author. Tel.: +442082237273. the resulting product is influenced by software risk management
E-mail addresses: [email protected] (S. Islam), [email protected] (H. Mouratidis),
[12]. Therefore, it is necessary to integrate risk management
[email protected] (E.R. Weippl). approaches into the early development and to create awareness

0950-5849/$ - see front matter Ó 2013 Elsevier B.V. All rights reserved.
http://dx.doi.org/10.1016/j.infsof.2013.06.003

Please cite this article in press as: S. Islam et al., An empirical study on the implementation and evaluation of a goal-driven software development risk
management model, Inform. Softw. Technol. (2013), http://dx.doi.org/10.1016/j.infsof.2013.06.003
2 S. Islam et al. / Information and Software Technology xxx (2013) xxx–xxx

among the practitioners about the impact of risk management identification, resolution and continuous monitoring of risks. How-
practice on software projects. ever, the approach requires intensive active involvement of project
Within this context, the novel contributions of this paper are (i) customers/users, which is difficult to attain in real on-going project
a goal-driven risk management method; (ii) an explicit integration situations. The Software Engineering Institute (SEI) provides a
of the risk management method into the requirements engineering comprehensive framework to support continuous risk manage-
phase; and (iii) an empirical evaluation of the impact of the risk ment activities [16,17]. The approach concerns identification, anal-
management method into a software development project. The ysis, communication and mitigation strategies for software risk
presented risk management method is based on the KAOS goal management and depends on risk taxonomy [18]. SEI also devel-
modelling language [8]. In particular, GSRM adopts goal and obsta- oped a process improvement model, Capability Maturity Model
cle concepts from the KAOS goal modelling language and extends it Integration (CMMI), which provides close correlation between
with risk assessment and treatment [6]. We have decided to build the quality of software products and the quality of the software
our work on existing research on goal modelling because goals and development processes [19]. CMMI is compatible with ISO stan-
risks are complementary entities of a software project. A risk is dard for software process assessment, i.e. ISO/IEC 15504 [14].
usually defined as negation to single or multiple goals or loss of CMMI considers continuous risk management as an important fea-
attainment of corresponding objectives [8]. As such, the goal-dri- ture with concepts like risk management strategy, identifying and
ven approach anchors the risk management activities and allows analysing risks and risk control. ISO 31000:2009 provides process,
to trace and rationalise the risk factors, events and control actions framework and a number of principles for an effective risk man-
with respect to the goals. Furthermore, Goal-orient Requirements agement practice [20]. Risk management is considered as an inte-
Engineering (GORE) has long been recognised in requirements gral part of all organisational processes, including strategic
engineering community as an important paradigm to elicit, ana- planning and all project and change management processes. Kon-
lyse, and document requirements. As such, the decision to build tio proposed the Riskit methodology [4], which provides a com-
on KAOS allows us to explicitly integrate risk management into plete conceptual framework for risk management using a goal/
the early development phase. The methodology is explained with expectation approach from the stakeholders and risks which threa-
the aid of a carefully designed case study for the development of ten the goals. It provides precise and unambiguous definitions of
an automation system in a public sector organisation (Ministry of risks and aims at modelling and documenting risks qualitatively.
Planning Commission) under the e-governance project. The main At the heart of the approach, is the visual formalism of the risk
goal of empirical investigation is to evaluate the effectiveness of by analysing risk factors, risk events, risk reactions, risk effect sets
GSRM and in particular the impact of an early risk management and utility loss that would occur due to risk events. However, Riskit
practice on a software development project. Our work combines has some important limitations. There are no clear sources speci-
a case study method with action research, so that identified fied from where the goals originate and how the identified goals
treatment actions can be used to control the potential project risks. are modelled. Risks are analysed and prioritised by deriving sce-
The results from the case study outline the impact of GSRM on the narios, which is a non-trivial task when a scenario depends upon
project and compare the identified results with other literature more than one probabilistic element. Moreover, it is always hard
results. Such comparison demonstrates the impact of risk manage- to formulate a scenario from factors and attempt to perform a com-
ment, at requirements engineering level, on software development parison amongst them. Foo et al. [3] make use of a comprehensive
projects. questionnaire to construct the Software Risk Assessment Model
The remainder of the paper is structured as follows: Section 2 (SRAM). A set of questions are chosen for nine critical risk ele-
outlines the state of the art on risk management methods and risk ments, i.e. complexity, staff, targeted reliability, requirements,
factors. Section 3 provides an overview of the goal-driven risk method of estimation, monitoring, process, usability and tools.
management model and Section 4 outlines the integration of GSRM However, the main limitations of this approach are the lack of a de-
into requirements engineering. Section 5 demonstrates the evalu- tailed implementation of the model and the lack of a set rule for
ation results on the implementation of the GSRM into a software common weight value, which means that practitioners need to
project. Section 7 provides a critical discussion of the various parts determine a risk probability for each element. Roy’s pro risk man-
of GSRM and it outlines some of our experiences from the studied agement framework [5] is an extension of the AS/NZS standard
project. Section 8 provides validity of the study results. Section 9 4360:1999 [21]. The main attention of the framework is on the
finally provides summary and directions for future work. business component in which the project is created and the oper-
ational domain where the project is actually carried out.

2. Related works
2.2. Study results

Risk, in ISO Guide 73:2002, is defined as combination of the

2.2.1. Risk factors
probability of an event and its consequence [13]. It is constituted
A well known ‘‘top-ten’’ list of risk factors is provided by Boehm
upon three basic concepts: event, likelihood, and severity. Risk
[15]. After that several lists of risk factors have been published
management in software projects describes an integrated engi-
[22–26]. Among these contributions, Barki and Schmidt composed
neering approach with methods, processes and artefacts for identi-
the most comprehensive ones. Barki et al. [22] compiled a list of 35
fying, analysing, controlling and continuously monitoring risks in
risk variables, which were represented in the form of a question-
order to reduce the chance of project failure [14,4]. This section fo-
naire consisting of 144 items. The results of their survey are based
cuses on existing work that relates to our approach.
on the questionnaire showing five influential factors: technological
newness, application size, lack of expertise, application complexity
2.1. Risk management framework and organisational environment for the most interpretable solu-
tion of software risk. Moynihan [27] focuses on project constructs,
The theoretical foundation of putting risk management into a i.e. personal constructs and application which need to be consid-
single framework is initially contributed to Boehm [15] risk-driven ered by the project manager. The study observed that risk variables
Spiral model. Later, Boehm extended the original Spiral model identified by Barki et al. [22] lack the client’s apparent knowledge.
using the theory W (Win–Win) model [1] to satisfy the objectives Schmidt et al. [23] published a comprehensive list of risk factors
and concerns of the stakeholders. The model also supports risk and categorised the risk factors into several different areas such

Please cite this article in press as: S. Islam et al., An empirical study on the implementation and evaluation of a goal-driven software development risk
management model, Inform. Softw. Technol. (2013), http://dx.doi.org/10.1016/j.infsof.2013.06.003
 S. Islam et al. / Information and Software Technology xxx (2013) xxx–xxx 3

as corporate environment, sponsorship, relationship management, identify a comprehensive list of risks or how to estimate the like-
project management, scope, requirements, funding, scheduling, lihood of risk factors. Moreover, details about the output from
development process, staffing, technology and external dependen- the activities are also missing. The standard emphasises on under-
cies. Iacovou et al. [28] summarised a risk profile by the top ten risk standing the risk management context considering internal and
factors for offshore-outsourced development projects. The risk fac- external organisational issues, risk management process and risk
tors are ranked as ‘‘very important’’, ‘‘important’’ and ‘‘less impor- criteria. GSRM adopts this context concept and follows the guide-
tant’’ by focusing on three main areas: communication, client’s line to define the risk management context within the process.
internal management and vendor capabilities. Nakatsu et al. [26] The existing risk management frameworks, standards and survey
further investigated and compared the risk factors between off- results from practitioners emphasise on the integration of risk
shore and domestic outsourcing. The results showed that many management from the early development process.However, details
risk factors related to project management commonly appeared on integrating risk management activities into software projects
on the top of both domestic and offshore risk lists such as lack of are still missing and little work has been undertaken to understand
top management commitment, inadequate user involvement and the impact of an overall risk management framework on software
failure to manage end user expectation. Some are unique for the development.In contrast, our work improves the current state of
offshore context such as lack of communication, poor change con- the art by (i) introducing a goal-driven approach for risk manage-
trols, lack of business know-how and failure to consider all costs. ment; (ii) explicitly integrating risk management activities into
early development; (iii) identifying the techniques necessary to
2.2.2. Risk factors impact and risk management barriers perform the risk management activities and precisely defining
Some works have been undertaken to investigate the potential the artefacts produced by the activities and (iv) empirically evalu-
effects of risk factors on software development projects. Wallace ating the impact of a risk management method, in particular
et al. focused on risk factors from six different dimensions and GSRM, on an ongoing software project.
their effect on a project [24]. The study considered low, medium
and high risk projects and observed the impact of risk dimensions
3. Overview of the goal-driven software development risk
on the projects. The six risk dimensions are: user, team, require-
management model
ments, planning and control, complexity and organisational envi-
ronment. The results showed that risks associated with
The Goal-driven Software Development Risk Management Mod-
requirements, planning and control and organisational environ-
el (GSRM) is a framework that supports assessment and manage-
ment are more obvious in high risk projects. Ropponen et al. [29]
ment of risks from the early requirements engineering phase. It
identified six components of software development risks, which
is an extension of the KAOS goal modelling language with concepts
are: scheduling and timing, system functionality, subcontracting,
related to risk management. It is worth stating that the focus of
requirement management, resource usage and performance, and
this paper is not to present in detail the GSRM framework, but
personnel management. Jiang et al. [30] examined the risk impact
rather to present its applicability and its empirical usage in a case
on different system development aspects, in particular the study
study. GSRM has been presented in other publications [32–34,6]
looked at the 12 most common software development risks pro-
and this section gives a brief overview of GSRM.
posed by Barki [22]. Ropponen et al. showed that 75% of the project
managers did not follow any detailed risk management practice
and did not have adequate knowledge about software risk manage- 3.1. Levels of abstraction
ment [11]. However, for a successful project, it is difficult to prove
that any part of the resulting product was influenced by the soft- The model supports different levels of abstraction from goal to
ware risk management [12]. Kwak et al. observed that project obstacle and finally to treatment. Fig. 1 gives an overview of the
managers and practitioners perceive risk management activities different levels of abstraction.These levels build the bridge from
as extra work and expenses [31]. A recent survey study of experi- the goals of a software project to the risks that obstruct these goals
enced project managers on their perception of software risk man- and finally the treatments that reduce the risks to satisfy these
agement outlines that tangible development cost, lack of resources goals. On the top level, there are the goals, i.e. objectives, expecta-
and efforts from organisation or process change due to risk mitiga- tions and constraints from the development components and these
tion are the top three identified barriers of software risk manage- goals match with the project success indicators. The middle two
ment [9]. The result also emphasises other barriers such as: too levels include the risk factors and events, which directly or indi-
much risks to control, reward for problem solving not for risk mit- rectly obstruct the fulfillment of the goals. Risk events are then as-
igation, overconfidence regarding individual measurements that sessed to estimate the severity of the risk. At the bottom level,
pose challenges for the implementation of risk management in there are the control actions that obstruct the risks and associated
the development. consequences to attain the goals. This abstraction supports refine-
To summarise, risk management frameworks, standards and ment of goals and obstacles and establishment of the obstruction
study results discussed above are important, but they also demon- and contribution links amongst them.The more goals and risks
strate a number of limitations. Although Kontio’s Riskit is a goal- are refined, the better the support of risk management at an early
driven approach, it is not clear from where goals can originate stage. The Risk specification is the main artefact produced by the
and the risk analysis is based on scenarios which are difficult to model as shown in the right hand side of Fig. 1. On the left side,
formulate. ISO 31000:2009 provides generic high level guidelines the figure shows different models, i.e. goal model, obstacle model
for implementing risk management into organisational processes and causal relationship model. A goal model refines higher level
using process, framework and principals. But how risk manage- goals to lower level finely grained sub-goals through AND or OR
ment should be integrated into organisation processes and what refinement so that sub-goals contribute to the main goals. An
techniques are needed to perform risk management activities is obstacle model is the refinement of risk factors to the risk event
not clearly described by the standard. For instance, the risk and obstruction link from obstacle to the goals. Therefore, an
management process includes risk identification, analysis and obstacle model is a goal-risk model as it links from the obstacles
treatment activities. The standard recommends to identify a com- to the goals. Finally, a causal relationship model visualises the
prehensive list of risks and quantify the risks based on likelihood factors that are responsible for the occurrence of a risk event.
and consequence. But it does not provide any detail on how to Fig. 2 shows an abstract view of the model. Goals and obstacles

Please cite this article in press as: S. Islam et al., An empirical study on the implementation and evaluation of a goal-driven software development risk
management model, Inform. Softw. Technol. (2013), http://dx.doi.org/10.1016/j.infsof.2013.06.003
4 S. Islam et al. / Information and Software Technology xxx (2013) xxx–xxx

Overview of Goals

model
What are the issues relating to the project success?

Goal
What are the goals, expectations, constraints of the development component-
element-factor from the perspective of project success ?

Obstacle

Causal relationship model

Levels of abstraction
What are the risk factors that obstruct the goals ?
Obstacle model

What are the risk factors that create problems within the development?

Consequence
How are risk factors related with each other and contribute for the risk event
occurence?
How severe are the risks within the projectt?
What are the important risks for the project ?
Treatment and monitor
Risks specification
What are the control actions that countermeasure the risks?
How effective are the control actions?
Are there any new risks that arise within the development environment ?

Fig. 1. Overview of risk abstraction levels.

are derived from the development components, stakeholder activities within the layer consider these components to under-
expectations and project success indicators. The goal-risk model stand goals, obstacles and treatments from both technical and
and associated artefacts are the main outputs under the risk non-technical perspectives.
specification.
3.2.1. Goal layer
3.2. Framework Goal is the core concept of this approach and the goal layer fo-
cuses on the factors that contribute effectively to complete the pro-
The framework consists of four layers to support software ject activities and directly link to the project success. Such goals are
development risk management. The advantage of a layer based important as they describe what needs to be done for a successful
modelling framework is that it includes suitable tasks, methods project and who is responsible for achieving the goal. Goals are
and techniques for performing specific activities under any layer. project specific and focus on the economic benefit, project success
We categorise development components into five dimensions: pro- criteria and boundary, knowledge gathering and reuse, user satis-
ject execution, development process, product, human, and envi- faction, quality, vendor reputation, successful delivery, and other
ronment (internal and external) [35–37,33]. Therefore specific critical issues of the product. Goals can be stated at different levels

Development Stakeholders Project

components expectations success
indicators

derive
e

de
riv

ri ve
de

Goals
by
ed

at
ta
ct
ru

in
st
ob

construct

construct Goal-risk support

Risks Control
model actions

le a
d
influence
re

select
qu
ire

Treat &
Assess
lead monitor
co
ns

ad

Causal
tru

le
ct

relationship
model

Fig. 2. Conceptual view of the proposed approach.

Please cite this article in press as: S. Islam et al., An empirical study on the implementation and evaluation of a goal-driven software development risk
management model, Inform. Softw. Technol. (2013), http://dx.doi.org/10.1016/j.infsof.2013.06.003
 S. Islam et al. / Information and Software Technology xxx (2013) xxx–xxx 5

of abstraction from higher level coarsely grained to lower level fi- the requirements engineering phase. This activity defines the risk
nely-grained goal assertions. management scope, threshold and boundary, it allocates schedule
and formulates the risk management team. A project’s inherent
3.2.2. Obstacle layer riskiness nature is important for this activity. In particular, it deter-
Obstacles are the main causes that reduce the ability to achieve mines how risky a project is so that an appropriate treatment strat-
a single or multiple goals. We treat risk factors as obstacles that di- egy can be determined for the risk management. Once the risk
rectly or indirectly lead to a goal negation and create problems in management plan is initialised, the next activity involves the iden-
the project. Obstacle categories should be aligned with and derived tification and modelling of goals by focusing on the state of the
from the goal categories and model the situation about how sev- development components and matching them with the project
eral obstacles violate the identified goals. The obstacle layer iden- success indicators. It mainly identifies and categorises goals and
tifies the potential software development risk factors and it refines high level goals to provide more concrete meaning in terms
formulates the obstruction to the goal dissatisfaction. It provides of their contribution to the project success. The goal refinement
an overview of the risk factors that exist in the project.The risk activity results in the construction of a goal model from a parent
obstacle layer establishes the obstruction link from risk factors to goal to the sub-goals. The artefact produced by this activity is goal
sub-goals and from events to the main goal. detail that triggers subsequent risk assessment and management
activities. The next activity involves the identification and model-
3.2.3. Assessment layer ling of obstacles. We need to identify as many likely obstacles as
The assessment layer quantifies the risk events as a conse- possible so that the development team is aware of the problems
quence of single or multiple risk factors. This layer precisely anno- from the beginning. Obstacle identification extends the goal model
tates individual risk events and it establishes the causal by including obstruction links from obstacles to goals. This exten-
relationship model between risk factors and related risk events. sion supports the construction of goal-risk models that visualise
It also focuses on the severity of the risk events’ impact on goals. influential risk factors, which obstruct multiple goals. The follow-
For highly prioritised goals, obstacle identification and refinement ing activity assesses risk by estimating the risk level and relevant
should be extensive. It is worth stating that the same risk factor priority. We start with the causal relationship model by following
may lead to more than one risk events and the same risk event risk factors and associated risk events as a consequence. This al-
can obstruct more than one goals. Conversely, a goal is obstructed lows us to focus on the relevant risk events for the risk level esti-
by multiple obstacles that relate risk events and associate fac- mation, rather than considering all raw risk factors. Risk
tors.This representation allows us to model situations where an estimation considers risk event likelihood and its severity of im-
event is influenced by more than one risk factor and impacts on pact on goal negation. The impact assessment builds the cause-
single or multiple goals. An obstruction link is established from consequence relationship from the risk event to the obstructed
the risk event to the specific goal that it obstructs. This approach goals. The risk assessment finally prioritises risks, so that highly
supports the construction of a goal-risk model by refining risk fac- prioritised ones get immediate attention. The final activity of
tors to risk events and their combined obstruction to goals. GSRM aims to control risks as early as possible and to monitor
the effectiveness of the control action throughout the project. Risk
3.2.4. Treatment layer treatment needs to be planned and matched with the risk manage-
The treatment layer focuses on the control actions to counter ment scope. There may have been multiple control actions, but
the risks so that goals can be attained. It also monitors the effec- depending on the project context, only potential ones should be se-
tiveness of the implemented control actions and identifies any lected. Once the selected control actions are implemented this
new risks throughout the development. The main aim is to gain activity monitors risks throughout the project life cycle.
control over the risks as early as possible, preferably during the
requirements engineering phase. Generally, there exists an alterna-
4. Integration of GSRM into requirements engineering
tive countermeasure to the obstacles but one should select the
most cost effective one for the risk mitigation. This layer includes
As stated previously, the proposed work considers risk manage-
two different links: contribution link from the control action to
ment activities from the early stage of the development, in partic-
the goal that it fulfills, obstruction link from the control action to
ular from the requirements engineering phase. However,
the specific obstacle that it obstructs. The treatment layer allows
requirements engineering and software risk management are two
modelling, reasoning, and tracing the adopted control action for
different processes. Therefore, performing an integration requires
the risk mitigation and goal satisfaction. It also includes a respon-
to consider the interactions among the underlying activities, tasks,
sibility link from the control action to the agent so that the specific
methods, roles and dependencies amongst the artefacts that are in-
active agent would be responsible for implementing the control ac-
volved in these two processes [6]. First, we examine two different
tion in order to mitigate the risks.
perspectives, i.e. artefact and process oriented views, which allow
Fig. 3 shows the modelling framework of GSRM. GSRM uses the
to understand the rationale in terms of the integration.
same notations for goals (parallelogram) and obstacles (reverse
parallelogram) as used in the KAOS model. Goals are refined
through AND and OR refinement into sub-goals. Obstacles are 4.1. Artefact oriented view
linked to the goals through obstruction links.The treatment layer
includes the agent who has the responsibility to implement se- Artefact oriented requirements engineering is a systematic
lected actions to control risks. methodology which describes the problem space of the system-
as-is as comprehensively as possible towards complete require-
3.3. GSRM activities ments specification documents [38]. It combines both structure
and content of the artefacts, expressed by the domain-specific con-
The activities describe all the tasks and steps that are required cept model. The structure is described with a taxonomy reflecting
for goal-driven risk management and creation of the risk specifica- the hierarchical structure of produced specification documents.
tion artefact.Concepts of the goal-driven risk management model, The included content (the underlying concept model) reflects the
such as goal, obstacle, and treatment are used within the activities. concepts of an application domain in terms of precisely defining
The first activity initialises goal-driven risk management during the used elements and their relations for specific description

Please cite this article in press as: S. Islam et al., An empirical study on the implementation and evaluation of a goal-driven software development risk
management model, Inform. Softw. Technol. (2013), http://dx.doi.org/10.1016/j.infsof.2013.06.003
6 S. Islam et al. / Information and Software Technology xxx (2013) xxx–xxx

Fig. 3. Framework of GSRM.

techniques [39]. Business and requirements specification are the Requirements engineering is mainly comprised of eliciting, analys-
two main artefact types of requirements engineering. These arte- ing, validating, and managing activities, which further contain fine-
fact types formulate goals, capabilities, constraints, system vision grained tasks and sub-tasks within these activities. The activities of
and requirements [40]. As stated previously, GSRM provides arte- both requirements engineering and risk management are sequen-
fact type risk specification that encompasses content items such tial and techniques used within the activities are partially similar.
as goals, obstacles, and treatment, which in turn consist of concept For instance, requirement elicitation commonly relies on the back-
types such as risk management plan, goals details, risks details and ground study of specific types of artefacts, including pre-existing
risk status report. Goals are one of the fundamental concepts to documents about the system-as-is, such as organisational charts,
support the integration, in particular they provide background policies, work procedure, business rules, data samples and scenario
foundation to elicit and analyse requirements and risks. Require- analysis of the interaction among the systems [8]. It also focuses on
ments are amongst one of the elementary inputs for risk identifica- the stakeholder-driven elicitation through structured and unstruc-
tion.The quality of the elicited requirements is a highly influential tured interviews and joint workshops. Goals and risks identification
factor in attaining project goals related to schedule, budget, prod- of the GSRM focus on the preliminary analysis of the system-as-is,
uct quality, and error free requirements. Reducing project risks is such as project information, project domain analysis and require-
one of the critical requirements of a software project. Risk concepts ment artefacts. The taxonomy-based questionnaires and brain-
contribute to reduce errors from the elicited requirement. Attri- storming sessions with stakeholders are very effective techniques
butes such as risk level, priority, control action, and risk status help for the risk identification [18,41]. Therefore the techniques used
to reduce requirement errors. Attributes like requirements accep- and input artefacts required for goals, requirements, and risk iden-
tance, time constraint and design decision are also supported by tification are similar. The activities and tasks under the process cou-
the risk status report. Fig. 4 shows the interaction between the ple to one or more roles which take the responsibilities for
requirement and the risk artefact concept. producing the related artefacts. Typical roles for requirements engi-
neering activities are customer/user representative, business ana-
4.2. Process oriented view lyst and requirement engineer. The requirement engineer is the
key responsible person who creates and manages the requirement
The activities, tasks and methods of the requirements engineer- specification by aligning the business needs to the software-to-be.
ing and risk management process are coherent and interrelated. He establishes the bridge among business analyst, architect, project

Please cite this article in press as: S. Islam et al., An empirical study on the implementation and evaluation of a goal-driven software development risk
management model, Inform. Softw. Technol. (2013), http://dx.doi.org/10.1016/j.infsof.2013.06.003
 S. Islam et al. / Information and Software Technology xxx (2013) xxx–xxx 7

Fig. 4. Attributes dependency of requirements and risk artefacts.

manager, and customer/user. On the other hand, a risk manager is 5. Evaluation and data collection
mainly responsible for performing risk assessment and manage-
ment activities. He should have adequate knowledge of the project 5.1. Study design
domain and sufficient skills to handle the risks in a specific project
situation. Generally in small and medium projects, the project man- We have chosen to employ an empirical research method to
ager is concerned with the overall project execution and takes the evaluate GSRM. In particular, we follow the case study method
additional role of risk manager. A successful project manager is along with action research as empirical inquiry based on a real life
always a good risk manager [42]. There is no strict rule, more spe- active on-going software development project [43]. Action re-
cifically, it is hard to define any fixed point when risk management search makes an effort to provide practical value of real subject
activities should initiate within requirements engineering. How- problems while simultaneously contributing to the acquisition of
ever, we argue to start the goal and risk identification activities in new theoretical knowledge [44]. However, performing an empiri-
parallel with the requirement elicitation activity. Goals and risks re- cal study within the software risk management domain is chal-
lated to the business needs and projects such as schedule and bud- lenging, because software projects are generally of long durations
get, staffing, participant competency, customer/user involvement, with focus on time, budgetary, and quality control; therefore a
development facilities, and complexity can be effectively identified comprehensive risk management practice is not always possible.
and analysed at this stage, even before analysing the system We carefully designed the study by mixing case study and action
requirements. Such early consideration also allows to capture research. The choice of research method selection depends on
non-technical project risks up-front so that appropriate counter- many factors such as availability of resources and alignment of
measures can be undertaken to produce robust and error-free method with the questions. We have confirmed these factors be-
requirements. Risk management plans should also align with the fore performing the study. Our study design is based on a single
project scope as well as requirements elicitation. In particular, an real software project and questions are constructed focusing on
appropriate schedule is necessary for the risk management activi- this single case. The risk management and project team members
ties within the requirements engineering process. GSRM follows a were learning by performing GSRM activities to identify and ana-
pattern of iterative activities and can be tailored to support the lyse the risks and implement the chosen actions to control the
requirements elicitation, analysis and verification activities. risks. Therefore, practitioners were directly involved in under-

Please cite this article in press as: S. Islam et al., An empirical study on the implementation and evaluation of a goal-driven software development risk
management model, Inform. Softw. Technol. (2013), http://dx.doi.org/10.1016/j.infsof.2013.06.003
8 S. Islam et al. / Information and Software Technology xxx (2013) xxx–xxx

standing and performing risk management activities. Results from kick-off, brainstorming and interview sessions. Closed questions
the activities were used to improve the project situation from po- were used during the interview sessions, considering the studied
tential risks. We followed typical study design components such as project context, and open questions were used to evaluate the
study construct, data collection techniques and analysis, observa- applicability of GSRM in a subjective way. The project team mem-
tion, action and conclusion [45,43,46]. Fig. 5 depicts the combined bers, users and sponsor representatives mainly participated in the
case study and action research design components. Data collection interview and brainstorming sessions. The kick-off session was
and analysis were based on the case context and GSRM concepts, used for the members of the risk management and project team.
and they followed a sequential explanatory strategy to support Closed questions focused on the exploratory basis to identify exis-
both the case study and action research. Concepts of GSRM such tence and frequency of problems and their causal dependencies to
as goal and obstacle, along with activities and artefacts, provided the risk events and consequences. The open questions were mainly
impact for the observation and interpretation of study results. In descriptive and comparative questions used to identify the partici-
particular, action research focused on mitigating the risks of the pants’ perception about the GSRM and its integration into require-
studied project context. The execution of GSRM activities and arte- ments engineering. Open questions also answered specific
facts, as a part of action research, directly contributed to control practitioners’ beliefs about risk management and its importance
the identified risks. Finally, the study concluded with observation, during the development process. In the beginning, a kick-off work-
comparison, and feedback for the overall observation refinement of shop was carried out amongst the members of the development
GSRM. The conclusion considered both case study and action re- team to provide an overview of GSRM, risk artefacts, and its integra-
search. Case study is a widely used empirical research method in tion into requirements engineering. The open and closed question
software engineering domain [47,45]; action research was in our responses and project artefacts were inputs to the brainstorming
case effectively used to control the risks of the project. When per- session. Three Master of Information Technology (MIT) students
forming action research, it is necessary to solve a real problem; the of the Institute of Information Technology (IIT), University of Dhaka,
next section shows that the project we have selected is a high-risk Bangladesh (former students of the first author) were employed for
project. Therefore, action research adequately supported the study the implementation of GSRM. The students had adequate knowl-
project to solve a real problem by identifying and controlling the edge and experience working in software projects.
risks. The combination of these two practical research methods The units of analysis for the study were mainly considered
provided us with in-depth understanding about GSRM and its im- GSRM concepts, automation project, and project team/users. The
pact into the studied context, allowing practitioners to play an ac- collected data was analysed both qualitatively and quantitatively
tive role for empirical investigation, and controlling the risks for based on the units of analysis. We followed a sequential explana-
the successfully project completion. tory strategy to collect and analyse the data [48,49]. In particular,
data collection and analysis were initiated from a short training
5.2. Study construct session about GSRM, then sequentially followed GSRM activities
and output of one activity subsequently used by the next activity.
The main focus of this study was to specify the impact of the For instance, a risk management plan was used by the goal identi-
goal-driven risk management model on the software development fication and modelling activities. Once the GSRM activities were
project. The study goals aim to: performed, the focus was to obtain information about the useful-
ness about GSRM and conclude the study. Therefore, feedback
1. evaluate the advantages and limitations of goal-driven risk about the method and the conclusion considered quantitative data
management in software development projects; from previous activities. The quantitative analysis considered vari-
2. improve our understanding of the issues involved in the inte- ables such as effort performing the risk management activities, the
gration of risk management activities into requirements number of risks factors and risk events, erroneous requirements
engineering. and the number of control actions and their effectiveness to sup-
port the units of analysis. Thus, the responses and observations
5.3. Data collection and analysis from the closed question sessions were used as input to analyse
the data. The qualitative analysis considered issues such as using
The data was collected from various sources using different data goal-driven approach for risk management, risk management
collection techniques. In particular, project documents were one of impact on the project, integration of risk management into
the main inputs and the data collection techniques were mainly requirements engineering, and adequacy of activities and artefacts

Fig. 5. Study design details.

Please cite this article in press as: S. Islam et al., An empirical study on the implementation and evaluation of a goal-driven software development risk
management model, Inform. Softw. Technol. (2013), http://dx.doi.org/10.1016/j.infsof.2013.06.003
 S. Islam et al. / Information and Software Technology xxx (2013) xxx–xxx 9

of GSRM, and comparing the risk factors with the other study re-  High reputation regarding successfully completing of the
sults. Project participants’ positive and negative observations, risk project.
management results, performance of the activities, and process
integration were mainly used to evaluate the benefits and weak- Despite the project being considered a high-risk project, the
nesses of GSRM. Furthermore, as mentioned before, quantitative management decided to undertake it in order to accomplish the vi-
metrics, such as effort spent on GSRM in requirements engineering, sion. The management vision is to successfully develop the office
number of risks and treatment actions over time were also consid- application and customise it for other government ministries and
ered to support the study goals. agencies. High reputation and user satisfaction may give Domain
Tech a competitive advantage on the market, given that the local
6. Case study government recently decided to digitialise its ministry offices un-
der the Digital Bangladesh Vision 2021 project [51]. Therefore, the
6.1. Study context project was important for the Domain Tech to secure the market
regarding future business opportunities.
6.1.1. Company profile and project context
The study reports on our empirical work of a software develop- 6.2. Introduction of GSRM process
ment project at Domain Software Technologies Ltd. (Domain Tech),
a subsidiary company of Domain Technologies Ltd [50]. Domain 6.2.1. Activity 1: Initialise goal-driven risk management
Tech started its operation in 2002 in Bangladesh with multi- A kick-off workshop as a tutorial for the risk management team
dimension lines of business, such as software development, global was initially carried out to provide an overview of GSRM. The final
IT support, and consultancy.The project considered in the case part of the workshop was used for the GSRM initialisation activity.
study aimed to automate the Planning Commission Campus of The Project Manager (PM) elaborated the factors, which made the
the Ministry of Planning, as part of the e-Governance project of project considered a risky one. The first task under this activity (i.e.,
the government of People’s Republic of Bangladesh. The project determine the riskiness nature of the project) was considered and
was officially initiated in November 2009, once Domain Tech ob- the main factors for the high risk project were noted. Project scope,
tained the work order from the ministry. The project context was success criteria, business goals, and initial high-level system spec-
the development of an application software which contained ten ifications were used to define the risk management context. The
separate modules to automate day-to-day ministry operational risk management scope was to control risks related to the applica-
activities. The modules were: Project Planning, Personnel Manage- tion development, specifically project execution, user, product
ment, Payroll Management, Budgeting, Auditing and Accounting, specification, quality, and user training, and was biased by the pro-
File Tracking, Letter Dispatching, Meeting minutes, Library Man- ject inherent challenges and scope. However, risks related to pro-
agement, Inventory and Vehicle Management. Every module has ject sponsor, fund support, and user-internal organizational
its own features, constraints and requirements, but it also relates problems were not considered within this scope. The Risk Manage-
to the other modules. The main project goals were: ment (RM) team consisted of a PM as the team leader along with 3
students and 2 development team members. The risk management
 Automate the whole planning commission campus. was scheduled at the first deliverable phase, i.e. requirements
 Digitise old and new data and dynamic report generation. specification and design phase.The interview participants, from
 Interconnect with existing software on the commission campus. both users and sponsor representatives, were identified and sched-
 Train employees for the new system. uled. However, no schedule was considered for risk monitoring and
the PM categorised it as a demand-based activity.
6.1.2. Project riskiness
This was the first government project by Domain Tech in which 6.2.2. Activity 2: Identify and model goals
the main users are government employees. Initially a high-level This activity was carried out in a brainstorming session by the
technical specification about the different project modules was members of the RM team. The risk management plan and project
developed by a consultant company on behalf of the ministry. documents were considered as input for this activity. Table 1 out-
Based on the specification, Domain Tech, similar to other vendors, lines the four goals and sub-goals which were agreed upon by the
submitted an expression of interest and successfully obtained the RM team. These were: complete project within estimated budget and
work order. Apart from the office automation application software schedule, complete user’s training, obtain positive reputation and gen-
to be developed, the project context also included a comprehensive eralise the application for other government ministries. The first two
user training. The development team consisted of 15 members, the goals were considered through focusing on the project contract
duration was 13 months. The management was convinced of the and the tasks needed to be achieved before the system goes oper-
need to integrate a comprehensive risk management practice in ative. The remaining two goals were critical for the future business
the project, due to the project’s inherent risk nature. From Domain vision of Domain Tech. It needed positive user feedback, overall
Tech’s perspective this study benefits them by understanding and product quality, and successful deployment and maintenance of
control the risks of the project. Furthermore, this study provides the software. High vendor reputation could give a competitive
a comprehensive risk management practice from the early stage advantage to obtain further work orders for similar government
of the project and awareness about risk management to the Do- projects. Generalising the application by gathering knowledge
main Tech employees. The Domain Tech management and main from this project could significantly reduce the development cost
project team members considered the project as high-risk even from the vendor’s perspective. These goals were refined during
though the practitioners have experience from similar projects. the session.
The reasons were:
6.2.3. Activity 3: Identify and model obstacles
 Lack of experience on how to handle government employees. Initially, interviews were carried out by student members with
 High-level initial specification. the selected practitioners of the development team. The interviews
 Large number of users to train who are government officials. continued further with the selected users and sponsor members.
 Effective usage of the product and user satisfaction. However, it took more time than expected with the user members

Please cite this article in press as: S. Islam et al., An empirical study on the implementation and evaluation of a goal-driven software development risk
management model, Inform. Softw. Technol. (2013), http://dx.doi.org/10.1016/j.infsof.2013.06.003
10 S. Islam et al. / Information and Software Technology xxx (2013) xxx–xxx

Table 1 who needed training for the new system environment. But the
Identified goals and sub-goals. planning ministry and UNDP refused to increase the training bud-
Goals Sub-goals get as total project budget was already approved and there was no
Complete project in Maintain estimated budget in development room for revision because several donor agencies supported the
estimated project fund. Domain Tech also had inadequate experience to han-
budget and schedule Maintain estimated schedule in development dle the government officials. Existing data was documented with
Maintain realistic estimation dissimilar structure, which made it difficult to convert the manual
Clear milestones
Competence practitioner
data into new electronic formats. Risk factors were summarised
Reduce errors from requirements from the raw list. These factors were mainly due to unavailable,
User active participation wrong and incomplete information, poor participation, errors from
User positive motivation the existing system, and data conversion difficulties, which re-
Complete user’s training Adequate training budget sulted in incomplete specifications and inaccurate estimations. Ta-
Professional and competence training ble 2 shows the risk factors.
Complete training and user manual
User active participation
User positive motivation 6.2.4. Activity 4: Assess risks
Improve effective communication & The RM team considered interview responses of the selected
coordination risk factors as initial input for the risk assessment. Moreover,
Improve practitioner motivation & PM’s and other members’ experience and the pre-assumptions pro-
productivity
vided by GSRM about the causal link of the risk factors to the risk
Reduce user and practitioner conflict
events and consequence to the goals were also considered for the
Obtain positive reputation, User satisfaction
generalise the application Complete project in estimated budget
estimation of the risk event likelihood and risk impact. Table 2 in-
and schedule cludes risk events caused by the identified risk factors. These
Quality product events directly obstruct the goals, for example budget overruns,
Successful training erroneous requirements, poor training, and system use obstruct
Details understanding of business process
the goals like complete project in estimated budget and schedule
Successful system usage
Complete elimination of existing manual or vendor high reputation. To construct a causal relationship mod-
system el, risk factor, event, and priority were considered as target, obser-
Complete specification vable and decision nodes of a Bayesian Belief Network. The RM
team agreed that budget overrun is the highest prioritised risk fac-
tor. Risk factors such as inaccurate estimation, high training cost,
as they did not follow the agreed schedule and demonstrated lack erroneous requirements, and schedule overruns casually link to
of IT and project domain knowledge. The RM team also needed to budget overruns as shown in Fig. 6.These factors are the most
provide an overview of goals, risks, and the main purpose of the influential for the project context and some of them were beyond
interview before the actual interviews took place. A raw list of risk the control of the project manager and the project environment.
factors was generated from the interview responses. The raw list The identified factors resulted in complete obstruction of the goal
was very large and was refined by following the identified goals to maintain estimated budget in the development. Erroneous
and the brainstorming session. Most of the identified risks were re- requirements and schedule overruns were considered as risk fac-
lated to the project execution, product, and human perspectives. tors but these factors were again treated as risk events in different
The development team only had a high-level initial specification. contexts. Finally, risk events were prioritised into the three scales,
As such, it was difficult to fully understand the necessary require- i.e. ‘‘very important’’, ‘‘important ‘‘, and ‘‘less important’’, so that
ments. Domain Tech Management agreed to accept any user severe ones get immediate attention. However, for confidentiality
change at any stage and believed it would increase the company reasons we cannot provide the detailed results.
reputation, which turned out to be a mistake. Users could not pro-
vide detailed information and lately numerous gaps were identi- 6.2.5. Activity 5: Treat and monitor risks
fied that needed to be considered. Users added several new The RM team initially planned to control the high and medium
things compared to the initial specification based on which sche- prioritised risks by completely eliminating or reducing the likeli-
dule and effort estimation were calculated. Numerous changes hood of risk event occurrences and severity of the impacts. The ses-
were requested by the users, such as change of staff profile and lay- sion identified the possible countermeasurs and potential ones
out as well as an addition of more functionalities under a specific were selected so that the goals could be attained. Erroneous
module related to personal and payroll management, budgeting, requirements were among the main events which negatively affect
auditing, and accounting. The collected information was some- the possible outcome of goals such as: complete project in esti-
times incomplete and ambiguous due to different interpretations mated budget and schedule, reduce erroneous requirements, im-
of the same context. It was difficult to obtain appointments from prove completeness in requirement specification, and generalise
high-level officials and several meetings were needed to collect rel- the application. Users’ factors were among the main reasons for
evant information. But their remarks were important for the pro- the risk events and a large number of users increased the overall
ject acceptance. Some of the users did not have adequate total training cost and,as a result, the overall project budget. But
knowledge of the software application, but played important roles these factors were beyond the PM’s controls and difficult to elim-
to support the key business process. There were ambiguities in inate completely. The RM team identified possible countermea-
describing the key operational process and roles involved within sures, i.e. train selected representatives from user groups, extensive
the process. Once an interview was completed, no user agreed to user involvement, signed frozen requirements, complete understand-
sign the interview documents. At the end, the requirements analy- ing of users business processes and dependencies among the modules,
sis consumed much more time than estimated. obtain and incorporate key users feedback, and include a J2EE expert
Another problem appeared concerning the number of users into the project. The identified countermeasures were potential and
who participated in the training. Initially Domain Tech agreed to would not incur additional cost except of the integration of a J2EE
train 500 users. However, during the requirements identification expert. However, it was difficult to involve the key users even
and on site observation, there were about 800 users identified though they were available on the commission campus. Users’ lack

Please cite this article in press as: S. Islam et al., An empirical study on the implementation and evaluation of a goal-driven software development risk
management model, Inform. Softw. Technol. (2013), http://dx.doi.org/10.1016/j.infsof.2013.06.003
 S. Islam et al. / Information and Software Technology xxx (2013) xxx–xxx 11

Table 2 for the project as well as for the future customisation. Domain Tech
Identified risk factors and events. planned to arrange workshop sessions with the high government
Risk factor Event officials to inform them about the status of the project and the nec-
Numerous change request Budget overruns essary further action. These sessions were effective to develop and
Users passive involvement Erroneous requirements implement the application into the planning commission campus.
Users lack of project domain knowledge Schedule overruns
User lack of IT competence Ineffective training
Inadequate training budget Unclear system vision 6.2.6. Goal-risk model
Large number of users requiring training Ineffective communication Fig. 7 illustrates the goal-risk model for the complete project in
High training cost User dissatisfaction
Incomplete & incorrect initial specification Poor system use
the estimated budget. The goal was refined into several sub-goals,
Over-promise & inaccurate estimation Poor reputation such as: maintain estimated budget and schedule, realistic estima-
Error in project contract Incomplete information tion, clear milestones, user active participation and motivation,
Lack of experience in handling government Unable to generalise and reduce errors from requirements. Risk factors, such as numer-
officials product
ous change requests, user passive involvement and lack of knowl-
Bureaucratic nature of organisation Poor feedback
Users’ lack of motivation Deployment problems edge, high training costs, inaccurate estimation, errors in contract
Data conversion difficulties High variation and incomplete specification obstructed the sub-goals and led to
Unwilling to sign interview document risk events such as erroneous requirements, schedule overruns
Complex interactions among the modules and budget overruns. The bottom part of Fig. 7 shows the treat-
Political biasness
ment actions and the associated agents responsible for implement-
ing the actions, e.g. comprehensive and competence training.
Fig. 8 shows the model for obtaining positive reputation. The
of project domain knowledge and lack of IT competency could not goal was rather subjective and refined into user satisfaction, qual-
be addressed by any means during the duration of the project. ity product, successful training, and system usage. Risk factors,
However, selected users’ training would reduce the overall training such as poor training, ineffective communication, incomplete and
cost significantly. The RM team emphasised the professional and complex product, and retaining the existing system obstructed
competence training and preparation of a comprehensive training the goal and led to user dissatisfaction and low reputation as the
manual before physically deploying the system. Still, there was no main risk events. User motivation for the new system, effective
effective way to address the numerous change requests, but the cooperation between users and practitioners, complete and com-
PM decided that once the feedback was integrated the frozen petence training are control strategies, which can mitigate the risk
requirements of a specific module would be signed by the key user. factors. The elimination of old manual systems was considered as a
At that stage, the project experienced a huge number of user requirement within the control action. Project contracts did not
change requests and several new things were added compared to cover the maintenance part which seemed to be an important issue
the initial specification. The PM agreed that it was not possible for this project. The PM decided to convince the project sponsor to
to maintain the estimated budget and schedule, and management allocate the maintenance budget for the project. Domain Tech
decided to accept 15% budget overrun. The management decided management also decided to arrange 2 or 3 common workshop
to obtain positive reputation from the users by any mean and a sessions apart from training to motivate the users for the new
complete understanding of the business processes was essential system.

Fig. 6. BBN network for budget overruns.

Please cite this article in press as: S. Islam et al., An empirical study on the implementation and evaluation of a goal-driven software development risk
management model, Inform. Softw. Technol. (2013), http://dx.doi.org/10.1016/j.infsof.2013.06.003
12 S. Islam et al. / Information and Software Technology xxx (2013) xxx–xxx

Fig. 7. Goal-risk model for project completion.

The difficult part regarding risk management was to convince the collection techniques were used to collect and analyse qualitative
key government officials regarding the implementation of the and quantitative data for a more comprehensive understanding
selected control actions. The PM communicated the agreed control about the context [49]. Two kick-off workshops and five brain-
actions to the management, and the management scheduled a meet- storming sessions, each approximately 4 and 6 h respectively,
ing with the planning commission secretary. The visual representa- and interview sessions (2 h for the closed questions by the practi-
tion of the goal-risk model effectively helped to communicate the tioner and 3 h and 30 min by the users and 1 h and 30 min for open
risk information to the management. In the meeting, the secretary questions by the practitioner) were used to evaluate the activities
of the planning ministry agreed on the recommended actions and of the GSRM process. There were in total 23 responses gathered
assigned 20 key users to be extensively involved in helping the from 200 closed questions including 10 practitioners, 12 users
development team. The secretary also circulated an office order at and 1 sponsor. The closed questions were on an exploratory basis
joint, deputy and assistant secretary level to collaborate with the that supported identification of the risk factors from the project
project by giving necessary information. Furthermore, only 58 users context [46]. 7 practitioners, including 2 members of the risk man-
were selected as champions for the training session conducted by agement team participated in 30 open questions providing feed-
Domain Tech. The PM decided to arrange a risk monitor meeting, ini- back about GSRM and its integration into requirements
tially twice a month and less frequently at the later stages of the engineering. The open questions response was mostly on positivis-
development. At the end, student members carried out the inter- tic basis from the practitioners’ observation about GSRM for man-
view sessions with the open questions to obtain feedback about aging risks within the project. The student members along with the
GSRM. The interview participants were the PM and other members other members of the risk management team analysed, as input
of the RM team, three practitioners of the development team, one data for the risk management process, the project documents
management representative of Domain Tech and two users. and in particular artefacts, such as the initial specification, the nat-
ure of change requests, the detailed requirements specification, the
project scope, and the success criteria. They also compiled the
7. Discussion interview responses to identify risks and to obtain feedback about
GSRM. Among the top 7 risk events and 15 influential risk factors
7.1. Study results and applicability of GSRM such as high training cost, overall budget overruns, erroneous
requirements, users participation, and detailed system specifica-
The data was collected and analysed by following sequential tion, only 3 risk events and 11 risk factors were fully or partially
explanatory strategy, from GSRM activities to the feedback about controlled through 8 potential countermeasures. Risk factors orig-
GSRM and its usefulness [48]. As stated previously, different data inating from the users’ perspectives were beyond the project man-

Please cite this article in press as: S. Islam et al., An empirical study on the implementation and evaluation of a goal-driven software development risk
management model, Inform. Softw. Technol. (2013), http://dx.doi.org/10.1016/j.infsof.2013.06.003
 S. Islam et al. / Information and Software Technology xxx (2013) xxx–xxx 13

Fig. 8. Goal-risk model for positive reputation.

ager’s control. The identified countermeasures were supported by goals make it easy to communicate with the users, project spon-
the management and directly implemented to improve the project sors, and development team members. The main users of the pro-
situation for managing risks. The action research method effec- ject were government officials who did not have adequate
tively used on this occasion. The analysis took approximately four knowledge of the purpose of automation due to lack of IT expertise.
complete working days. Goals and risks identification and model- However, they were able to demonstrate their expectations regard-
ling activities consumed the highest effort. Project complexity ing the project. From the user’s perspective, several goals were
and a large number of goals certainly increased the overall risk identified such as user satisfaction, successful training, system
management effort. A total of approximately 26% effort of initial usage and project team perspective; goals were emphasised more
deliverable phase including the student members effort was used on issues related to project completion, training, user participation
for risk management. A short tutorial was adequate to provide and reputation. However, some participants observed that goal
the basics about GSRM to the practitioner. Therefore, applicability modelling was difficult to perform and required more experience.
of GSRM in terms of the effort to perform the activities and integra-
tion into early requirements engineering is reasonable and feasible 7.2.2. Integration of GSRM into requirements engineering
for any software project. Empirical study has well proven to be an Based on the open question responses, the PM and practitioners
effective research method to collect and analyse relevant data for appreciated the integration of risk management into requirements
investigating a specific issue of the software engineering domain engineering. It provided them with early warnings about the prob-
[52]. We believe the studied project and obtained results based lems that existed in the project. There were dependencies among
on combination of research methods provide comprehensive the requirements and risks specification artefacts. In particular,
understanding about the issues relating to the goal-driven risk goals were one of the fundamentals to support the integration.
management approach into software development projects. For instance, in the studied project, positive reputation was consid-
ered one of the highly prioritised goals. Several risk factors such as
7.2. Usefulness of the GSRM poor training and ineffective communication were directly
obstructing the goals. Elicited requirements were considered to
7.2.1. Goal-driven approach for risk management identify risk factors. Risk artefacts, such as risk level, priority and
A goal-driven approach for risk management is reasonably ben- list of erroneous requirements directly contributed to reduce er-
eficial during the early development. A project contains goals, and rors from the requirements. In the project, a brainstorming session

Please cite this article in press as: S. Islam et al., An empirical study on the implementation and evaluation of a goal-driven software development risk
management model, Inform. Softw. Technol. (2013), http://dx.doi.org/10.1016/j.infsof.2013.06.003
14 S. Islam et al. / Information and Software Technology xxx (2013) xxx–xxx

was used to identify the goals, requirements and risk factors. The outcomes. Such comparison allows us to generalise our findings
Project Manager and other members of the risk management team and to identify contextual factors from the current study context.
played main roles in the risk management and requirements anal-
ysis. Furthermore, active user participation observed influential
factors for both requirements engineering and risk management. 7.3.1. Commonality in risk factors
Therefore, roles like project manager, requirements engineering, There is a substantial commonality fully or partially amongst
risk management, and user participation can effectively support our results and those discussed in existing literature. For instance,
activities within requirements engineering and risk management. Schmidt et al. [23] identified a comprehensive list of risk factors
and top factors such as lack of adequate user involvement and
7.2.3. GSRM process cooperation, lack of frozen requirements, change scope, and un-
Activity definition: Based on the open questions response, the clear/misunderstood scope/objective are similar to our findings.
activities under the GSRM process were identified as fully opera- The most noticeable areas, which included distinctive risk factors
tional and adequate, as the approach provided adequate tech- by our study, are user, user’s organisational and requirements con-
niques to identify and analyse goals, risks and treatment actions. text which match important risk components such as schedule and
The initial activity explicitly allocated schedule and resource for budget, requirements management, and personnel management
risk management within the project. The composition of the risk demonstrated by Ropponen et al. [29]. Our results also show sim-
management team from different roles was beneficial as different ilarities with Wallance’s et al. [53,24] findings on requirements,
views of the goals and risks were identified and combined. Under- user, and complexity risk dimensions. Procaccino’s et al. [37] study
standing the riskiness nature of the project at early stages was results emphasised the factors related to the customer/user and
appreciated by the project manager and other members of the requirements, such as user involvement, realistic expectations,
development team, as it allowed them to understand the necessity complete and accurate requirements, and well defined project
of the risk management activities in the project. The GSRM process scope which highly influence project success. Linberg [54] also
provides three techniques (i.e. structured interviews with closed summarised factors such as effective leaders, technologically real-
questions, brainstorming sessions, and analysis of project docu- istic requirements, and realistic schedule and effort estimation rel-
ments) to identify and analyse goals and risks. The information evant to the project success. Keil [55] identified a changing project
gained from the project brainstorming session was verified during scope and the lack of frozen requirements as critical risk factors of
the interview session. The combination of these techniques was IS projects. Our results fully or partially match with these findings.
treated as being systematic, reasonably applicable and in particu-
lar, reducing the bias for the risk identification.
Artefacts: GSRM provides both textual and graphical representa- 7.3.2. Risk management barriers
tion of artefacts produced by its activities. In particular, visual pre- In the case study, the PM realised the need for risk management
sentation is provided through goal-risk models and causal and he was the main authority for the task. However, practitioners
relationship models which made it easy to communicate the risk were not motivated to implement a complete risk management
information with the project team and management, as observed process. A similar situation is observed by other study results. This
during the studied project. The central textual artefact of GSRM situation is treated as the main barrier that obstructs the imple-
is a risk status report which is the final output of the activities. mentation of a formal risk management practice. Our interview re-
Thus the risk status report provides complete information about sponses summarised a large number of risk factors from the
the goals, risk factors, risk events and treatment actions and status project. A recent survey study result shows that intangible benefit,
of the risks. lack of resource, and too many risks to control are the main per-
ceived barriers by the experienced project manager to a successful
7.2.4. Limitations of GSRM implementation of software risk management [9]. Nyfjord et al.
The studied project also allowed us to identify the limitations of [10] and Kwak et al. [31] emphasise the organisational problems,
GSRM based on the feedback obtained about GSRM from the open such as variation of risk perception by different roles, lack of com-
questions response. First, when the project focuses on a large num- petence and process problems, lack of plan and coordination are
ber of goals, the efforts for developing and maintaining the arte- additional barriers besides resource problems to integrate risk
facts are considered too high. In particular, GSRM needs more management into development. We observed that formal risk
effort in analysing the goals and constructing goal-risk and causal management practice was missing in the Domain Tech case study.
relationship models. Secondly, constructing a casual relationship However, implementation of GSRM and its results certainly advo-
model through BBN is a complex undertaking, especially if several cate for a formal risk management practice in the upcoming
risk factors are considered as intermediate nodes, which increase projects.
the size of the probability density table. Thirdly, from the studied
project, the risk monitoring activity was not properly performed
at later stages due to the pressure to handle user changes and later 7.3.3. Factors related to current study context
deliverables. Finally, based on the studied project, it was difficult to It is worth mentioning that some important factors derived
answer some of the closed questions, in particular, quantify the from case studies found in the literature were not applicable to
state of the factors under the development components into three the Domain Tech case study. These included no planning or inade-
different scales at an early development stage. We agreed with quate planning, lack of management support, problems related to
these limitations as they reflected the practitioners’ realistic justi- the development process, and development team. Moreover, sev-
fication about GSRM. eral risk factors, seem to have a partial or no match compared to
the other studies, such as user unwillingness to provide informa-
7.3. Comparison with other study results tion and sign the interview document, bureaucratic nature of the
organisation, political biasness, users’ lack of IT knowledge, numer-
We compare the result of our study with other study results ous change/update requests, errors in contracts, training for large
found in the literature, specifically those concerned with software number of users, and over-promise made by the vendor. These fac-
development risk factors and their influence on the project tors dominate our case and mostly originate from the user context.

Please cite this article in press as: S. Islam et al., An empirical study on the implementation and evaluation of a goal-driven software development risk
management model, Inform. Softw. Technol. (2013), http://dx.doi.org/10.1016/j.infsof.2013.06.003
 S. Islam et al. / Information and Software Technology xxx (2013) xxx–xxx 15

7.4. Conclusion clear roles in performing the study. Therefore, student members
were part of the related research and their knowledge outcomes
Our experience based on the case study concludes that GSRM is were considered by the project team to mitigate the risks. A pre-
reasonably suitable for any complex project. The studied project condition for the action research is the involvement of the partic-
was complex, GSRM was well integrated to identify and control ipants from both studied context and researchers. In our case,
the software development risks. The project manager agreed that this condition was satisfied. Data was collected not only from the
by using GSRM it was possible to identify and tackle the problems development team but also from the customer and project sponsor
in a structured way from the beginning of the development. We representatives. Additionally, project documents were also ana-
observed similarities of the identified risk factors comparing with lysed to understand the goals and risk factors. Therefore, several
other studied results as well as unique factors based on the project sources were used to collect data, and this limits the effects of
specific context. Therefore, GSRM supports identification of both the interpretation of one single data source. Interview responses
project specific and generic risks that need adequate attention. and results of the project documents inspection were further ana-
Analysis of the observations from the studied project helps us to lysed and discussed in the brainstorming sessions. The PM and
better integrate our goal-driven risk management method into other team members were not fully motivated for a formal risk
requirements engineering. A tutorial session about goal modelling management practice in the project. This attitude changed later
and risk management helps to understand the basic concepts of in the project by observing the outcome of GSRM activities. This
GSRM. Our observation is that when treatment of risk factors is be- demonstrates the importance of GSRM and the fact that it assisted
yond the control of the project manager and development environ- in quickly creating visual and critical insights. Maturation effects
ment, it is difficult to control the risk. The studied project failed to also intimidate the internal validity, in particular when the partic-
be completed within the estimated budget and schedule. The pro- ipants react differently the perception of risk during the course of
ject suffered approximately 25% budget overrun even though some development. However, in our work activities GSRM took place in
of the risk factors were mitigated. The risk management team the requirement engineering phase, and therefore that threat can-
agreed on 15% acceptable limits for budget overruns, which was not significantly affect our study.
also exceeded. The project users were government officials and
the project was funded by UNDP, therefore there was no room 8.2. External validity
for budget increase. The Domain Tech management considered
the project as a loss project. Nevertheless, Domain Tech obtained The case study context is located in a single geographical region,
the next work order from the planning ministry which implies that and as such there is possibility for cultural bias. Our findings are
the government officials were happy with the final outputs of the based on multiple data sources which allow for stronger conclu-
project. The project manager believed that without integration of sions. The study results were compared with other results from
risk management, budget overrun could have been even more, the literature in order to generalise our findings. The comparisons
government officials would not consider to significantly participate confirmed several commonalities in terms of goals and risk factors,
in the project and the requirement errors would not have been however, there are also some unique factors caused by local con-
controllable from an early stage. The project is planned to be de- text. Therefore, factors related to goals and risks are influenced
ployed in the ministry campus and selected users are currently un- by the local context.
der training. Domain Tech already obtained phase 2 of the project,
which includes three more modules in the existing application
8.3. Construct validity
software. Therefore, one goal is attained i.e. good reputation. The
upcoming project concerns the annual development program, pro-
We considered benefits and limitations of the GSRM and cor-
ject and foreign aid monitor systems of the planning ministry, and
rectly measured quantitative metrics, such as the effort required
maintenance. The Domain Tech management hopes that phase 2
to undertake risk management activities in a software project.
will compensate the loss suffered by the developed project. One
The quality of the case study questionnaires was improved by fol-
further conclusion is that it is not necessary to always consider
lowing the feedback from our previous study. The participants an-
budget and schedule factors at the highest priority. Software devel-
swered most of the questions apart from those which are perceived
opment projects are complex undertakings. There exist other is-
as difficult at an early stage. The threat of asking the wrong ques-
sues such as requirements, users, change management, user
tions was mitigated. The project manager and the development
satisfaction, and system usage, which directly or indirectly influ-
team members had adequate experience from several software
ence the budget and schedule constraints. These factors need early
development projects. User representatives also had adequate
attention in the development. Early determination of the nature of
experience in the government service. The student members con-
project riskiness is very effective to plan the risk management
ducted a kick-off workshop for the practitioners and explained de-
activities. Our study concluded that the project scope affects all
tails about the interview purpose to the users. We believe that the
dimensions of risk and for high-risk projects; risks associated to
participants understood the terms being used.
requirements specification, users, change management and project
execution are more obvious.
9. Summary

8. Study validity In this paper we presented the GSRM method and reported our
experience from its usefulness and integration of risk management
8.1. Internal validity activities into requirements engineering through an empirical
evaluation, by combining a case study method with action re-
We tried to reduce the expectation bias on the case study result. search. The study is based on a single active ongoing software pro-
The interview responses were commonly analysed in the brain- ject case, and project team members were actively involved within
storming session by the risk management team. None of the prin- the case study. The application of GSRM to a real case study has
ciple investigators of GSRM were directly involved in the case been very promising. Our results showed that a goal-driven ap-
study, in order to reduce the bias of the findings. However, the stu- proach is suitable for risk management and risk management is
dent members were adequately trained with GSRM and assigned well integrated into the early requirements engineering. The re-

Please cite this article in press as: S. Islam et al., An empirical study on the implementation and evaluation of a goal-driven software development risk
management model, Inform. Softw. Technol. (2013), http://dx.doi.org/10.1016/j.infsof.2013.06.003
16 S. Islam et al. / Information and Software Technology xxx (2013) xxx–xxx

sults indicate that GSRM is a practical and reasonable risk manage- Empirical Software Engineering and Measurement, IEEE Computer Society,
Washington, DC, USA, 2009, pp. 418–421. doi:http://dx.doi.org/10.1109/
ment method that can be employed in an industrial context. The
ESEM.2009.5316014.
goal oriented view made it easy to understand and communicate [10] J. Nyfjord, M. Kajko-Mattsson, Integrating risk management with software
the concepts of GSRM, and a short training session was adequate development: state of practice, in: Proceedings of the International
for this purpose. The results from the risk management activities Multiconference of Engineers and Computer Scientists 2008 (IMECS), 2008.
[11] J. Ropponen, Risk assessment and management practices in software
have directly incorporated into the project to control the potential development, in: L.P. Willcocks, S. Lester (Eds.), Beyond the IT Productivity
risks. We have noted our gained experience and insight lessons Paradox, John Wiley & Sons, Chichester, 1999, pp. 247–266.
learned from the case study. These can be useful for integrating [12] P.L. Bannerman, Risk and risk management in software projects: a
reassessment, The Journal of Systems and Software 81 (12) (2008) 2118–
risk management practice into software projects. We also observed 2133. http://dx.doi.org/10.1016/j.jss.2008.03.059.
limitation of GSRM mainly related to the risk management model- [13] Risk management-vocabulary-guidelines for use in standards, iso/iec guide 73,
ling artefacts and closed questions. 2002.
[14] ISO/IEC 15504 information technology – process assessment, International
To generalise our results we compared the study results with Organization for Standardization (ISO)/International Electrotechnical
other similar study results. We believe this study is useful to im- Commission (IEC), 2004.
prove GSRM and to inform and motivate practitioners about the [15] B.W. Boehm, Software risk management: principles and practices, IEEE
Software 8 (1) (1991) 32–41. http://dx.doi.org/10.1109/52.62930.
effectiveness of integrating risk management activities in the early [16] F. Sisti, S. Joseph, Software risk evaluation method version 1.0., Tech. rep.,
phase of the development. However, a single case study is not ade- SEI,Carnegie-Mellon University, 1994.
quate to generalise the findings. Therefore more case studies are [17] C.J. Alberts, A.J. Dorofee, R. Higuera, R.L. Murphy, J.A. Walker, R.C. Williams,
Continuous Risk Management Guidebook, Software Engineering Institute,
necessary so that the validity of the results can be further general-
Carnegie Mellon University, Pittsburgh, PA, 1996.
ised and the risk management method can be improved based on [18] M. Carr, S. Konda, I. Monarch, C. Ulrich, C. Walker, Taxonomy based risk
the case study observations and limitations. We are currently identification (cmu/sei-93-tr-6, ada266992), Tech. rep., Software Engineering
working on defining comprehensive application guidelines for Institute, Carnegie Mellon University, Pittsburgh, PA, 1993.
[19] M.K.M.B. Chrissis, S. Shrum, CMMI Guidlines for Process Integration and
the GSRM so that it can provide better support for risk analysis into Product Improvement, Addison-Wesley Longman Publishing Co., Inc., Boston,
a software project. Additionally, our effort will focus on developing MA, USA, 2003.
a goal-risk taxonomy. A goal-risk taxonomy identifies and classifies [20] ISO 31000:2009 Risk Management Principles and Guidelines, International
Organization for Standardization (ISO)/International Electrotechnical
goals and risks that are associated with certain project characteris- Commission (IEC), 2009.
tics and links with possible potential control actions in controlling [21] Standard australia, as/nzs 4360:1999 risk management, 1999.
specific types of risks. Project managers can reuse the taxonomy [22] H. Barki, S. Rivard, J. Talbot, Toward an assessment of software development
risk, Journal of Management Information Systems 10 (2) (1993) 203–225.
for any upcoming project. Some factors within the course of the [23] R. Schmidt, K. Lyytinen, M. Keil, P. Cule, Identifying software project risks: an
product life cycle such as deployment, operation, and maintenance international delphi study, Journal of Management Information Systems 17 (4)
do not get adequate attention at an early stage. Later on these fac- (2001) 5–36.
[24] L. Wallace, M. Keil, A. Rai, Understanding software project risk: a cluster
tors can pose a real threat for the project. Therefore these factors analysis, Information and Management 42 (1) (2004) 115–125.
need further investigation. We would like to integrate continuous [25] N.H. Arshad, A. Mohamed, Z.M. Nor, Risk factors in software development
risk management activities to analyse these factors. Finally, it is projects, in: SEPADS’07: Proceedings of the 6th WSEAS International
Conference on Software Engineering, Parallel and Distributed Systems,
important to develop automated mechanisms and tools to support
World Scientific and Engineering Academy and Society (WSEAS), Stevens
risk management activities. Our future work will be dedicated to- Point, Wisconsin, USA, 2007, pp. 51–56.
wards these aims. [26] R.T. Nakatsu, C. Iacovou, A comparative study of important risk factors
involved in offshore and domestic outsourcing of software development
projects: a two-panel delphi study, Information and Management 46 (1)
Acknowledgments (2009) 57–68. http://dx.doi.org/10.1016/j.im.2008.11.005.
[27] T. Moynihan, How experienced project managers assess risk, IEEE Software 14
(3) (1997) 35–41. http://dx.doi.org/10.1109/52.589229.
We would like to thank all the project team members of the [28] C.L. Iacovou, R. Nakatsu, A risk profile of offshore-outsourced development
automation project and the ministry of planning employees. A spe- projects, Communication of the ACM 51 (6) (2008) 89–94. http://doi.acm.org/
cial thanks goes to the MIT students for the endless support in the 10.1145/1349026.1349044.
[29] J. Ropponen, K. Lyytinen, Components of software development risk: how to
empirical investigation. Finally, we thank the management of Do- address them? a project manager survey, IEEE Transactions on Software
main Techx for giving us the opportunity to work in this project. Engineering 26 (2) (2000) 98–112. http://dx.doi.org/10.1109/32.841112.
[30] J. Jiang, G. Klein, Software development risks to project effectiveness, The
Journal of Systems and Software 52 (1) (2000) 3–10. doi:http://dx.doi.org/
References 10.1016/S0164-1212(99)00128-4.
[31] Y.H. Kwak, J. Stoddard, Project risk management: lessons learned from
[1] B. Boehm, A. Egyed, J. Kwan, D. Port, A. Shah, R. Madachy, Using the winwin software development environment, Technovation 24 (11) (2004) 915–920.
spiral model: a case study, Computer 31 (7) (1998) 33–44. http://dx.doi.org/ doi:10.1016/S0166-4972(03)00033-6.
10.1109/2.689675. [32] S. Islam, S.H. Houmb, D. Mendez-Fernandez, M.M.A. Joarder, Offshore-
[2] D.W. Karolak, Software Engineering Risk Management, IEEE Computer Society outsourced software development risk management model, in: Proc. of the
Press, 1995. 12th IEEE International Conference on Computer and Information Technology
[3] S.-W. Foo, A. Muruganantham, Software risk assessment model, in: (ICCIT 2009), Dhaka, Bangladesh, 2009, pp. 514–519, doi:http://dx.doi.org/
Proceedings of the IEEE International Conference on Management of 10.1109/ICCIT.2009.5407292.
Innovation and Technology (ICMIT 2000), 2000. [33] S. Islam, S.H. Houmb, Integrating risk management activities into
[4] J. Kontio, Software engineering risk management: A method, improvement requirements engineering, in: Proc. of the 4th IEEE Research International
framework and empirical evaluation, Ph.D. thesis, Helsinki University of Conference on Research Challenges in Information Science(RCIS2010), Nice,
Technology, 2001. France, 2010.
[5] G. Roy, A risk management framework for software engineering practice, in: [34] S. Islam, S.H. Houmb, Towards a framework for offshore outsource software
ASWEC ’04: Proceedings of the 2004 Australian Software Engineering development risk management model, Journal of Software (JSW), Special
Conference, IEEE Computer Society, Washington, DC, USA, 2004, p. 60. Issue: Selected Papers of the IEEE International Conference on Computer and
[6] S. Islam, Software development risk management model-a-goal-driven Information Technology (ICCIT 2009) 6 (1) 2011.
approach, Ph.D. thesis, Institute für Informatik, Technische Universität [35] S. McConnell, Rapid Development, Taming Wild Software Schedules, Microsoft
München, Germany, 2011. Press, 1996.
[7] R.L. Glass, Software Runaways: Monumental Software Disasters, Prentice-Hall, [36] T. Saarinen, An expanded instrument for evaluating information system
1998. success, Information and Management 31 (2) (1996) 103–118. doi:http://
[8] A. van Lamsweerde, Requirements Engineering: From System Goals to UML dx.doi.org/10.1016/S0378-7206(96)01075-0.
Models to Software Specifications, Wiley, 2009. [37] J.D. Procaccino, J.M. Verner, S.P. Overmyer, M.E. Darter, Case study: factors for
[9] E.E. Odzaly, P.D. Greer, Software risk management barriers: an empirical study, early prediction of software development success, Information and Software
in: ESEM ’09: Proceedings of the 2009 3rd International Symposium on Technology 44 (1) (2002) 53–62.

Please cite this article in press as: S. Islam et al., An empirical study on the implementation and evaluation of a goal-driven software development risk
management model, Inform. Softw. Technol. (2013), http://dx.doi.org/10.1016/j.infsof.2013.06.003
 S. Islam et al. / Information and Software Technology xxx (2013) xxx–xxx 17

[38] M. Broy, A. Fleischmann, S. Islam, L. Kof, C. Leuxner, K. Lochmann, D. Mendez- [47] S.L. Pfleeger, B.A. Kitchenham, Principles of survey research: part 1: turning
Fernandez, B. Penzenstadler, W. Sitou, S. Winter, Towards an integrated lemons into lemonade, SIGSOFT Software Engineering Notes 26 (6) (2001) 16–
approach to requirement engineering, Tech. rep., TUM-I0935, Technische 18. http://doi.acm.org/10.1145/505532.505535.
Universität München, Germany, December 2009. [48] J.W. Creswell, Research Design: Qualitative, Quantitative, and Mixed Methods
[39] B. Schätz, Model-based development of software systems: from models to Approaches, Sage Publications, 2002.
tools, habilitation, Technische Universität München, 2008. [49] I.P.C.M. Varkevisser, A. Brownlee, Designing and conducting health systems
[40] D.M. Fernández, B. Penzenstadler, M. Kuhrmann, M. Broy, A meta model for research projects: vol. 1 – proposal development and fieldwork. chapter 10:
artefact-orientation: Fundamentals and lessons learned in requirements Data collection techniques, Tech. rep., KIT Publishers, Amsterdam, 2002.
engineering, in: MoDELS (2), 2010, pp. 183–197. [50] Domain Software Technologies Ltd., <http://www.domtech.co.uk/> (accessed
[41] B. Freimut, S. Hartkopf, P. Kaiser, J. Kontio, W. Kobitzsch, An industrial case October 2010).
study of implementing software risk management, SIGSOFT Software [51] Digital Bangladesh Vision 2021, <http://www.boi.gov.bd> (accessed October
Engineering Notes 26 (5) (2001) 277–287. http://doi.acm.org/10.1145/ 2010).
503271.503247. [52] D.I.K. Sjoberg, T. Dyba, M. Jorgensen, The future of empirical methods in
[42] C. Chittister, Y. Haimes, Risk associated with software development: A holistic software engineering research, in: FOSE ’07: 2007 Future of Software
framework for assessment and management, IEEE Transactions on Systems, Engineering, IEEE Computer Society, Washington, DC, USA, 2007, pp. 358–
Man and Cybernetics 23 1993. 378. doi:http://dx.doi.org/10.1109/FOSE.2007.30.
[43] R.K. Yin, Case Study Research: Design and Methods, SAGE Publications Inc., [53] L. Wallace, M. Keil, Software project risks and their effect on outcomes,
2008. Communications of the ACM 47 (4) (2004) 68–73. http://doi.acm.org/10.1145/
[44] R.M. Davison, M.G. Martinsons, N. Kock, Principles of canonical action 975817.975819.
research, Information Systems Journal 14 (2004) 65–86. [54] K.R. Linberg, Software developer perceptions about software project failure: a
[45] B.A. Kitchenham, S.L. Pfleeger, M.L. Pickard, P.W. Jones, D.C. Hoaglin, K.E. case study, The Journal of Systems and Software 49 (2–3) (1999) 177–192.
Emam, J. Rosenberg, Preliminary guidelines for empirical research in software doi:http://dx.doi.org/10.1016/S0164-1212(99)00094-1.
engineering, IEEE Transactions on Software Engineering 28 (8) (2002) 721– [55] M. Keil, P.E. Cule, K. Lyytinen, R.C. Schmidt, A framework for identifying
734. http://dx.doi.org/10.1109/TSE.2002.1027796. software project risks, Communications of the ACM 41 (11) (1998) 76–83.
[46] S. Easterbrook, J. Singer, M. Storey, D. Damian, Selecting Empirical Methods for http://doi.acm.org/10.1145/287831.287843, issues 11.
Software Engineering Research, Springer, 2007.

Please cite this article in press as: S. Islam et al., An empirical study on the implementation and evaluation of a goal-driven software development risk
management model, Inform. Softw. Technol. (2013), http://dx.doi.org/10.1016/j.infsof.2013.06.003